ipfsel (Sat Oct 25 2014 01:31:01)

ipfsel

This program is used to Select and sort lines from an incoming data files using
a selection file.

- There is a debugging switch, -X, which can be run ONLY with the single shot
switch, -1. This will prompt and display what is going on at each point.

A parameter file is used to determine where the files should be sent and other
parameters. This file is in tables/from/select and default to SELECTION. This
may be overridden by the contents of the Fip Hdr field 'DF'.

Whereever you see a reference to the DATA file, that is the incoming file that
'ipfsel' has found in its spooled queue.

A SELECTION file is a list of keys of those records in the DATA file.

There are various options on whther you want to Merge the two files and/or
Match records. They are essentially :
	1. Only data items that match the selection file
	2. Data items that do NOT match the selection
	3. Merge complete data and selection files
	4. Merge only records from both files that match
	5. All Selection file plus matched items of the data file

In order to Match, you need to define a common key is both files to compare.
This can be a field or a fixed position in each record. Naturally the key can
be in a different place in the each file but must be consistent - it cannot
wander about in the record !

As neither file is probably in the correct order, both files are normally
sorted so that the keys are in ascending order (thats a Must by the way !!).
This uses the Unix SORT command (qv manual pages). 90% of tuning 'ipfsel' is
getting the two files sorted correctly first, so you may find it useful to do
the sort manually in a gash area until it does the job.

Keywords for SELECTION parameter file are :
	; comment line
	dest:	One or more FIP destinations for the output file as per the
		USERS file. There is NO default and the parameter is advised !
	before:	String of FipSeq to stuff at the start of the file	def:none
	after:	String of FipSeq to stuff at the end of the file	def:none
	script:	Script to run on the new file before sending to the output que
		The first parameter will be the outputfile containing the
		correct data using a TMP filename. The second is the name
		requested.					default: none
	nohdr:	Flag to ask for no Fip Header on the output file. def:allow hdr
	nodata:	Flag to ask for NO output file at all.
		If a script is flagged, a file is created for the script
		and is then zapped.			default: allow data
	chrset:	Name of the Source Character Set to use for IPXCHG. def:ascii
	extra:
	extra-fiphdr:	Extra Fip Header information to be added.	default: none
			Also see -h which does the same (extra-fiphdr: overwrites -h)
		This is added ONLY if a FipHdr is added to the file.
	archive: Archive the data in IPWHEEL using this Logname. default: no
		
	filename: FipSeq for a replacement filename.	default: as input
	doneq:	Done queue for original input file	default: file is deleted

	keycase:(in/sensitive)
		Set the key comparison case sensitive or INsensitive
		default is sensitive.
	sort-program: (path to sort)
		Specify path/name of sort program
		This so you can fix on either 'old' sort with syntax like +0 -1 +4rn -5
		or 'new' / POSIX sort with -k1,2 etc
		default is 'sort' - so it will pickup the first in you path
	sortdone: Sort parameters for output file as per the unix 'sort'.
		default: -n +0 -1 for ascending, numeric order, first field ('old' version of
sort)

Keywords describing the data file :
---------------------------------
You must define the key of the data line. This can be at a fixed position on
the line (using fixedkey:) or the nth field (using fieldkey:).
EITHER	fieldkey: Field number of the sortkey or index field.
OR	fixedkey: Character position on the line of the key starting at 1.
	datasep: Field separator.	default: any combination of spc or tab
		This is a single chr and maybe specified as
		a FipSeq. Eg:	 fieldsep:177	for an octal 177
	keylen:	 Maximum length of key.

	sortdata: Sort parameters for incoming file as per the unix 'sort'.
		default: -d +0 -1 for descending order, first field
		if using old sort, fields start at 0 :	-d +0 -1 +1 -2
		if using new sort, fields start at 1 :	-d -k1,2 -k2,3
		Note that if the field separator is NOT spc/tab you will also
		have to specify this.
	nosortdata: Do NOT sort the incoming data.
	befline: String of FipSeq to stuff at the start of the file	def:none
	aftline: String of FipSeq to stuff at the end of the file	def:none
	eoln:	Line separator		default: any combination of CR NL

Keywords describing the selection file :
--------------------------------------
	selfile: Full Path and Filename of the selection file - NO DEFAULT.
	sortsel: Sort parameters for selection file as per the unix 'sort'.
		default: -d +1 -2 for descending order, SECOND field
		Why the Second field ?
		Because ipfsel adds/prefixes a sequence number to the begining of each
selection line - so it can resort the Output file back to eaxctly the same
order as it was specified.
		Note that if the field separator is NOT spc/tab you will also have to specify
this.
	nosortsel: Do NOT sort the selection file before use.
	selsep: Field separator.	default: any combination of spc or tab
		This is a single chr and maybe specified as
		a FipSeq. Eg:	 fieldsep:177	for an octal 177
	offset: where the selection key is NOT the first part of the line,
		this is the number of chrs from the left.	default: 0
	ignorenulkeys:	Ignore any Selection record with a no key specified
							default: pass as is

The actual selection file itself (as specified in 'selfile' parameter), obeys
the following Syntax:
	; comment line with preceeding semicolon (eoln)
	(key) (spc/tabs) (field2) (spc/tabs) (field3) .. etc (eoln)


Keywords describing how to match/merge
--------------------------------------
Which file is sorted in the required output sequence.
	major:data
	major:sel	- this is the default
Combination of match and merge defines what is output :
	1. Only data items that match the selection file
		match:
	2. Data items that do NOT match the selection
		nomatch:
	3. Merge complete data and selection files
		merge:
	or	merge:selfirst
	or	merge:datafirst
		default is selection record first
	4. Merge only records from both files that match
		match:
		merge:onmatch
	5. All Selection file plus matched items of the data file
		match:
		merge:
The default, if neither match or merge is stated is option 1 - match only.

Where the records in the Selection file is in a different format from that of
the Data file, you can reformat the output of the Selection data with :
	seldata: (Fip Seq data)
where (internal) Hdr Fields :
		DK is the key
		DD is the data part of the Selection.
		DR is the whole record
eg:	seldata:XX,DK\zDD		will give
	selection record	:	123456 Sample Data for Selection
	outgoing data		:	XX,123456zSample Data for Selection NL
while	seldata:smalltst-DR		will give for the same data
			smalltst-123456 Sample Data for Selection
A NL, or whatever 'eoln' is set to, is added automatically to the line.


Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).
 
	They are normally specified :
		fixed:QZ	1234543
		partial:QT	ST,3,2,U,<,>
		combie:QY	ep|na,(0000000)a 
		option:QE	ep,11,7,s
		repeat:QK	XK,-,3  
	or	repeat:QP	PK,,4,#X
		style:QS	XN,%.03d 

********************* Notes **************************************

-- Pls never allow any of your scripts to loop as ipfsel waits for completion
before continung !


---- How to setup a new Fsel ----

- There is a debugging switch, -X, which can be run ONLY with the single shot
switch, -1. This will prompt and display what is going on at each point.

	Get an input file from somewhere !
	Create a selection file which is the list of records you want
	Create a parameter file - remember to add the selection file parameter !
	run fsel manually
		ipfsel -1 rawtestdata -z selpramfile -X

- Remember that with -X, the input file is NOT deleted and there is the output
file is left in the tmp queue 'x'.

- A large proportion of the time taken to get the merge right will be to get
the SORT correct (qv manual pages). We suggest you run these manually several
times until you are 100 % certain before slotting into the parameter file.

---- Restrictions in this version of ipfsel ----

-- Note that offset and fixedkey do the same thing for data and sel but one
works
from 0 and the other 1 !
!
..
-- fieldkey does NOT work - no code yet
-- merge:selfirst and merge:datafirst can be overruled by the final sort. So
take care with the sort parameters as to which records go first.

Input Parameters are :
EITHER
	-i : input queue			default: spool/formselect
		If this does NOT start with a '/', it is assumed under spool.
OR	-1 : name of a single file (complete path pls)	default: spooled queue
		This will run the program once for that file only
		The input file is NOT deleted.
	-o : output or done queue for output file. default: file set to 'form'
		If this does NOT start with a '/', it is assumed under spool.
		Any original file in the output queue is overwritten.
	-z : default parameter file in tables/form/select.	def: SELECTION
	-w : file wait for files arriving across a network.	def: 0 secs
	-u : owner if not that of the logon	default: logon at start
	-l : do NOT log files in		default: log
	-X : set debugging ON			default: no
		This only takes effect with the -1 single shot switch
		And stops the outputfile being sent on.
	-v : print version number and exit


Version Control
;004f	16sep96 WINNT
	;a-e 08mar06 cleanups, keycase, fixed len keys
	;f 11jul12 added sort-program and sortdone parameters

(copyright) 2014 and previous years FingerPost Ltd.