ipfsel This program is used to Select and sort lines from an incoming data files using a selection file. - There is a debugging switch, -X, which can be run ONLY with the single shot switch, -1. This will prompt and display what is going on at each point. A parameter file is used to determine where the files should be sent and other parameters. This file is in tables/from/select and default to SELECTION. This may be overridden by the contents of the Fip Hdr field 'DF'. Whereever you see a reference to the DATA file, that is the incoming file that 'ipfsel' has found in its spooled queue. A SELECTION file is a list of keys of those records in the DATA file. There are various options on whther you want to Merge the two files and/or Match records. They are essentially : 1. Only data items that match the selection file 2. Data items that do NOT match the selection 3. Merge complete data and selection files 4. Merge only records from both files that match 5. All Selection file plus matched items of the data file In order to Match, you need to define a common key is both files to compare. This can be a field or a fixed position in each record. Naturally the key can be in a different place in the each file but must be consistent - it cannot wander about in the record ! As neither file is probably in the correct order, both files are normally sorted so that the keys are in ascending order (thats a Must by the way !!). This uses the Unix SORT command (qv manual pages). 90% of tuning 'ipfsel' is getting the two files sorted correctly first, so you may find it useful to do the sort manually in a gash area until it does the job. Keywords for SELECTION parameter file are : ; comment line dest: One or more FIP destinations for the output file as per the USERS file. There is NO default and the parameter is advised ! before: String of FipSeq to stuff at the start of the file def:none after: String of FipSeq to stuff at the end of the file def:none script: Script to run on the new file before sending to the output que The first parameter will be the outputfile containing the correct data using a TMP filename. The second is the name requested. default: none nohdr: Flag to ask for no Fip Header on the output file. def:allow hdr nodata: Flag to ask for NO output file at all. If a script is flagged, a file is created for the script and is then zapped. default: allow data chrset: Name of the Source Character Set to use for IPXCHG. def:ascii extra: extra-fiphdr: Extra Fip Header information to be added. default: none Also see -h which does the same (extra-fiphdr: overwrites -h) This is added ONLY if a FipHdr is added to the file. archive: Archive the data in IPWHEEL using this Logname. default: no filename: FipSeq for a replacement filename. default: as input doneq: Done queue for original input file default: file is deleted keycase:(in/sensitive) Set the key comparison case sensitive or INsensitive default is sensitive. sort-program: (path to sort) Specify path/name of sort program This so you can fix on either 'old' sort with syntax like +0 -1 +4rn -5 or 'new' / POSIX sort with -k1,2 etc default is 'sort' - so it will pickup the first in you path sortdone: Sort parameters for output file as per the unix 'sort'. default: -n +0 -1 for ascending, numeric order, first field ('old' version of sort) Keywords describing the data file : --------------------------------- You must define the key of the data line. This can be at a fixed position on the line (using fixedkey:) or the nth field (using fieldkey:). EITHER fieldkey: Field number of the sortkey or index field. OR fixedkey: Character position on the line of the key starting at 1. datasep: Field separator. default: any combination of spc or tab This is a single chr and maybe specified as a FipSeq. Eg: fieldsep:\177 for an octal 177 keylen: Maximum length of key. sortdata: Sort parameters for incoming file as per the unix 'sort'. default: -d +0 -1 for descending order, first field if using old sort, fields start at 0 : -d +0 -1 +1 -2 if using new sort, fields start at 1 : -d -k1,2 -k2,3 Note that if the field separator is NOT spc/tab you will also have to specify this. nosortdata: Do NOT sort the incoming data. befline: String of FipSeq to stuff at the start of the file def:none aftline: String of FipSeq to stuff at the end of the file def:none eoln: Line separator default: any combination of CR NL Keywords describing the selection file : -------------------------------------- selfile: Full Path and Filename of the selection file - NO DEFAULT. sortsel: Sort parameters for selection file as per the unix 'sort'. default: -d +1 -2 for descending order, SECOND field Why the Second field ? Because ipfsel adds/prefixes a sequence number to the begining of each selection line - so it can resort the Output file back to eaxctly the same order as it was specified. Note that if the field separator is NOT spc/tab you will also have to specify this. nosortsel: Do NOT sort the selection file before use. selsep: Field separator. default: any combination of spc or tab This is a single chr and maybe specified as a FipSeq. Eg: fieldsep:\177 for an octal 177 offset: where the selection key is NOT the first part of the line, this is the number of chrs from the left. default: 0 ignorenulkeys: Ignore any Selection record with a no key specified default: pass as is The actual selection file itself (as specified in 'selfile' parameter), obeys the following Syntax: ; comment line with preceeding semicolon (eoln) (key) (spc/tabs) (field2) (spc/tabs) (field3) .. etc (eoln) Keywords describing how to match/merge -------------------------------------- Which file is sorted in the required output sequence. major:data major:sel - this is the default Combination of match and merge defines what is output : 1. Only data items that match the selection file match: 2. Data items that do NOT match the selection nomatch: 3. Merge complete data and selection files merge: or merge:selfirst or merge:datafirst default is selection record first 4. Merge only records from both files that match match: merge:onmatch 5. All Selection file plus matched items of the data file match: merge: The default, if neither match or merge is stated is option 1 - match only. Where the records in the Selection file is in a different format from that of the Data file, you can reformat the output of the Selection data with : seldata: (Fip Seq data) where (internal) Hdr Fields : DK is the key DD is the data part of the Selection. DR is the whole record eg: seldata:XX,\DK\\z\DD will give selection record : 123456 Sample Data for Selection outgoing data : XX,123456\zSample Data for Selection NL while seldata:smalltst-\DR will give for the same data smalltst-123456 Sample Data for Selection A NL, or whatever 'eoln' is set to, is added automatically to the line. Where sections of FipHdr fields are required or changes to the output style, use keywords : fixed, partial, combie, optional, repeat, newdate and/or style. (see The SysAdmin manual for more information). They are normally specified : fixed:QZ 1234543 partial:QT ST,3,2,U,<,> combie:QY ep|na,(0000000)a option:QE ep,11,7,s repeat:QK XK,-,3 or repeat:QP PK,,4,#X style:QS XN,%.03d ********************* Notes ************************************** -- Pls never allow any of your scripts to loop as ipfsel waits for completion before continung ! ---- How to setup a new Fsel ---- - There is a debugging switch, -X, which can be run ONLY with the single shot switch, -1. This will prompt and display what is going on at each point. Get an input file from somewhere ! Create a selection file which is the list of records you want Create a parameter file - remember to add the selection file parameter ! run fsel manually ipfsel -1 rawtestdata -z selpramfile -X - Remember that with -X, the input file is NOT deleted and there is the output file is left in the tmp queue 'x'. - A large proportion of the time taken to get the merge right will be to get the SORT correct (qv manual pages). We suggest you run these manually several times until you are 100 % certain before slotting into the parameter file. ---- Restrictions in this version of ipfsel ---- -- Note that offset and fixedkey do the same thing for data and sel but one works from 0 and the other 1 ! ! .. -- fieldkey does NOT work - no code yet -- merge:selfirst and merge:datafirst can be overruled by the final sort. So take care with the sort parameters as to which records go first. Input Parameters are : EITHER -i : input queue default: spool/formselect If this does NOT start with a '/', it is assumed under spool. OR -1 : name of a single file (complete path pls) default: spooled queue This will run the program once for that file only The input file is NOT deleted. -o : output or done queue for output file. default: file set to 'form' If this does NOT start with a '/', it is assumed under spool. Any original file in the output queue is overwritten. -z : default parameter file in tables/form/select. def: SELECTION -w : file wait for files arriving across a network. def: 0 secs -u : owner if not that of the logon default: logon at start -l : do NOT log files in default: log -X : set debugging ON default: no This only takes effect with the -1 single shot switch And stops the outputfile being sent on. -v : print version number and exit Version Control ;004f 16sep96 WINNT ;a-e 08mar06 cleanups, keycase, fixed len keys ;f 11jul12 added sort-program and sortdone parameters (copyright) 2017 and previous years FingerPost Ltd.