ipfsel
ipfsel
This program is used to Select and sort lines from an incoming data files using
a selection file.
- There is a debugging switch, -X, which can be run ONLY with the single shot
switch, -1. This will prompt and display what is going on at each point.
A parameter file is used to determine where the files should be sent and other
parameters. This file is in tables/from/select and default to SELECTION. This
may be overridden by the contents of the Fip Hdr field 'DF'.
Whereever you see a reference to the DATA file, that is the incoming file that
'ipfsel' has found in its spooled queue.
A SELECTION file is a list of keys of those records in the DATA file.
There are various options on whther you want to Merge the two files and/or
Match records. They are essentially :
1. Only data items that match the selection file
2. Data items that do NOT match the selection
3. Merge complete data and selection files
4. Merge only records from both files that match
5. All Selection file plus matched items of the data file
In order to Match, you need to define a common key is both files to compare.
This can be a field or a fixed position in each record. Naturally the key can
be in a different place in the each file but must be consistent - it cannot
wander about in the record !
As neither file is probably in the correct order, both files are normally
sorted so that the keys are in ascending order (thats a Must by the way !!).
This uses the Unix SORT command (qv manual pages). 90% of tuning 'ipfsel' is
getting the two files sorted correctly first, so you may find it useful to do
the sort manually in a gash area until it does the job.
Keywords for SELECTION parameter file are :
; comment line
dest: One or more FIP destinations for the output file as per the
USERS file. There is NO default and the parameter is advised !
before: String of FipSeq to stuff at the start of the file def:none
after: String of FipSeq to stuff at the end of the file def:none
script: Script to run on the new file before sending to the output que
The first parameter will be the outputfile containing the
correct data using a TMP filename. The second is the name
requested. default: none
nohdr: Flag to ask for no Fip Header on the output file. def:allow hdr
nodata: Flag to ask for NO output file at all.
If a script is flagged, a file is created for the script
and is then zapped. default: allow data
chrset: Name of the Source Character Set to use for IPXCHG. def:ascii
extra:
extra-fiphdr: Extra Fip Header information to be added. default: none
Also see -h which does the same (extra-fiphdr: overwrites -h)
This is added ONLY if a FipHdr is added to the file.
archive: Archive the data in IPWHEEL using this Logname. default: no
filename: FipSeq for a replacement filename. default: as input
doneq: Done queue for original input file default: file is deleted
keycase:(in/sensitive)
Set the key comparison case sensitive or INsensitive
default is sensitive.
sort-program: (path to sort)
Specify path/name of sort program
This so you can fix on either 'old' sort with syntax like +0 -1 +4rn -5
or 'new' / POSIX sort with -k1,2 etc
default is 'sort' - so it will pickup the first in you path
sortdone: Sort parameters for output file as per the unix 'sort'.
default: -n +0 -1 for ascending, numeric order, first field ('old' version of
sort)
Keywords describing the data file :
---------------------------------
You must define the key of the data line. This can be at a fixed position on
the line (using fixedkey:) or the nth field (using fieldkey:).
EITHER fieldkey: Field number of the sortkey or index field.
OR fixedkey: Character position on the line of the key starting at 1.
datasep: Field separator. default: any combination of spc or tab
This is a single chr and maybe specified as
a FipSeq. Eg: fieldsep:\177 for an octal 177
keylen: Maximum length of key.
sortdata: Sort parameters for incoming file as per the unix 'sort'.
default: -d +0 -1 for descending order, first field
if using old sort, fields start at 0 : -d +0 -1 +1 -2
if using new sort, fields start at 1 : -d -k1,2 -k2,3
Note that if the field separator is NOT spc/tab you will also
have to specify this.
nosortdata: Do NOT sort the incoming data.
befline: String of FipSeq to stuff at the start of the file def:none
aftline: String of FipSeq to stuff at the end of the file def:none
eoln: Line separator default: any combination of CR NL
Keywords describing the selection file :
--------------------------------------
selfile: Full Path and Filename of the selection file - NO DEFAULT.
sortsel: Sort parameters for selection file as per the unix 'sort'.
default: -d +1 -2 for descending order, SECOND field
Why the Second field ?
Because ipfsel adds/prefixes a sequence number to the begining of each
selection line - so it can resort the Output file back to eaxctly the same
order as it was specified.
Note that if the field separator is NOT spc/tab you will also have to specify
this.
nosortsel: Do NOT sort the selection file before use.
selsep: Field separator. default: any combination of spc or tab
This is a single chr and maybe specified as
a FipSeq. Eg: fieldsep:\177 for an octal 177
offset: where the selection key is NOT the first part of the line,
this is the number of chrs from the left. default: 0
ignorenulkeys: Ignore any Selection record with a no key specified
default: pass as is
The actual selection file itself (as specified in 'selfile' parameter), obeys
the following Syntax:
; comment line with preceeding semicolon (eoln)
(key) (spc/tabs) (field2) (spc/tabs) (field3) .. etc (eoln)
Keywords describing how to match/merge
--------------------------------------
Which file is sorted in the required output sequence.
major:data
major:sel - this is the default
Combination of match and merge defines what is output :
1. Only data items that match the selection file
match:
2. Data items that do NOT match the selection
nomatch:
3. Merge complete data and selection files
merge:
or merge:selfirst
or merge:datafirst
default is selection record first
4. Merge only records from both files that match
match:
merge:onmatch
5. All Selection file plus matched items of the data file
match:
merge:
The default, if neither match or merge is stated is option 1 - match only.
Where the records in the Selection file is in a different format from that of
the Data file, you can reformat the output of the Selection data with :
seldata: (Fip Seq data)
where (internal) Hdr Fields :
DK is the key
DD is the data part of the Selection.
DR is the whole record
eg: seldata:XX,\DK\\z\DD will give
selection record : 123456 Sample Data for Selection
outgoing data : XX,123456\zSample Data for Selection NL
while seldata:smalltst-\DR will give for the same data
smalltst-123456 Sample Data for Selection
A NL, or whatever 'eoln' is set to, is added automatically to the line.
Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).
They are normally specified :
fixed:QZ 1234543
partial:QT ST,3,2,U,<,>
combie:QY ep|na,(0000000)a
option:QE ep,11,7,s
repeat:QK XK,-,3
or repeat:QP PK,,4,#X
style:QS XN,%.03d
********************* Notes **************************************
-- Pls never allow any of your scripts to loop as ipfsel waits for completion
before continung !
---- How to setup a new Fsel ----
- There is a debugging switch, -X, which can be run ONLY with the single shot
switch, -1. This will prompt and display what is going on at each point.
Get an input file from somewhere !
Create a selection file which is the list of records you want
Create a parameter file - remember to add the selection file parameter !
run fsel manually
ipfsel -1 rawtestdata -z selpramfile -X
- Remember that with -X, the input file is NOT deleted and there is the output
file is left in the tmp queue 'x'.
- A large proportion of the time taken to get the merge right will be to get
the SORT correct (qv manual pages). We suggest you run these manually several
times until you are 100 % certain before slotting into the parameter file.
---- Restrictions in this version of ipfsel ----
-- Note that offset and fixedkey do the same thing for data and sel but one
works
from 0 and the other 1 !
!
..
-- fieldkey does NOT work - no code yet
-- merge:selfirst and merge:datafirst can be overruled by the final sort. So
take care with the sort parameters as to which records go first.
Input Parameters are :
EITHER
-i : input queue default: spool/formselect
If this does NOT start with a '/', it is assumed under spool.
OR -1 : name of a single file (complete path pls) default: spooled queue
This will run the program once for that file only
The input file is NOT deleted.
-o : output or done queue for output file. default: file set to 'form'
If this does NOT start with a '/', it is assumed under spool.
Any original file in the output queue is overwritten.
-z : default parameter file in tables/form/select. def: SELECTION
-w : file wait for files arriving across a network. def: 0 secs
-u : owner if not that of the logon default: logon at start
-l : do NOT log files in default: log
-X : set debugging ON default: no
This only takes effect with the -1 single shot switch
And stops the outputfile being sent on.
-v : print version number and exit
Version Control
;04g 16sep96 WINNT
;a-e 08mar06 cleanups, keycase, fixed len keys
;f 11jul12 added sort-program and sortdone parameters
(copyright) 2024 and previous years FingerPost Ltd.