ipfsel

ipfsel

This program is used to Select and sort lines from an incoming data files using
a selection file.

- There is a debugging switch, -X, which can be run ONLY with the single shot
switch, -1. This will prompt and display what is going on at each point.

A parameter file is used to determine where the files should be sent and other
parameters. This file is in tables/from/select and default to SELECTION. This
may be overridden by the contents of the Fip Hdr field 'DF'.

Whereever you see a reference to the DATA file, that is the incoming file that
'ipfsel' has found in its spooled queue.

A SELECTION file is a list of keys of those records in the DATA file.

There are various options on whther you want to Merge the two files and/or
Match records. They are essentially :
    1. Only data items that match the selection file
    2. Data items that do NOT match the selection
    3. Merge complete data and selection files
    4. Merge only records from both files that match
    5. All Selection file plus matched items of the data file

In order to Match, you need to define a common key is both files to compare.
This can be a field or a fixed position in each record. Naturally the key can
be in a different place in the each file but must be consistent - it cannot
wander about in the record !

As neither file is probably in the correct order, both files are normally
sorted so that the keys are in ascending order (thats a Must by the way !!).
This uses the Unix SORT command (qv manual pages). 90% of tuning 'ipfsel' is
getting the two files sorted correctly first, so you may find it useful to do
the sort manually in a gash area until it does the job.

Keywords for SELECTION parameter file are :
    ; comment line
    dest:   One or more FIP destinations for the output file as per the
        USERS file. There is NO default and the parameter is advised !
    before: String of FipSeq to stuff at the start of the file  def:none
    after:  String of FipSeq to stuff at the end of the file    def:none
    script: Script to run on the new file before sending to the output que
        The first parameter will be the outputfile containing the
        correct data using a TMP filename. The second is the name
        requested.                  default: none
    nohdr:  Flag to ask for no Fip Header on the output file. def:allow hdr
    nodata: Flag to ask for NO output file at all.
        If a script is flagged, a file is created for the script
        and is then zapped.         default: allow data
    chrset: Name of the Source Character Set to use for IPXCHG. def:ascii
    extra:
    extra-fiphdr:   Extra Fip Header information to be added.   default: none
            Also see -h which does the same (extra-fiphdr: overwrites -h)
        This is added ONLY if a FipHdr is added to the file.
    archive: Archive the data in IPWHEEL using this Logname. default: no

    filename: FipSeq for a replacement filename.    default: as input
    doneq:  Done queue for original input file  default: file is deleted

    keycase:(in/sensitive)
        Set the key comparison case sensitive or INsensitive
        default is sensitive.
    sort-program: (path to sort)
        Specify path/name of sort program
        This so you can fix on either 'old' sort with syntax like +0 -1 +4rn -5
        or 'new' / POSIX sort with -k1,2 etc
        default is 'sort' - so it will pickup the first in you path
    sortdone: Sort parameters for output file as per the unix 'sort'.
        default: -n +0 -1 for ascending, numeric order, first field ('old' version of
sort)

Keywords describing the data file :
---------------------------------
You must define the key of the data line. This can be at a fixed position on
the line (using fixedkey:) or the nth field (using fieldkey:).
EITHER  fieldkey: Field number of the sortkey or index field.
OR  fixedkey: Character position on the line of the key starting at 1.
    datasep: Field separator.   default: any combination of spc or tab
        This is a single chr and maybe specified as
        a FipSeq. Eg:    fieldsep:\177  for an octal 177
    keylen:  Maximum length of key.

    sortdata: Sort parameters for incoming file as per the unix 'sort'.
        default: -d +0 -1 for descending order, first field
        if using old sort, fields start at 0 :  -d +0 -1 +1 -2
        if using new sort, fields start at 1 :  -d -k1,2 -k2,3
        Note that if the field separator is NOT spc/tab you will also
        have to specify this.
    nosortdata: Do NOT sort the incoming data.
    befline: String of FipSeq to stuff at the start of the file def:none
    aftline: String of FipSeq to stuff at the end of the file   def:none
    eoln:   Line separator      default: any combination of CR NL

Keywords describing the selection file :
--------------------------------------
    selfile: Full Path and Filename of the selection file - NO DEFAULT.
    sortsel: Sort parameters for selection file as per the unix 'sort'.
        default: -d +1 -2 for descending order, SECOND field
        Why the Second field ?
        Because ipfsel adds/prefixes a sequence number to the begining of each
selection line - so it can resort the Output file back to eaxctly the same
order as it was specified.
        Note that if the field separator is NOT spc/tab you will also have to specify
this.
    nosortsel: Do NOT sort the selection file before use.
    selsep: Field separator.    default: any combination of spc or tab
        This is a single chr and maybe specified as
        a FipSeq. Eg:    fieldsep:\177  for an octal 177
    offset: where the selection key is NOT the first part of the line,
        this is the number of chrs from the left.   default: 0
    ignorenulkeys:  Ignore any Selection record with a no key specified
                            default: pass as is

The actual selection file itself (as specified in 'selfile' parameter), obeys
the following Syntax:
    ; comment line with preceeding semicolon (eoln)
    (key) (spc/tabs) (field2) (spc/tabs) (field3) .. etc (eoln)

Keywords describing how to match/merge
--------------------------------------
Which file is sorted in the required output sequence.
    major:data
    major:sel   - this is the default
Combination of match and merge defines what is output :
    1. Only data items that match the selection file
        match:
    2. Data items that do NOT match the selection
        nomatch:
    3. Merge complete data and selection files
        merge:
    or  merge:selfirst
    or  merge:datafirst
        default is selection record first
    4. Merge only records from both files that match
        match:
        merge:onmatch
    5. All Selection file plus matched items of the data file
        match:
        merge:
The default, if neither match or merge is stated is option 1 - match only.

Where the records in the Selection file is in a different format from that of
the Data file, you can reformat the output of the Selection data with :
    seldata: (Fip Seq data)
where (internal) Hdr Fields :
        DK is the key
        DD is the data part of the Selection.
        DR is the whole record
eg: seldata:XX,\DK\\z\DD        will give
    selection record    :   123456 Sample Data for Selection
    outgoing data       :   XX,123456\zSample Data for Selection NL
while   seldata:smalltst-\DR        will give for the same data
            smalltst-123456 Sample Data for Selection
A NL, or whatever 'eoln' is set to, is added automatically to the line.

Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).

    They are normally specified :
        fixed:QZ    1234543
        partial:QT  ST,3,2,U,<,>
        combie:QY   ep|na,(0000000)a 
        option:QE   ep,11,7,s
        repeat:QK   XK,-,3  
    or  repeat:QP   PK,,4,#X
        style:QS    XN,%.03d 

********************* Notes **************************************

-- Pls never allow any of your scripts to loop as ipfsel waits for completion
before continung !

---- How to setup a new Fsel ----

- There is a debugging switch, -X, which can be run ONLY with the single shot
switch, -1. This will prompt and display what is going on at each point.

    Get an input file from somewhere !
    Create a selection file which is the list of records you want
    Create a parameter file - remember to add the selection file parameter !
    run fsel manually
        ipfsel -1 rawtestdata -z selpramfile -X

- Remember that with -X, the input file is NOT deleted and there is the output
file is left in the tmp queue 'x'.

- A large proportion of the time taken to get the merge right will be to get
the SORT correct (qv manual pages). We suggest you run these manually several
times until you are 100 % certain before slotting into the parameter file.

---- Restrictions in this version of ipfsel ----

-- Note that offset and fixedkey do the same thing for data and sel but one
works
from 0 and the other 1 !
!
..
-- fieldkey does NOT work - no code yet
-- merge:selfirst and merge:datafirst can be overruled by the final sort. So
take care with the sort parameters as to which records go first.

Input Parameters are :
EITHER
    -i : input queue            default: spool/formselect
        If this does NOT start with a '/', it is assumed under spool.
OR  -1 : name of a single file (complete path pls)  default: spooled queue
        This will run the program once for that file only
        The input file is NOT deleted.
    -o : output or done queue for output file. default: file set to 'form'
        If this does NOT start with a '/', it is assumed under spool.
        Any original file in the output queue is overwritten.
    -z : default parameter file in tables/form/select.  def: SELECTION
    -w : file wait for files arriving across a network. def: 0 secs
    -u : owner if not that of the logon default: logon at start
    -l : do NOT log files in        default: log
    -X : set debugging ON           default: no
        This only takes effect with the -1 single shot switch
        And stops the outputfile being sent on.
    -v : print version number and exit

Version Control
;004f   16sep96 WINNT
    ;a-e 08mar06 cleanups, keycase, fixed len keys
    ;f 11jul12 added sort-program and sortdone parameters

(copyright) 2017 and previous years FingerPost Ltd.