This program prepares incoming data to tweak it before IPFORMAT is run against

Each line of data - or each record if it is not a line - is checked against a
series of tests to try and determine the type of line it is. In the parlance of
this program is tries to find a KEY. According to this key the record is either
thrown away or, more usually, labelled with a record type. This record type is
normally at the beginning of the line as that is the easiest for IPFORMAT to
catch and work on.

A series of parameter lines in the normal FIP syntax are used to describe the
patterns and tests. (see below for a fuller description of all the variables).

Firstly the file is described with the parameters :
    filtyp      - type of file
    recsep      - each record ends with ...
    reclen      - (optional fixed record length)
    wild        - if you are using WILD cards, what chr signifies

The next thing is a series of FMTs and KEYs which try to test the line eg :
    fmt:BLANK       s:0 b:0
    key:BLANK       tst:2=""        delete
ie split the line into two fields - the first contains all the leading spaces
and the second is the rest of the line. If field 2 has nothing in it, it is a
blank line. The action is strip the line.

Normally if the result of the tests is good, you just want to label the line.
In this case, if fields 3, 6, 9 and 12 each have a single space and field 1 is
NOT a space, then we prefix "ZOG," to the begining of the line
    fmt:ZOG         b:1 b:9 s:1 b:1 b:9 s:1 b:1 b:9 s:1 b:1 b:9 s:1 b:0
    key:ZOG         chk:3 chk:6 chk:9 chk:12        tst:1#" "       addkey:1:ZOG,

You can also create new records from the fields you have created :
    fmt:STX         b:40 s:0 n:0 s:0 b:0
    key:STX         chk:1   chk:3   tst:5=""        delete  newrec:STX,{3},{1}
This deletes the original record and adds a new one starting "STX," followed by
field 3 comma field 1.

Lastly it is good idea to have a catch all for the lines you missed or for the
majority of lines :
    ; setup the default keyname for text files
    defkey:     addkey:1:DAT,
This just prefixes four chrs "DAT," at position 1 of the line. Note there
should be at least one space/tab after 'defkey:' and any (or all) default
action - addkey/delete/newrec

Notes - the NAME of a 'fmt' must be unique but there can be several 'key's for
that fmt which may be testing different things - and creating newrecords.
    ; Headers Par Value - strip
    fmt:HDR         p:10 b:0
    key:HDR         tst:1="Par Value "              delete  newrec:PAR,next
    key:HDR         tst:1="Div NvG   "              delete  newrec:DIV,next
    key:HDR         tst:1="Yld Notes "              delete  newrec:YLD,next
    key:HDR         tst:1="Chg on wk "              delete  newrec:CHG,next
    key:HDR         tst:1="Earn/Shr  "              delete  newrec:ERN,next
Here the original record is replaced by one of five records with no extra data.

Each FMT is checked sequentially - ie top down in the parameter file. If a
single key matches, then the parameter file is checked for more key lines with
the same name. Processing stops on the next 'fmt' record. This means all the
keys for a fmt should follow immediately after as in th examples above.

Note that if you have a 'newrec' and do not want to delete the original, the
new one is inserted BEFORE the original.

FMT - Syntax : and a single line , field marked [] are optional
    fmt:(name) (typ1):(len1)[:(startpos1):(startstring1):(endstr1)]
        ... (typn):(lenn)
    where   name is the name of the record format (case insensitive)
        tabs or spaces are separators between fields
        colons are separators between parms for the same field
        length or size of field is 0 for variable size
        type is a-alphabetic, u-uppercase, l-lowercase
          n-number, p-printable, s-space(or tab, CR, NL or FF)
          b-binary (ie anything), x-alphanumeric,
          c-control (ie < 040 or >= 0177),
          z-anpa hdr field (ie alnum plus non-quad/format punct.
        startposn is the starting posn in the record from posn1
        startstring is the starting string for the field
        endstring is the end string for the field

KEY - Syntax :
    key:(name) chk:(fldx) tst:(fldx)=(string) tst:(fldn)#(string)
        addkey:(posn):(len) delete: newrec:(syntax for newrec)
    key:(name)      same name as FMT previously specified
        chk:3       chk field 3 is of the format is the right length
                or if field 3 is varibale length, has 1 or more.
        tst:7=ABC   Does field 7 equal ABC
        tst:9#"   "   Does field 9 NOT equal 4 spaces
                Note that tst is case sensitive.
        addkey:1:3  Add a key of the name to the record at 1
        delete:     Delete this record
        newrec:PAG0003  Add a record before with data PAG0003 (recsep)
                DO NOT ADD THE END OF LINE it is done automatically

RECKEY Check the type and size of the key
Syntax :    reckey: : (length) : (type)
where   length or size of key is 0 for no size check
    type is a-alphabetic, u-uppercase, l-lowercase
        n-number, p-printable, s-space(or tab, CR, NL or FF)
        b-binary (ie anything), x-alphanumeric,
        c-control (ie < 040 or >= 0177),
        z-anpa hdr field (ie alnum plus non-quad/format punct.

Input parameters are :
    Optional :
    -1 : filename (and optional path) of a single file  def: spool queue
        The input file is NOT deleted in this case.
    -i : input queue            default: spool/fprep
    -o : output queue           default: spool/form
    -f : file creep time for files      default: 0
        arriving across the network - normally make it 5 secs
    -t : scan time of directory     default: 5 secs
    -l : do NOT log every incoming file default: log
    -z : use the basename of the file for the parameter file if there is ..
        .. no FP FIP header field   default: use FP only
    -Z : always use this parameter file default: use the FP field.
    -v : display version number and exit.

Example :

; type of file is text

; each record is separated by a CR CR NL

; strip all cr nl combinations including blank lines
; this boils the 'recsep' into a single CR

; There are no reclen and reckey commands - comment them out

; change the wild card chr to a dollar (normally it is a star '*')

; blank lines - to be stripped - put first to catch all of them
fmt:BLANK       s:0 b:0
key:BLANK       tst:2=""        delete
; text 8 col lines
fmt:TX8         s:1 p:38 p:1
key:TX8         tst:3="-"       addkey:1:TX8
; format of the trailer records
; (spc) AJ (4 num) (2 spc) (10 prinatable repeated 4 times)
fmt:TRAILER     s:1 u:2 n:4 s:2 p:28 p:28 p:28 p:28
; a trailer record needs to be deleted - and subrecords created from it 
key:TRAILER     chk:1 tst:2=AJ chk:3 chk:4 delete
key:TRAILER     chk:1 tst:2=AJ chk:3 chk:4 tst:5#"          "  newrec:E{5}
key:TRAILER     chk:1 tst:2=AJ chk:3 chk:4 tst:6#"          "  newrec:E{6}
key:TRAILER     chk:1 tst:2=AJ chk:3 chk:4 tst:7#"          "  newrec:E{7}
key:TRAILER     chk:1 tst:2=AJ chk:3 chk:4 tst:8#"          "  newrec:E{8}
; Main Heading - check the syntax of fields 1 and 3 and make sure f2 = '1'
fmt:HEAD        S:8 N:1 S:20 P:14 P:12:44 p:4:56
key:HEAD        chk:1 tst:2=1 chk:3     addkey:1:HD1
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="CITROEN" newrec:PAG0003
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="PEUGEOT" newrec:PAG0024
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="RENAULT" newrec:PAG0104
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="TALBOT"  newrec:PAG0105
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="FORD"    newrec:PAG0106
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="MAZDA"   newrec:PAG0107
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="LADA"    newrec:PAG0108
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="ROVER"   newrec:PAG0109
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="NISSAN"  newrec:PAG0141
key:HEAD        chk:1 tst:2=1 chk:3     tst:4="BMW"     newrec:PAG0144
; 1st Col headers
fmt:COL         n:0 s:39 p:10 p:10 p:10 p:10 p:10 p:10 p:10 p:10
key:COL         tst:1#0 tst:3#"  "      addkey:1:COL
; 2nd Col headers
fmt:CL2         s:40 p:10 p:10 p:10 p:10 p:10 p:10 p:10 p:10
key:CL2         chk:1   tst:2#"   "     addkey:1:CL2
; Section Head
fmt:SECTION     s:1 u:1 t:1 p:0
key:SECTION     chk:1 chk:2 tst:3="-"   addkey:1:SCT
; SubSection Head
fmt:SUBSECT     s:1 u:1 n:0 t:1 p:0
key:SUBSECT     chk:1 chk:2 chk:3 tst:4="-"     addkey:1:SUB

; setup the default keyname for text files
defkey:     addkey:1:TX8

(copyright) 2017 and previous years FingerPost Ltd.