iproute

iproute

IPROUTE may be used in three areas :
    - normal use as the routing stage for all wire files arriving on a system and
passing the files to 'ipwheel' to distribute.
Two lessers uses are :
    - applying extra Metadata according selections of FipHdr fields
        This is triggered by the '-METAONLY' input switch
    - moving incoming files to folders in /fip/spool depending on FipHdr fields
        This is triggered by the 'DEST-IS-FOLDER:' keyword in a single parameter file
OR the -Q input switch.

Normal routing is :

IPROUTE reads a file generated by one of the WIRE programs and attempts to
decide where to route it. This it does by looking at the FIP Header fields DU
(destination or end user), SU (logical source or user i.e. wire name) or SR for
the alternate routing filename, SC (source character set) and SH (source
header).

A lookup table with the name of SU header field (in uppercase) (or SR if it is
optionally there) is inspected to find where the file should be sent. This
parameter file can be forced to be always the same if the -S input switch is
used to specify it.

If the file has a SD (duplicate wire) then the file is copied (well, linked)
with the new name.

Optionally if the file has a DU then no further processing needs to be done.
This is flagged by the -r or %REUSEDU flags.

This program does NOT send it - it merely creates a link in the output queue
2GO in the form :

    #SU:srcuser#DU:destuser#SN:filename#SC:chrset_date_time[_optionals]

On input you can change the default settings of DU, SU, SN and SC.

All parameters are optional - although there MUST be routing file for each
incoming feed even if it contains a single comment line.

Normally the minimum file will contain :
    - comments (naturally!)
    - a default destination (using the default-dest: or %DU keyword)
    - a source character set (using the default-chrset or %SC keyword)

Other normal keywords are :
    - a number of routing lines
    - redun-hosts or %SZ field if there is a redundant feed
    - %LOG to display the headline or other meaningful fields.

Note that the '%' in front of each keyword in the parameter file is no longer
required. So each of these keywords may be specified as either:
    %HDRKEY:mykeywordfile
or  hdrkey:mykeywordfile

The syntax of the routing file is :
    ; comment line (comments are also valid on the rest of line).
    DU: (default destination setting if not matched)
        eg: %DU:logdel
        which will send all unrouted copy to the archive and no further
        eg: %DU:apspo+spoprt+chrisq
        which will send unrouted copy to the three destinations
    SC: (set source character set)
    SZ: (primary,secondary host for redundant wires)
        SZ:fip1, gate2
        Note BOTH systems must be specified, neither need be the
        system that the file first came in on
    KEY: (keyword file containing keywords to search for)
        %KEY:DAILY_WORDS
        A file in tables/keyword/DAILY_WORDS exists with a list of
        keywords to search for on separate lines. If such a Keyword is
        found it is added to the FipHdr as the HK field. A match is made
        is a string is found in the stream of data. To look only for words,
        use the PIPE, '|' to specify the begining/end.
        ie  mac will find 'macintosh', 'machine', 'applemac'
            |mac    will find the first two only
            mac|    will find 'applemac' only
            |mac|   will find 'mac' only
        A word is one that starts/ends with a space or ends with ',.;?!:)([]{}'.
        Apostrophes are ignored.
    KEY-TEXT-MAX: (number or FipSeq)
        Maximum number of chrs to check in the Keyword file
        Use this to trim the search to the first 400 chrs for example
        default: up to 64k
    HDRKEY: (Also look in these fields for the Keywords)
        HDRKEY:\XK \AB
        Look in the K field of the SH (source header) plus the AB field
        for the same keywords as the text.
        The syntax is denote each field with a preceeding backslash
        plus a space to separate : "\AB (space) \QQ (space) \FQ"
    CASE: (keywords and text searches are case SENSITIVE)
        Normally words specified in the Keyword file are case insensitive.
        So an entry like 'Humbug' will find :
            HUMBUG, humbug, hUmBuG
        Set this flag to force the program to only search for an EXACT match
            - 'Humbug' in this example.
    CONDCASE: (Condition lines are case SENSITIVE)
        The test in a condition line is normally caseINsensitive. So the following
        line will trigger on any case like 'TV', 'tv', 'tV' or 'Tv' :
            1   c=tv        >qpstv       ; send tv pages to qps
        Almost all wire services send codes which are case-insensitive.
        AFP's EAA service however has category codes like 'FsI' or 'Sss'
        where 's' and 'S' have different meanings. Use '%CONDCASE' to
        flag this.
    LOG: (Custom logging : text, unix and octal chrs, FIP hdr fields)
        Each file though is logged as a one line entry in the Item Log
        (log/ALL) The default format is :
       date time iproute !i Source(SU) Dest(DU) FileName(SN) Keyword(HK)
       eg:  Sat Jan 22 15:15:55 iproute !i : AP wsport APN0174 eggs
        which is represented by %LOG:\SU\t\DU\s\SN\s\HK
    NOLOGCTL: Strip all control chrs - such as BELL - from the log file
        default is to display all chrs
    NOLOG:  Do NOT log each item through at all
    NOLOGDELETE:    Do NOT log any item which is sent to 'delete'
    WILD: (Change the wild Chr from '*' to this chr)
    SINGLEWILD: (Change the single wild Chr from '?' to this chr)
        wild and single wild chrs may be used in any condition below
        to signify any chr or any string of chrs
        eg: %WILD:%
            1   c=Z%L
        will prove true for any category code (XC field of the Source
        Header (SH)) that begins Z and ends L.`
    WORD-BOUNDARY: (Chr in FipSeq)
        This is a single punctuation chr which can be used to mean the START or END
of a Word or field
        eg  %WORD-BOUNDARY:|
                1   WT=|ABC*
            will look for the contents of the WT Fiphdr field STARTING ABC
                1   WT=*XYZ|
            will look for the contents of the WT Fiphdr field ENDING XYZ
        The following chrs may NOT be used + > " # _ - =
        There is no default and so word/field boundaries are NOT tracked normally.
    TEXT: (no of chrs to look to routing information)
        This defaults to 8k and is 64k maximum.
    MAX-DOLLAR-T: (no of chrs for \$t - first bit of text)
        This can be up to a maximum of 2000 chrs - default is 80.
    HASH-IN-HDR: (FipSeq)
or  HASH-IN-DOLLAR-T: (FipSeq)
or  HASH-IN-DOLLAR-1: (FipSeq)
        Hash character ('#') to be used in \$t and \$1 to \$9. (Remember '#' is an
        end of FipHdr character, so it needs to be mapped out of the way)

    HASH-KEY-IGNORE-TOP: (number)
        Number of lines at the top of the file to ignore BEFORE starting the hash
        Use this if a date/time is on the first line or so.
        default: none
    HASH-KEY-START: (FipSeq)
        Start generating the hash-key (\%x) AFTER this character/string.
        default:none - ie no start string meaning ALL data from the start.
    HASH-KEY-STOP: (FipSeq)
        Stop generating the hash-key (\%x) on this character/string.
        default:none - ie no end string meaning data continues to the end
        ; stop hashing at the ^C/end-of-text as the datetime is ALWAYS different
        HASH-KEY-STOP:\003
    HASH-KEY-DATA: (FipSeq)
        Use this string to generate the hash
        default use the data in the file
        eg  hash-key-data:\XS-\XN-\XC-\XP-\XK

    TYPE-MARKERS: (2 punctuation chrs to be used as markers)
        For Tests, types-of-characters may also be used.
        There MUST be two, non-space characters used for the start
        and end of the types which must be punctuation and NOT the same
        you have used for wither wilds or word boundarys.
        eg  look for (space) (3 uppercase letters) (4 numbers) (spc)
            %TYPE-MARKERS:<>
            1   XK=*<SUUUNNNNS>*  +someDestination
        Note the start and end markers may be the same chr.
        You can also mix exact strings with types :
        eg  look for 'ABC' (4 numbers) (spc)
            %TYPE-MARKERS:|
            1   XK=*ABC<NNNNS>*       >weHaveABCfolks
    DE-DUPLICATE-DEST: (DU)
        Destination (DU) in sys/USERS to send 2nd (and other) duplicates
        Strip re-route duplicate files which do NOT have the same
        ItemNumber/FileName.
        default is - no deduplicate.
        A common de-dup destination might be 'logdel' to log a copy only.
        The incoming files list is in log/route/DEDUP_(SU)
    DE-DUPLICATE-HEAD: (FipSeq field for the Headline)
        (this depends on DE-DUPLICATE-DEST)
        THis can be a combie or some other sequence of FipSeq to generate a unique
string.
        eg  ; dedup on name - catcode - priority - take number - taketime
            fixed:Q1    \SN-\NC-\NP-\NQ-\NT
            de-duplicate-head:Q1
        default is 'XK'
        To use the hash key '\%x' in a De-Dup head, unfortunately you cannot add it
in a 'fixed:'
        So add it to ADDHDR-BEFORE-ROUTING eg
            ADDHDR-BEFORE-ROUTING:Q0:\%x
            fixed:Q1   \XK-\XC-\Q0
            DE-DUPLICATE-HEAD:Q1
    DE-DUPLICATE-DIFF: (number of bytes)
        This allows for slight differences in the text of the files. This fudge
factir is
        the allowable difference. So DE-DUPLICATE-DIFF:100 will catch files that have
the same
        Headline (and Source) and the file length is within 100 plus/minus.
        ie if a file is 987 bytes long and the original was 1086, it is de-duped.
        BUT if it is 1187, it will be considered a new file and allowed thru.
        If you do NOT want to dedup on size, set the value to -1 eg:
de-duplicate-diff:-1
            default: 0 for zero bytes difference.
    DE-DUPLICATE-MAX: (number of Items)
        (this depends on DE-DUPLICATE-DEST)
        Total number of entries to be tracked in the time period
            default: 100
        As duplicate files normally follow each other, generally this total dows not
have to be very big.
    DE-DUPLICATE-TIME: (number of seconds)
        Time period for duplicates in seconds.      default: 3600 for 1 hour
    DE-DUPLICATE-SOURCE: (name)
        This is the name of the 'collection' of files to check for duplicates and/or
throttling.
        It is normally the Source (SU) but can be FipSeq.
        This is need to define the uniqueness of the data.
        It is used to create the name of the log file in log/route.
    DE-DUPLICATE-BALANCE: (balance group)           default: none
        Name of a Group in the sys/BALANCE file to syncronise the DEDUP file across
systems
        Note there will normally be up to a second delay in balancing, so duplicate
files arriving on both system within the baalnce time may be un-duped.
        The new items are APPENDED to the DEDUP file on the remotes. As cleanup (iw
trimming according to time) is only done on a new file on that system, remote
systems which never receive files (ie pure DR sites) will need an occasional
full copy - or nightly maintenance should be used to zap the files in log/route
on those systems.
    DE-DUPLICATE-IGNORE-DU: (yes/no)
        Ignore the Destination/DU for this traffic  default: do NOT ignore
        Use this where the two files may have different DUs because of round-robin

    THROTTLE-MAX: (number of Items in FipSeq)
        Only allow this number of files through for the time period.
        Use this where you not only want to deduplicate but also restrict data.
        Note that throttling also uses de-duplications. In particular, DD-DEST, HEAD,
MAX and/or TIME must be specified.
        The parameter can either be a number OR a FipSeq Fiphdr field :
        In this example the max is normally 10 files during the period unless the
FipHdr SU is 'AFX' which allows 99 files.
            THROTTLE-MAX:\Q3
            option:Q1   SU,,,,AFX
            fixed:Q2    \Q199\$o
            combie:Q3   Q2,10
        default: no max to allow ALL files through
    THROTTLE-FIPHDR: (FipSeq)
        Extra FipHdr information to add to any throttled copy   default: none
        This could be a banner or some extra Meta data for the next process.
    LOCALE: (locale for this source)
        Use a different 'locale' ( look at the man pages for locale)
        Most computers are set to US English and never changed.
        Use this parameter to customise any date/time or Chr translations
        The parameter MUST be a valid locale on your system!
        To find out what valid locale exist, type 'locale -a' on unix.
        Eg  ; Set for for Brazil, portugese
            locale:pt_BR
    ADDHDR: Extra, optional, fixed FipHdr information to added
        AFTER any routing and before the file is passed on.
        Often this is add extra fixed fields or to change the unique
        name for redundancy. eg:
            %ADDHDR:QQ:APTEST#ZH:SNXV$T#QZ:temp
    ADDHDR-BEFORE-ROUTING: Extra, optional, fixed FipHdr information
        added BEFORE any routing is checked.
            ; map the extra fields to LCC
            ; eg of incoming : KI:LCC MRG LCCX EMRG
            ; we want KI:LCC MRG EMRG
            replace:Q1  KI  LCCX=LCC LCCY=LCC LCCZ=LCC
            unique:Q2   Q1
            ADDHDR-BEFORE-ROUTING :XX:oldKI=\KI\mKI:\Q2\n
        If using any FipSeq - like in the example -, they MUST be
        specified ABOVE/BEFORE the addhdr-before-routing line in the
        parameter file. Any specified below will be ignored.
    ADDHDR-BEFORE-SCRIPT: Extra, optional FipHdr information generated by an
external program or script
    ADDHDR-SCRIPT: Extra, optional FipHdr information generated by an external
program or script
        ADDHDR-SCRIPT:/fip/local/find_iim.pl \E4 > \E3
        \E3 holds the name of a TMP file to create that will be read for the list.
        \E4 holds the name of a TMP file that holds the contents of the incoming file
            this is a copy of the incoming file but with a safe filename.
        ** if you are going to use the FipHdr fields from the script for routing or
logging, pls use the '-BEFORE-' versions
        This is a 'must use' if using the Fip version of rdjpgcom.
        eg ADDHDR-BEFORE-SCRIPT:/fip/3rdparty/rdjpgcom -fiphdr \E4 > \E3
        NOTE - that the script uses Forward Slashes for the file names. So Windows
uses may need to convert in the script.
    REUSEDU: If the DU exists for this service, use it and ignore the
        routing table.  Normally if a DU exists, it is ignored and the
        tables searched.
    US-TICKERS: Look for tickers in text - (NASDAQ:MSFT) (NYSE:FPOST)
        and stick in the R* FipHdr fields.
            R1:NYSE:FPOST
            R2:NASDAQ:MSFT
            up to 9 codes can be saved.
            format is (Exchange) (colon) (Ticker)
        This may be of value for Business Wire and PRnewswire services.
        (Note for versions 158z - currently please do NOT specify BOTH BIZWIRE-HEADER
(or PRNEWSWIRE_HEADER) and US-TICKERS)
    TICKER-TAG:<.> start, separater and end chrs for US-Tickers.  default is (:)
for (NYSE:F)
    REPLACE-TICKER: Replace a US-Ticker in text (as bracketed by (:) or <.>) with
this FipSeq
        Several agencies - Reuters, DPA - send a Ticker code after the
        company name : BP Plc <BP.LSE> Use this to replace the ticker with other
links
        The ticker data is available as FipHdr Z1
        eg  REPLACE-TICKER:<a href="http://www.nasty.metabank.org/tickers=\Z1>\Z1</a>
    EURO-TICKERS:
    EURO-NAMES:
        These are lists of FipHdr fields with tickers and names to search in text.
            euro-tickers:EG,EL,EQ,EV
            euro-names:EF,EK,EP,EU
            us-tickers:
            ticker-tag:<.>
            replace-ticker:\T9
            fixed:T9    \V3 (<a href="\T0\T4:\T5">\T5</a>) \$o
    BIZWIRE-HEADER: Interpret the Business Wire Header at the top of text.
        ( BW)(TEST-TEST-2)(ABC) This is a Test; Please Disregard This Test
        These are put in B* FipHdr fields.
            BU for Source   'BW'
            BK for Keywords 'TEST-TEST-2'
            B1 to B9 for Ticker codes   'ABC'
            BH for Headline 'This is a Test; Please Disregard This Test'
                The headline MAY be multiline, any end-of-lines are forced to a space.
            BD: Keywords            'EUROPE'
            BI: Industry Keywords   'MULTIMEDIA'
            BF: for Supplier Source 'Ardvark Associates'
            For RNS work,
                BS for ShortName    'Abbey National PLC'
                BC for FSA Category 'SUS'
                BZ for Sequence Number  00000007
                BR for Time of Receipt
    PRNEWSWIRE-HEADER: Interpret the PR Newswire Header at the top of text.
        These are put in B* FipHdr fields.
            BR for Ticker codes 'ABC'
            BH for Headline 'This is a Test; Please Disregard This Test'
            The headline MAY be multiline, any end-of-lines are forced to a space.
            BD: Subject Keywords    'ASI PDT'
            BI: Industry Keywords   'ERN TLS'

    AFPXML: Look for AFP tagged keywords - <org>DASSAULT AVIATION<orgid
idsr="ISIN" etc
        and stick in the R* FipHdr fields.
            R1 - First TickerCode   - ISIN*FR0000121725*DASSAULT AVIATION
            R2 - Second TickerCode  - SICOVAM*12172*DASSAULT AVIATION
            ..
            R9 - Ninth Ticker Code
                format is (ORG) * (Code) * (Company Name)
        There is a variant - AFPXML-link which also looks for the first occurrence
        of the name of the company in the text and adds
    BLOOMKWD: Look for Bloomberg-style keywords - NI EUROPE NI M&A NI FIP etc
        and stick in the HX FipHdr field.
    ANPA-HEADLINE:(FipHdrField)
        Look for the first ANPA-style Bold line and put in this FipHdr field.
        The Anpa bold is started with a '^'. In theory the headline is
        only one line, but many suppliers make it over two or more. So
        this flag will take data until a '^ @ < > ='
    REUTERS-DATELINE:KM,MH,MN,B8,B7
        Look for Reuters-style Dateline, Advisory and Bylines and put in
        the 5 FipHdr fields noted ie L1 implies :
            KM  DateLine
            MH  ByLine
            MN  Advisory
            B8  Correction Advisory
            B7  Embargo Advisory
    REUTERS-HEADLINE:KK,M9
    REUTERS-SLUGLINE:MD,M8
    REUTERS-LANGUAGE:KF
    REUTERS-SIGNOFF:K9
    REUTERS-SIGNOFF-SCRIPT:/fip/local/r_naomi_contacts.pl lang=\QL fh=bu,bx
output=\E3 data='\E5'
        Script for pulling apart the Reuters Signoff
    REUTERS-PRIORITY:KH
        Look in the KK FipHdr field for UPDATE or WRAPUP and create a Revision FipHdr
(M9)
            M9  This Revision|Last|Next|Y/N
    REUTERS-BUI-DATELINE:KM
        Put ShiftOut and ShiftIn around the Dateline for BUI
    REUTERS-CHK4BUIS:
        Cleanup any extraneous Bui chrs in the text
    REUTERS-TOTAL:15
        Number of lines to look for a Reuters Dateline - default 15
        Minimum is 1, max is 25
    ALLOWSPACES: Leave all spaces in the data field for comparison.
        Normally a field will have leading, trailing and multiple spaces stripped.
        eg WITHOUT this option a field containing (pipes show start and end of data)
            |   This is a   line with tabs   and spaces in  |
        will be tested as :
            |This is a line with tabs and spaces in|
    SEND-SINGLE-COPY: If the same destination is specified more than once,
        this option will only send a single copy of the file rather
        than as many times as it is specified.
    SEQNO-BALANCE-GROUP: (balance group)
    SEQNO-BALANCE-QUEUE: (balance queue)
    SEQNO-NAME: (Name for a Seqno)
        This will save and balance a seqno to this balance group
        If no QUEUE is specified, the default is spool/2balance
        If no NAME is specified, the default is the name of the SU (or SR) FipHdr
field
    CHECK-SEQNO: (fiphdr field)
        Check this FipHdr field contains a value which increments
        This is used to log any missing seqnos
    CHECK-SEQNO-MIN: (minimum value)
    CHECK-SEQNO-MAX: (maximum value)
        Set bounds for the sequence number checking - defaults are 1 and 9999
    SEQNO-RESETS-AT: (Hour:minute)
        Sequence number is reset at this time - default is 0000 for midnight
    FIPHDR-FOR-EXPAND: (FipHdr field)
        The fiphdr field which needs to be expanded - no default
    EXPAND-CODE: (code) (codes to add)
        Add extra codes to a FipHdr field if the 'code' exists.
        The 'code's are space separated.
        There can be one or many (up to 300) 'expand-code' lines. eg:
            fiphdr-for-expand:NO
            expand-code:A    C D E M O T U MTL GRO SOF OIL
            expand-code:C    C MTL GRO SOF
            expand-code:D    D T
    CREATE-ZQ:
        Use this to create a fake output sequence number in the FipHdr
        field ZQ. If there is a number to this parameter it will be used
        as the amount to increment - default is 2.
        eg default create ZQ:3-4, the next ZQ:5-6
        if create-ZQ:12, file will have ZQ:22-33 then ZQ:34-45 etc.
        if create-ZQ:1,  file will have ZQ:2-2 then ZQ:3-3 etc.

    DEST-IS-FOLDER: (yes/no)
        Destination names are actually folders under /fip/spool
        default is DU are destinations in the sys/USERS file.
        This overrides the -Q switch
        Note all folder names are forced lowercase.
        These folders are now speedy-fip if that option is running.
        The DU is NOT appended in this case - so if you are putting into spool/2go,
add fiphdr fields
            1   SU=SPTZ  #DU:dev2sc+sptz2saves#SC:no
            1   SU=SPTZ  >2go

    DONE-QUEUE: (Folder name in FipSeq)
        Put the input file into this folder after processing
        This overrides the -f Input switch.
    BEFORE: (FipSeq)
    AFTER:  (FipSeq)
        Text to conditionally add at the top or bottom of text
    SCRIPT: Optional Script to run AFTER the file has been sent successfully.
        eg  script:/fip/local/FTP_NEXT_STAGE \YI-\YS
    TRACKER-SCRIPT: Optional Tracking script - run AFTER script (if any)
        default: none

    MAX-ROUND-ROBIN: (no or 1-49)              default: no
        add a round-robin number to the output queue (same as -R)
    ROUND-ROBIN-FIPHDR: (2 letter FipHdr code)      default: none
        both these parameters need to be specified
    BIG-FILE-LIMIT: (size in bytes)             default: none
    BIG-FILE-FIPHDR: (2 letter FipHdr code)         default: none
        Flag YES or NO if the file is bigger than the limit.
        both these parameters need to be specified
        This is added as a FipHdr before routing and can be used to Route on.

    FILE-TRACE: yes/no, + hdr + in + out
        Trace all files using this parameter file   default: no
    FILE-TRACE-NAME: (FipSeq) unique name of this file  default: \SU-\SN-\HR
    FILE-TRACE-LOG: (FipSeq) additional logging     default: none
        These can be individually turned on by FipHdr fields in the incoming data :
            ZFILE_TRACE: ZFILE_TRACE_NAME: ZFILE_TRACE_LOG:
        (notice the '_' if using in the FipHdr but '-' if parameters)

----
The rest of the file is a series of conditions or rules are applied to each
incoming file.
The format of these is :
    tab (level_no) tab (condition)
    tab (level_no) tab (condition) tab > (destination)
    tab (level_no) tab (condition) tab (+opt additional DU)
    tab (level_no) tab (condition) tab (>) DU1+DU2+DU3..+DUn)
    tab (level_no) tab (condition) tab (+DU1+DU2+DU3..+DUn)

    - Each rule is checked starting from the top of the file.
    - where tab can be any combination of tabs and spaces.
    - where the level_no is a number. If it is greater then 1, then this line is
        only checked IF the preceeding level was matched.
    - each rule is one and only one line.
    - destinations - which MUST be entries in the tables/sys/USERS file - are
specified as :
            >dest1
        or  >dest1+dest2
        or  +dest3
        - Multiple destinations are separated by a '+'.
        - If a line has NO destination, processing continues with the next line.
        - If a line has one or more destinations starting with a '>', Searching
            STOPS and the file sent to those destinations (without the '>' or '+').
            NO more tests are processed.
        - If a line has one or more destinations starting with a '+', Searching
            CONTINUES while those destinations are saved for sending later (without the
'+').

    - Syntax of a condition is :

            (FipHdr Field) = (data)
        eg:     1   p=3 >prio3

        If the FipHdr field is a single letter it is from the Source Header (SH).
            eg for standard wires
                s is service designator
                n is item number
                p is priority
                c is category
                w is number of words
                k is keyword or headline
                etc
        which has a format in the data file like :
            SH::SEAA:N0436:P3:CI:W0260:RUSA /AFP-EC08:KUS-Asia-radio

        If the FipHdr field is two letters, it is that actual field such as EK in
this example.
        eg:     1   EK=88   >2FatLadies

        Routing may also be on data in the text using the '$$'as the base. eg:
            2   $$="New York Times"   >nyt
        For data, the condition is not EQUALS but CONTAINS.
        ie The following will be true :
                AbcdefThe New York Timeszzzzzzz
            and New York Times\n

        Actual type-of-file or the 'magic number' can be tested with '$f'
        eg  ; Is it a Word 2000 file ?
            1   $f=\xCF\xD0\xE0\x11 >translateWord2000

        If a Keyword file is being searched, Routing can be done on a match.
        In this case '%%' is used and the name of the Keyword file.
        (The %KEY parameter is ignored in this case).
        In this example a file is held in 'tables/keyword/FOREIGN_LIST'.
            1   c=INT               ; for Category = International ..
            2   %%=FOREIGN_LIST >mainforeign ; Main For queue for matches
            2   c=*     >junkforeign ; Rest to junk

    - Condition is either '=' to signify equals or '#' meaning NOT equal. eg:
        1   c=INT   >allints
        1   c#INT   >noints

    - in the case where several tests need to be made to determine a destination
        then the preceeding lines have no destination
            ; If category starts 's' ..
                1   c=s*
            ; send a COPY of all high priority files to the editor, and keep searching
                    2   p=1     +hied
            ; if Keyword is SOCCER - send to the soccer destination
                    2   k=SOCCER    >soccer      ; send race cards
            ... etc
        There can be up to 100 levels if you really need.

    - Use wild strings and chrs using (by default) '*' for strings and '?' for a
single chr.
        (These can be changed using %WILD or %SINGLEWILD as above)
    eg  7   k=*SOCCER*
        will be true for XK fields such as :
            SOCCER_RESULTS
            These are the Soccer Headlines
            And so for soccer
    - Case - upper or lowercase - are NORMALLY ignored so that SOCCER' and
'soccer' are equal
        unless specifically noted
            either  with the '%CONDCASE' flag for FipHdr fields
            or  with the '%CASE' flag for Text and Keyword matches.

    - Note that if any spaces are embedded, double quotes should surround the
condition.
        3   k="Liverpool Beatles"

    - Testing for something or nothing !
        The following test will always be true, no matter whether there
        is a FipHdr L1, a blank L1 or an L1 with data.
            1   L1=*    >IamTrue
        So to test if there IS something there :
            1   L1#"" >thereIsSomut
        Or nothing (ie no L1 or L1 with no data) :
            1   L1="" >thereisNothing

    - Special Case - you can also add extra FipHdr information on the end of a
destination
        (or destinations)
        1   p=1 >w4rns+rnsEd#JH:RNS-PRIORITY
    or and example where you want to continue
        1   p=1
            2   c=HHH   #JH:RNS-PRIORITY
            2   c=HHH   +w4rns+rnsEd
        Note that the '#' sign marks the end of any destinations and the begining of
FipHdr.
        The FipHdr bit may be in FipSeq :
            1   p=1 >w4rns+rnsEd#JH:RNS-TIME-\$d

For example :

; routing table for AP
; default destination is junk
%DU:junk
; chrset is 'anpa'
%SC:anpa
;
; If category starts 's' ..
    1   c=s*
; level 2 - so for all category 's' etc, if the priority is '1' ..
; send to sptpri1 and do not look any further down the table.
        2   p=1 >spthipri    ; Priority 1 to sports hi pri
        2   SU=AP   >rac     ; send race cards
    1   c=t     >tv      ; tv pages

Note that if you use one of the \$1 to \$9 FipSeq functions to get the 1st to
9th line of text, any HASH is replaced by x9D.

Note also that a couple of extra builtin FipHdrs exist
    \%x - generate a 32bit hash of the first 64k of the data

Input parameters are (all optional) :
    -1 : (minus one) filename for single-shot   default: folder is spooled
    -b : Balance ALL data files - balance queue default: (if -B specified)
spool/2balance
    -B : Balance ALL data files - balance group default: not balanced
    -c : default SC - source chrset         default: ascii
    -d : default DU - destination           default: woops
    -D : display contents of header fields as they are checked  def.no
        This is ONLY valid when using '-1' single shot and NO other
        output is generated - ie no files, just the check.
        eg iproute -1/fip/spool/test/abc.wire.9001.1234 -D
    -f : done queue for the input file      default: file is deleted
        This can be overridden by the DONE-QUEUE parameter
    -h : hostname to use                def: name of this host
        This is used for the XP flag for redundant feeds and should
        only be necessary on systems connected to more than one
        ethernet where the base hostname is NOT that to be used.
    -G : WINNT/2k - grab the first file only    default: no
    -H : display contents of header fields in HTML  def.no
            This also requires the -D input switch
    -i : input queue in /fip/spool      default: 2brouted
    -o : output queue in /fip/spool     default: 2go
    -l : (minus el) do NOT log          default: log
    -METAONLY : special case of iproute - pls see below.    default: no
    -m : name of the top MetaData file      default: METADATA or (WinNT -
METADATA.fip)
    -n : default SN - filename          default: UNK000
    -Q : all destinations are really folders NOT DU default: DU
        This can be selectively turned off for a single parameter file
        with DEST-IS-FOLDER:no
    -r : Do NOT search for a routing if a DU exits. def: always use table
        With this switch, if a DU field exists, it is used and the
        routing table ignored. Normally the contents of the DU are
        ignored and are overwritten with whatever destination
        is selected via the table. This switch is for all incoming
        services. If only one or two services need this, use the
        %REUSEDU flag in the routing table for that service (see above).
    -R : Running Round Robin on the output folder      default: no
    -s : default SU - source (usually the agency)   default: noname
    -S : force the parameter file to ALWAYS be this. default: FipHdr SR or SU is
used
    -t : interval between scans of the queue    default: 3 secs
    -X : ignore SZ redundancy           default: use SZ
        This stops a copy of an incoming file being sent automatically
        to the redundant system as in the SZ line of the routing file.
    -W : 10 ths of a second - MUST wait if nothing processed    default: 0
    -8 : input is NOT speedy (valid on a speedy system only)
        ie the input folder is NOT piped but the rest of the system is.
    -9 : run in Speedy mode          default: no
    -v : print version number and exit

-- The above defaults are ONLY used if any defaults are NOT set in the
parameter file (%DU, %SC etc).

-- Note that if 'iproute' is used for files which are not directly from the
wires - perhaps files which have to pass through twice, A selection of existing
FipHdr fields are zapped before processing if present.
 These are :
    - XZ (resent file flag - do NOT archive)
    - XS (do not send to other systems flag)
    - CX and DC (for xchg)
    - CQ change queue for xchg
    - SZ (to denote redundant systems)
    - XP and CP : source and destination FIP Hosts

This has no effect on FipHdr fields being ADDED at this stage (such as SZ being
added by the %SZ line) - only to delete any old FipHdr fields inserted by
another program.

The XZ flag stops 'ipwheel' from archiving the data.
The XS flag stops 'ipwheel' from sending to other systems.
CX and DC are usually added by 'ipwheel' to denote which XCHG to use for
'ipxchg'.
SZ is used to send a copy of the file over to the redundant system.
XP and CP are the source and destination FIP Hosts (ie system file arrived on
and where it was sent to)

-- If you do not want to log every time, then either use the parameter to STOP
ALL logging :
    'NOLOG:'
or, just a single, incoming file, add the FipHdr field ROUTE_LOG_EVERY:(999)
or, in the parameter file, for a filter :
    1   NN=grabeachsecond   #ROUTE_LOG_EVERY:600
    1   NN=grabeachsecond   >delete
The value for ROUTE_LOG_EVERY is the number of seconds to skip before loggin
the next occurance OR a negetive numbre for No logging at all. In the example
above, the first file AFTER 600 secs/5mins is logged.

-- Why would you send a file through 'iproute' twice ??

Perhaps you have several publications. The main wire feeds come into one (pair
of) systems and most of the data is sorted and processed on that system for the
main publication.
But a smaller publication - or a specialist publication such as a Sports or
Financial magazine wants a subset of data with totoally different sorting
requirements.
It is easy to split a copy of all the wire files using a line like :
route file :
    1   c=* +otherpub   ; copy all to destination otherpub
USERS file
    otherpub    DP:smlpubhost
and send to the 2nd pubications servers when 'iproute' would resort properly.
USERS file on 'smlpubhost' :
    otherpub    DP:localhost    DQ:2brouted SC:no DC:no

-- File tracing thru the Fip

There are a series of parameters common to serveral Fip Programs which will log
what is happening to all files, a service or a single file.

    file-trace: (valid actions - yes/no, + hdr + in + out)
        Actions are 'Yes' - log, 'No' - dont; if NOT NO, then in addition to logging,
we can save a copy of the FipHdr, Input file and/or Output
        (delete is ignored in iproute))

    file-trace-name: (UniqueId in FipSeq if not default -  \SU-\SN-\HR)

    file-trace-log: (extra File Tracing logging in FipSeq)
        The Hdr and files are left in /fip/log/file_trace (which should be purged in
the nightly maintenance zapfiplog)

    These can be overridden by FipHdr fields in each file if a particular file -
or service needs tracing.
        (notice the '_' if using in the FipHdr but '-' if parameters)
    eg  ZFILE_TRACE: hdr
        ZFILE_TRACE_NAME: louislouis
        ZFILE_TRACE_LOG: seqno.\ZQ

---------------- META ONLY ------------------------
Normally 'iproute' is used as the routing stage for all wire files arriving on
a system.

There is a special case where a second copy of the program may be used to add
MetaData (in the form of FipHdr fields) only to any incoming files. This is
flagged by the input switch -METAONLY. In addition the input switches '-i' and
'-o' for input and output folders MUST be specified. This is to make sure that
the 2nd copy does NOT interfere with the main copy flow.

You MUST also define an input and output folders (-i and -o).
 eg for the SYSTEM file :
    meta    iproute -METAONLY -m testmetadata.fip -i 2meta -o 2edsys

The parameter to the -m switch is a file in tables/metadata whose syntax is
different from the normal route file :
    ; comment
    check-file:(name of file)

Example of a MetaData file
; Sports - look in /fip/tables/metadata/PSP.FIP
check-file:PSP.fip

There can be several check-file(s) which all reside in tables/metadata.

  The syntax of the check-file is the normal 'ROUTE' as described above except

  1. 3 new keywords
        ADDHDR-IF-VALID:(FipHdr)
        ADDHDR-IF-INVALID:(FipHdr)
            These give the strings of FipHdr to add to the file if the tests are true or
false.
        DEFAULT-RESULT: (valid/invalid)
            The default result if all conditions are met is either 'valid' or 'notvalid'
or 'invalid' (the last 2 are the same !)

  2. The only valid destinations are
        stop-valid
            Stop looking in this MetaData file as the result is valid
            Continue with any further Metadata files specified in the top file
        stop-invalid
            Stop looking in this MetaData file as the result is invalid
            Continue with any further Metadata files specified in the top file
        stop-all-valid
            Stop looking in this MetaData file as the result is valid
            Do NOT continue looking any further
        stop-all-invalid
            Stop looking in this MetaData file as the result is valid
            Do NOT continue looking any further
        continue
            Stop checking at this level in this MetaData file
            Continue checking at the next (level - 1)
            ie
                1   SU=PA
                    2   XC=HHH
                        ; if Prioriy is 1 or 2, drop down to the next level 2
                        3   XP=1    >continue
                        3   XP=2    >continue
                        3   XP=*    >stop-invalid
                    2   XC=PPP

                        etc etc etc
  Example of a check-file:
; ------------------------------------------------------
; if still valid at the end, add VT:PSC
%ADDHDR-IF-VALID:\nNO:\NO PSC\n

; we have wilds
%WILD:*

; we assume that it is valid...
%default-result:Valid

%word-boundary:|

; ------------------------------------------------------
; If Status is NOT New or Priority 1 - always ignore
    1   NV#New  >stop-all-invalid
    1   IP=1    >stop-all-invalid

; ------------------------------------------------------
; want Product Codes G or SPO or E (continue to next level 1 or end)
    1   NO=*
        2   NO=|G|      >continue
        2   NO=|SPO|    >continue
        2   NO=|E|      >continue
        2   NO=*        >stop-invalid

; ------------------------------------------------------
; *** End ***

Version Control
;162f62 23oct07 added extra-fiphdr-script
    ;b 14jan08 allowed DE_DUPLICATE-DIFF:-1 for NO checking at all
    ;c 26apr08 keyword stripped ';' as comment - needed for '&aring;'
    ;d 31jul08 -b, -B added for dataBalGroup, balQue and SX
    ;e 27aug08 bugette with Reuters which are ONLY Xref/Signoff
    ;f1-4 18feb09 allow speedy with DEST-IS-FOLDER ;2 allow de-duplicate-time of
0=dont test time;3 NOLOGDELETE:
        ;4-5 3jun09 allow SD to tbe multiples SD:AP1,AP2,AP3
        ;6 22oct09 CHG - redid addhdr-script so it is the same as ipspool (with
E3/E4)
        ;7 16sep10 added DE-DUPLICATE-BALANCE
        ;8-9 24sep10 added ADDHDR-BEFORE-SCRIPT:..
        ;10 10oct10 ignore dedup if DU=logdel or delete !
        ;11 23oct10 bugette with DEDUP and no file
        ;12-14 3mar11 wrinkle with speedy and do NOT allow ';' to mean comments on
non-route lines and added \%x for CRC
        ;15 10may11 bugette in chk_route (WIN only)
        ;16-18 11jul11 added -8 for NOT speedy in but speedy out and \$1 back; added
HASH-IN-HDR as -$1 or-$t
        ;19-20 7dec11 added time for done queue duplicates
        ;21-26 30jan12 allow xml in Reuters-dateline and added HASH-KEY-STOP
        ;27  4jul12 added KEY-TEXT-MAX
        ;28-32 19jul12 do NOT add DU for DEST-IS-FOLDER and added ROUTE_LOG_EVERY as
FipHdr field
        ;33-35 20nov12 bugette of addhdr-before-route with no routing lines
        ;36 8mar13 added big-file-limit and max-round-robin and allow parsed routing
line : 1 c=* >2xml\RR
        ;37-38 7may13 wrinkle with ignoring the FipHdr in '1 c=* >abc+def#SC:ghi'
        ;39 6jun13 allow SH to change in ADDHDR, ADDHDR-BEFORE-ROUTING
        ;40 12sep13 added DE-DUPLICATION-IGNORE-DU
        ;41-43  1oct13 bugette in DE-DUP balance
        ;44 3jan14 added SHOW-DEDUP:yes/no
        ;45-48 16jan14 added SCRIPT and TRACKER-SCRIPT and refresh SN if it has
changed with ADDhdr etc
        ;49-52 added SEQNO_BALG/BALQ/NAME and REUTERS-SIGNOFF-SCRIPT ;52 added
file-trace
        ;53 added -T showTimes for debugging slowdowns ;54 ADDHDR-IF_NOT/INVALID doc
problem
        ;55 SX filesize \$c ;56-60 win error in FHscript
        ;61 20jul16 better big file support - EACCES
        ;62  7oct16 better DEST-IS-FOLDER for dups
;161z2  03dec04 bugette with more than one kwd search (1 %%=KWDFILE >abc)
    ;a 15dec04 added -f done queue and DONE-QUEUE:
    ;b-h 17dec04 bugette with >a#SC:a and %SC:def (f-Roy xref)
    ;i-j 28apr05 added REPLACE-TICKER
    ;k-l 16may05 mod for mtime/nfs waits
    ;m-p 05jul05 use AFP <org> markup in text and buglette in size of 1st dedup
    ;q-r 03sep05 added euro-tickers and euro-names
    ;s-v 18jan06 for THROTTLE-MAX and THROTTLE-FIPHDR and DE-DUPLICATE-SOURCE
        added BEFORE and AFTER
    ;w 1aug06 rename copied file if already exists
    ;x-y2 17nov06 bugette in test if singlewild '?' is first chr
    ;z2 08may07 added CREATE-ZQ
;160z   14nov03 tuning dedups (size is now JUST the data part)
    ;a 12dec03 added reuters-lines:
    ;b 12jan04 added anpa-headline:
    ;c-g 16feb04 cleanup HTML display and added expand-code
    ;h 01apr04 added RTR code for Advisory->Emb or Corr
    ;i 25apr04 added stop-all-valid/invalid for Metadata
    ;j-o 06jun04 bugette in AFPXML plus added REUTERS-BUI-DATELINE
    ;p-q 27jul04 no BUI in Dateline pls and single spaces in Dateline
    ;r-s 01sep04 -9 speedy (s for R-Adv)
    ;t-v 22sep04 added REUTERS-CHK4BUIS and tweak to Advisories
    ;w 04oct04 extra FipHdr lines are now added to FipHdr immediately
        .. plus Pilcrow again
    ;x-y 03nov04 RTRS - case of Xref after a Signoff
    ;z  17nov04 added dest-is-folder: and default-dest and -w filewait
;159z   06jan03 minor mod to Reuters/Byline
    ;a 13jan03 RTRS - added UPDATE/WRAPUP revision
    ;b-g 30jan03 RTRS - added Genre
    ;h 04mar03 RTRS - Signoff
    ;i 25may03 RTRS - get/zap HASHes in MetaFields
    ;j 29may03 RTRS - add xB6-Pilcrow for extra lines in Advisories
    ;k-l 04jun03 RTRS - strip trailing spaces/lines from Advisory
            and bugette for signoff
    ;m 29jun03 VON as a By plus poss correction advisory
    ;n 04jul03 added REUTERS-PILCROW
    ;o 23jul03 added PRNEWSWIRE-HEADER
    ;p 24jul03 redid $f for filetype - which was NOT working well
    ;q-v 04aug03 added de-duplicates
    ;w-y 06oct03 stripp and <Fip...> markers for Datelines
    ;z 31oct03 added timings
;158zi  28dec01 added TYPE-MARKERS and LOCALE and US-TICKERS
        ;a 22jan02 restrict size of market for US ticker to 20
        ;b 05feb02 added -1 for single shot
        ;c 12feb02 bugette with wildstrings
            ie k=*sport* sometimes true erroneously
        ;d/e 27feb02 added \$t and \$1-9 for ADDHDR
        ;f/g 03apr02 added BIZWIRE-HEADER
        ;h 14apr02 ADDHDR can now contain FipSeq for other FipHdrs
        ;i 18apr02 widgets put in - fixed, partial, combie etc
        ;j/k/l 21apr02 added check-seqno, seqno-resets-at, check-seqno-max,
check-seqno-min
        ;m 22may02 allow CC!="abc" as well as CC#"abc" and defaultSU put back in
        ;n 28may02 bugette in word boundary code
        ;o  5jun02 bugette in ADDHDR - not allowing \$1 etc
        ;p 13jun02 allow more than 2 source redundancy hosts (SZ).
        ;r 18jun02 added -D and -H for display only
        ;s/t/u 24jun02 added addhdr-before-routing
                and -G to grab-first-file
        ;v 16sep02 added $f for checking the file magic number
        ;w 17oct02 use calculated DU even if there is an incoming DU
        ;x/y 26nov02 added %REUTERS-DATELINE
        ;zi 11dec02 added more BIZWIRE kwds at end
;157b   12nov01 added -METAONLY/-m and word-boundary
;156p   16jul99 added %SEND-SINGLE-COPY:
        ;b zap FipHdr DP if in the incoming file.
        ;c 02jul00 added %AFPXML
        ;d 07aug00 bugette with multiple DUs which blow the filename.
        ;e 06nov00 added ExtraFipHdr info on the end of a dest.
        ;f 22nov00 bugette - when using duplicates (SD), sometimes overwrote original
file !
        ;g 02apr01 bugette - SendSingleCopy sometimes duplicated the 1st DU
        ;h/j 21may01 redid ExtrafipHdr
        ;k/l/m 25aug01 strip NULs from data part of file for comparison
        ;o 05sep01 redid ability to run external program
        ;p 28sep01 AFP changed their XML .... for %AFPXML

(copyright) 2017 and previous years FingerPost Ltd.