iproute
IPROUTE may be used in three areas :
- normal use as the routing stage for all wire files arriving on a system and
passing the files to 'ipwheel' to distribute.
Two lessers uses are :
- applying extra Metadata according selections of FipHdr fields
This is triggered by the '-METAONLY' input switch
- moving incoming files to folders in /fip/spool depending on FipHdr fields
This is triggered by the 'DEST-IS-FOLDER:' keyword in a single parameter file
OR the -Q input switch.
Normal routing is :
IPROUTE reads a file generated by one of the WIRE programs and attempts to
decide where to route it. This it does by looking at the FIP Header fields DU
(destination or end user), SU (logical source or user i.e. wire name) or SR for
the alternate routing filename, SC (source character set) and SH (source
header).
A lookup table with the name of SU header field (in uppercase) (or SR if it is
optionally there) is inspected to find where the file should be sent. This
parameter file can be forced to be always the same if the -S input switch is
used to specify it.
If the file has a SD (duplicate wire) then the file is copied (well, linked)
with the new name.
Optionally if the file has a DU then no further processing needs to be done.
This is flagged by the -r or %REUSEDU flags.
This program does NOT send it - it merely creates a link in the output queue
2GO in the form :
#SU:srcuser#DU:destuser#SN:filename#SC:chrset_date_time[_optionals]
On input you can change the default settings of DU, SU, SN and SC.
All parameters are optional - although there MUST be routing file for each
incoming feed even if it contains a single comment line.
Normally the minimum file will contain :
- comments (naturally!)
- a default destination (using the default-dest: or %DU keyword)
- a source character set (using the default-chrset or %SC keyword)
Other normal keywords are :
- a number of routing lines
- redun-hosts or %SZ field if there is a redundant feed
- %LOG to display the headline or other meaningful fields.
Note that the '%' in front of each keyword in the parameter file is no longer
required. So each of these keywords may be specified as either:
%HDRKEY:mykeywordfile
or hdrkey:mykeywordfile
The syntax of the routing file is :
; comment line (comments are also valid on the rest of line).
DU: (default destination setting if not matched)
eg: %DU:logdel
which will send all unrouted copy to the archive and no further
eg: %DU:apspo+spoprt+chrisq
which will send unrouted copy to the three destinations
SC: (set source character set)
SZ: (primary,secondary host for redundant wires)
SZ:fip1, gate2
Note BOTH systems must be specified, neither need be the
system that the file first came in on
KEY: (keyword file containing keywords to search for)
%KEY:DAILY_WORDS
A file in tables/keyword/DAILY_WORDS exists with a list of
keywords to search for on separate lines. If such a Keyword is
found it is added to the FipHdr as the HK field. A match is made
is a string is found in the stream of data. To look only for words,
use the PIPE, '|' to specify the begining/end.
ie mac will find 'macintosh', 'machine', 'applemac'
|mac will find the first two only
mac| will find 'applemac' only
|mac| will find 'mac' only
A word is one that starts/ends with a space or ends with ',.;?!:)([]{}'.
Apostrophes are ignored.
KEY-TEXT-MAX: (number or FipSeq)
Maximum number of chrs to check in the Keyword file
Use this to trim the search to the first 400 chrs for example
default: up to 64k
HDRKEY: (Also look in these fields for the Keywords)
HDRKEY:XK AB
Look in the K field of the SH (source header) plus the AB field
for the same keywords as the text.
The syntax is denote each field with a preceeding backslash
plus a space to separate : "AB (space) QQ (space) FQ"
CASE: (keywords and text searches are case SENSITIVE)
Normally words specified in the Keyword file are case insensitive.
So an entry like 'Humbug' will find :
HUMBUG, humbug, hUmBuG
Set this flag to force the program to only search for an EXACT match
- 'Humbug' in this example.
CONDCASE: (Condition lines are case SENSITIVE)
The test in a condition line is normally caseINsensitive. So the following
line will trigger on any case like 'TV', 'tv', 'tV' or 'Tv' :
1 c=tv >qpstv ; send tv pages to qps
Almost all wire services send codes which are case-insensitive.
AFP's EAA service however has category codes like 'FsI' or 'Sss'
where 's' and 'S' have different meanings. Use '%CONDCASE' to
flag this.
LOG: (Custom logging : text, unix and octal chrs, FIP hdr fields)
Each file though is logged as a one line entry in the Item Log
(log/ALL) The default format is :
date time iproute !i Source(SU) Dest(DU) FileName(SN) Keyword(HK)
eg: Sat Jan 22 15:15:55 iproute !i : AP wsport APN0174 eggs
which is represented by %LOG:SUtDUsSNsHK
NOLOGCTL: Strip all control chrs - such as BELL - from the log file
default is to display all chrs
NOLOG: Do NOT log each item through at all
NOLOGDELETE: Do NOT log any item which is sent to 'delete'
WILD: (Change the wild Chr from '*' to this chr)
SINGLEWILD: (Change the single wild Chr from '?' to this chr)
wild and single wild chrs may be used in any condition below
to signify any chr or any string of chrs
eg: %WILD:%
1 c=Z%L
will prove true for any category code (XC field of the Source
Header (SH)) that begins Z and ends L.`
WORD-BOUNDARY: (Chr in FipSeq)
This is a single punctuation chr which can be used to mean the START or END
of a Word or field
eg %WORD-BOUNDARY:|
1 WT=|ABC*
will look for the contents of the WT Fiphdr field STARTING ABC
1 WT=*XYZ|
will look for the contents of the WT Fiphdr field ENDING XYZ
The following chrs may NOT be used + > " # _ - =
There is no default and so word/field boundaries are NOT tracked normally.
TEXT: (no of chrs to look to routing information)
This defaults to 8k and is 64k maximum.
MAX-DOLLAR-T: (no of chrs for $t - first bit of text)
This can be up to a maximum of 2000 chrs - default is 80.
HASH-IN-HDR: (FipSeq)
or HASH-IN-DOLLAR-T: (FipSeq)
or HASH-IN-DOLLAR-1: (FipSeq)
Hash character ('#') to be used in $t and $1 to $9. (Remember '#' is an
end of FipHdr character, so it needs to be mapped out of the way)
HASH-KEY-IGNORE-TOP: (number)
Number of lines at the top of the file to ignore BEFORE starting the hash
Use this if a date/time is on the first line or so.
default: none
HASH-KEY-START: (FipSeq)
Start generating the hash-key (%x) AFTER this character/string.
default:none - ie no start string meaning ALL data from the start.
HASH-KEY-STOP: (FipSeq)
Stop generating the hash-key (%x) on this character/string.
default:none - ie no end string meaning data continues to the end
; stop hashing at the ^C/end-of-text as the datetime is ALWAYS different
HASH-KEY-STOP:03
HASH-KEY-DATA: (FipSeq)
Use this string to generate the hash
default use the data in the file
eg hash-key-data:XS-XN-XC-XP-XK
TYPE-MARKERS: (2 punctuation chrs to be used as markers)
For Tests, types-of-characters may also be used.
There MUST be two, non-space characters used for the start
and end of the types which must be punctuation and NOT the same
you have used for wither wilds or word boundarys.
eg look for (space) (3 uppercase letters) (4 numbers) (spc)
%TYPE-MARKERS:<>
1 XK=*<SUUUNNNNS>* +someDestination
Note the start and end markers may be the same chr.
You can also mix exact strings with types :
eg look for 'ABC' (4 numbers) (spc)
%TYPE-MARKERS:|
1 XK=*ABC<NNNNS>* >weHaveABCfolks
DE-DUPLICATE-DEST: (DU)
Destination (DU) in sys/USERS to send 2nd (and other) duplicates
Strip re-route duplicate files which do NOT have the same
ItemNumber/FileName.
default is - no deduplicate.
A common de-dup destination might be 'logdel' to log a copy only.
The incoming files list is in log/route/DEDUP_(SU)
DE-DUPLICATE-HEAD: (FipSeq field for the Headline)
(this depends on DE-DUPLICATE-DEST)
THis can be a combie or some other sequence of FipSeq to generate a unique
string.
eg ; dedup on name - catcode - priority - take number - taketime
fixed:Q1 SN-NC-NP-NQ-NT
de-duplicate-head:Q1
default is 'XK'
To use the hash key '%x' in a De-Dup head, unfortunately you cannot add it
in a 'fixed:'
So add it to ADDHDR-BEFORE-ROUTING eg
ADDHDR-BEFORE-ROUTING:Q0:%x
fixed:Q1 XK-XC-Q0
DE-DUPLICATE-HEAD:Q1
DE-DUPLICATE-DIFF: (number of bytes)
This allows for slight differences in the text of the files. This fudge
factir is
the allowable difference. So DE-DUPLICATE-DIFF:100 will catch files that have
the same
Headline (and Source) and the file length is within 100 plus/minus.
ie if a file is 987 bytes long and the original was 1086, it is de-duped.
BUT if it is 1187, it will be considered a new file and allowed thru.
If you do NOT want to dedup on size, set the value to -1 eg:
de-duplicate-diff:-1
default: 0 for zero bytes difference.
DE-DUPLICATE-MAX: (number of Items)
(this depends on DE-DUPLICATE-DEST)
Total number of entries to be tracked in the time period
default: 100
As duplicate files normally follow each other, generally this total dows not
have to be very big.
DE-DUPLICATE-TIME: (number of seconds)
Time period for duplicates in seconds. default: 3600 for 1 hour
DE-DUPLICATE-SOURCE: (name)
This is the name of the 'collection' of files to check for duplicates and/or
throttling.
It is normally the Source (SU) but can be FipSeq.
This is need to define the uniqueness of the data.
It is used to create the name of the log file in log/route.
DE-DUPLICATE-BALANCE: (balance group) default: none
Name of a Group in the sys/BALANCE file to syncronise the DEDUP file across
systems
Note there will normally be up to a second delay in balancing, so duplicate
files arriving on both system within the baalnce time may be un-duped.
The new items are APPENDED to the DEDUP file on the remotes. As cleanup (iw
trimming according to time) is only done on a new file on that system, remote
systems which never receive files (ie pure DR sites) will need an occasional
full copy - or nightly maintenance should be used to zap the files in log/route
on those systems.
DE-DUPLICATE-IGNORE-DU: (yes/no)
Ignore the Destination/DU for this traffic default: do NOT ignore
Use this where the two files may have different DUs because of round-robin
THROTTLE-MAX: (number of Items in FipSeq)
Only allow this number of files through for the time period.
Use this where you not only want to deduplicate but also restrict data.
Note that throttling also uses de-duplications. In particular, DD-DEST, HEAD,
MAX and/or TIME must be specified.
The parameter can either be a number OR a FipSeq Fiphdr field :
In this example the max is normally 10 files during the period unless the
FipHdr SU is 'AFX' which allows 99 files.
THROTTLE-MAX:Q3
option:Q1 SU,,,,AFX
fixed:Q2 Q199$o
combie:Q3 Q2,10
default: no max to allow ALL files through
THROTTLE-FIPHDR: (FipSeq)
Extra FipHdr information to add to any throttled copy default: none
This could be a banner or some extra Meta data for the next process.
LOCALE: (locale for this source)
Use a different 'locale' ( look at the man pages for locale)
Most computers are set to US English and never changed.
Use this parameter to customise any date/time or Chr translations
The parameter MUST be a valid locale on your system!
To find out what valid locale exist, type 'locale -a' on unix.
Eg ; Set for for Brazil, portugese
locale:pt_BR
ADDHDR: Extra, optional, fixed FipHdr information to added
AFTER any routing and before the file is passed on.
Often this is add extra fixed fields or to change the unique
name for redundancy. eg:
%ADDHDR:QQ:APTEST#ZH:SNXV$T#QZ:temp
ADDHDR-BEFORE-ROUTING: Extra, optional, fixed FipHdr information
added BEFORE any routing is checked.
; map the extra fields to LCC
; eg of incoming : KI:LCC MRG LCCX EMRG
; we want KI:LCC MRG EMRG
replace:Q1 KI LCCX=LCC LCCY=LCC LCCZ=LCC
unique:Q2 Q1
ADDHDR-BEFORE-ROUTING :XX:oldKI=KImKI:Q2n
If using any FipSeq - like in the example -, they MUST be
specified ABOVE/BEFORE the addhdr-before-routing line in the
parameter file. Any specified below will be ignored.
ADDHDR-BEFORE-SCRIPT: Extra, optional FipHdr information generated by an
external program or script
ADDHDR-SCRIPT: Extra, optional FipHdr information generated by an external
program or script
ADDHDR-SCRIPT:/fip/local/find_iim.pl E4 > E3
E3 hold the name of a TMP file to create that will be read for the list.
E4 hold the name of a TMP file that holds the contents of the incoming file
this is a copy of the incoming file but with a safe filename.
** if you are going to use the FipHdr fields from the script for routing or
logging, pls use the '-BEFORE-' versions
This is a 'must use' if using the Fip version of rdjpgcom.
eg ADDHDR-BEFORE-SCRIPT:/fip/3rdparty/rdjpgcom -fiphdr E4 > E3
NOTE - that the script uses Forward Slashes for the file names. So Windows
uses may need to convert in the script.
REUSEDU: If the DU exists for this service, use it and ignore the
routing table. Normally if a DU exists, it is ignored and the
tables searched.
US-TICKERS: Look for tickers in text - (NASDAQ:MSFT) (NYSE:FPOST)
and stick in the R* FipHdr fields.
R1:NYSE:FPOST
R2:NASDAQ:MSFT
up to 9 codes can be saved.
format is (Exchange) (colon) (Ticker)
This may be of value for Business Wire and PRnewswire services.
(Note for versions 158z - currently please do NOT specify BOTH BIZWIRE-HEADER
(or PRNEWSWIRE_HEADER) and US-TICKERS)
TICKER-TAG:<.> start, separater and end chrs for US-Tickers. default is (:)
for (NYSE:F)
REPLACE-TICKER: Replace a US-Ticker in text (as bracketed by (:) or <.>) with
this FipSeq
Several agencies - Reuters, DPA - send a Ticker code after the
company name : BP Plc <BP.LSE> Use this to replace the ticker with other
links
The ticker data is available as FipHdr Z1
eg REPLACE-TICKER:<a href="http://www.nasty.metabank.org/tickers=Z1>Z1</a>
EURO-TICKERS:
EURO-NAMES:
These are lists of FipHdr fields with tickers and names to search in text.
euro-tickers:EG,EL,EQ,EV
euro-names:EF,EK,EP,EU
us-tickers:
ticker-tag:<.>
replace-ticker:T9
fixed:T9 V3 (<a href="T0T4:T5">T5</a>) $o
BIZWIRE-HEADER: Interpret the Business Wire Header at the top of text.
( BW)(TEST-TEST-2)(ABC) This is a Test; Please Disregard This Test
These are put in B* FipHdr fields.
BU for Source 'BW'
BK for Keywords 'TEST-TEST-2'
B1 to B9 for Ticker codes 'ABC'
BH for Headline 'This is a Test; Please Disregard This Test'
The headline MAY be multiline, any end-of-lines are forced to a space.
BD: Keywords 'EUROPE'
BI: Industry Keywords 'MULTIMEDIA'
BF: for Supplier Source 'Ardvark Associates'
For RNS work,
BS for ShortName 'Abbey National PLC'
BC for FSA Category 'SUS'
BZ for Sequence Number 00000007
BR for Time of Receipt
PRNEWSWIRE-HEADER: Interpret the PR Newswire Header at the top of text.
These are put in B* FipHdr fields.
BR for Ticker codes 'ABC'
BH for Headline 'This is a Test; Please Disregard This Test'
The headline MAY be multiline, any end-of-lines are forced to a space.
BD: Subject Keywords 'ASI PDT'
BI: Industry Keywords 'ERN TLS'
AFPXML: Look for AFP tagged keywords - <org>DASSAULT AVIATION<orgid
idsr="ISIN" etc
and stick in the R* FipHdr fields.
R1 - First TickerCode - ISIN*FR0000121725*DASSAULT AVIATION
R2 - Second TickerCode - SICOVAM*12172*DASSAULT AVIATION
..
R9 - Ninth Ticker Code
format is (ORG) * (Code) * (Company Name)
There is a variant - AFPXML-link which also looks for the first occurrence
of the name of the company in the text and adds
BLOOMKWD: Look for Bloomberg-style keywords - NI EUROPE NI M&A NI FIP etc
and stick in the HX FipHdr field.
ANPA-HEADLINE:(FipHdrField)
Look for the first ANPA-style Bold line and put in this FipHdr field.
The Anpa bold is started with a '^'. In theory the headline is
only one line, but many suppliers make it over two or more. So
this flag will take data until a '^ @ < > ='
REUTERS-DATELINE:KM,MH,MN,B8,B7
Look for Reuters-style Dateline, Advisory and Bylines and put in
the 5 FipHdr fields noted ie L1 implies :
KM DateLine
MH ByLine
MN Advisory
B8 Correction Advisory
B7 Embargo Advisory
REUTERS-HEADLINE:KK,M9
REUTERS-SLUGLINE:MD,M8
REUTERS-LANGUAGE:KF
REUTERS-SIGNOFF:K9
REUTERS-SIGNOFF-SCRIPT:/fip/local/r_naomi_contacts.pl lang=QL fh=bu,bx
output=E3 data='E5'
Script for pulling apart the Reuters Signoff
REUTERS-PRIORITY:KH
Look in the KK FipHdr field for UPDATE or WRAPUP and create a Revision FipHdr
(M9)
M9 This Revision|Last|Next|Y/N
REUTERS-BUI-DATELINE:KM
Put ShiftOut and ShiftIn around the Dateline for BUI
REUTERS-CHK4BUIS:
Cleanup any extraneous Bui chrs in the text
REUTERS-TOTAL:15
Number of lines to look for a Reuters Dateline - default 15
Minimum is 1, max is 25
ALLOWSPACES: Leave all spaces in the data field for comparison.
Normally a field will have leading, trailing and multiple spaces stripped.
eg WITHOUT this option a field containing (pipes show start and end of data)
| This is a line with tabs and spaces in |
will be tested as :
|This is a line with tabs and spaces in|
SEND-SINGLE-COPY: If the same destination is specified more than once,
this option will only send a single copy of the file rather
than as many times as it is specified.
SEQNO-BALANCE-GROUP: (balance group)
SEQNO-BALANCE-QUEUE: (balance queue)
SEQNO-NAME: (Name for a Seqno)
This will save and balance a seqno to this balance group
If no QUEUE is specified, the default is spool/2balance
If no NAME is specified, the default is the name of the SU (or SR) FipHdr
field
CHECK-SEQNO: (fiphdr field)
Check this FipHdr field contains a value which increments
This is used to log any missing seqnos
CHECK-SEQNO-MIN: (minimum value)
CHECK-SEQNO-MAX: (maximum value)
Set bounds for the sequence number checking - defaults are 1 and 9999
SEQNO-RESETS-AT: (Hour:minute)
Sequence number is reset at this time - default is 0000 for midnight
FIPHDR-FOR-EXPAND: (FipHdr field)
The fiphdr field which needs to be expanded - no default
EXPAND-CODE: (code) (codes to add)
Add extra codes to a FipHdr field if the 'code' exists.
The 'code's are space separated.
There can be one or many (up to 300) 'expand-code' lines. eg:
fiphdr-for-expand:NO
expand-code:A C D E M O T U MTL GRO SOF OIL
expand-code:C C MTL GRO SOF
expand-code:D D T
CREATE-ZQ:
Use this to create a fake output sequence number in the FipHdr
field ZQ. If there is a number to this parameter it will be used
as the amount to increment - default is 2.
eg default create ZQ:3-4, the next ZQ:5-6
if create-ZQ:12, file will have ZQ:22-33 then ZQ:34-45 etc.
if create-ZQ:1, file will have ZQ:2-2 then ZQ:3-3 etc.
DEST-IS-FOLDER: (yes/no)
Destination names are actually folders under /fip/spool
default is DU are destinations in the sys/USERS file.
This overrides the -Q switch
Note all folder names are forced lowercase.
These folders are now speedy-fip if that option is running.
The DU is NOT appended in this case - so if you are putting into spool/2go,
add fiphdr fields
1 SU=SPTZ #DU:dev2sc+sptz2saves#SC:no
1 SU=SPTZ >2go
DONE-QUEUE: (Folder name in FipSeq)
Put the input file into this folder after processing
This overrides the -f Input switch.
BEFORE: (FipSeq)
AFTER: (FipSeq)
Text to conditionally add at the top or bottom of text
SCRIPT: Optional Script to run AFTER the file has been sent successfully.
eg script:/fip/local/FTP_NEXT_STAGE YI-YS
TRACKER-SCRIPT: Optional Tracking script - run AFTER script (if any)
default: none
MAX-ROUND-ROBIN: (no or 1-49) default: no
add a round-robin number to the output queue (same as -R)
ROUND-ROBIN-FIPHDR: (2 letter FipHdr code) default: none
both these parameters need to be specified
BIG-FILE-LIMIT: (size in bytes) default: none
BIG-FILE-FIPHDR: (2 letter FipHdr code) default: none
Flag YES or NO if the file is bigger than the limit.
both these parameters need to be specified
This is added as a FipHdr before routing and can be used to Route on.
FILE-TRACE: yes/no, + hdr + in + out
Trace all files using this parameter file default: no
FILE-TRACE-NAME: (FipSeq) unique name of this file default: SU-SN-HR
FILE-TRACE-LOG: (FipSeq) additional logging default: none
These can be individually turned on by FipHdr fields in the incoming data :
ZFILE_TRACE: ZFILE_TRACE_NAME: ZFILE_TRACE_LOG:
(notice the '_' if using in the FipHdr but '-' if parameters)
----
The rest of the file is a series of conditions or rules are applied to each
incoming file.
The format of these is :
tab (level_no) tab (condition)
tab (level_no) tab (condition) tab > (destination)
tab (level_no) tab (condition) tab (+opt additional DU)
tab (level_no) tab (condition) tab (>) DU1+DU2+DU3..+DUn)
tab (level_no) tab (condition) tab (+DU1+DU2+DU3..+DUn)
- Each rule is checked starting from the top of the file.
- where tab can be any combination of tabs and spaces.
- where the level_no is a number. If it is greater then 1, then this line is
only checked IF the preceeding level was matched.
- each rule is one and only one line.
- destinations - which MUST be entries in the tables/sys/USERS file - are
specified as :
>dest1
or >dest1+dest2
or +dest3
- Multiple destinations are separated by a '+'.
- If a line has NO destination, processing continues with the next line.
- If a line has one or more destinations starting with a '>', Searching
STOPS and the file sent to those destinations (without the '>' or '+').
NO more tests are processed.
- If a line has one or more destinations starting with a '+', Searching
CONTINUES while those destinations are saved for sending later (without the
'+').
- Syntax of a condition is :
(FipHdr Field) = (data)
eg: 1 p=3 >prio3
If the FipHdr field is a single letter it is from the Source Header (SH).
eg for standard wires
s is service designator
n is item number
p is priority
c is category
w is number of words
k is keyword or headline
etc
which has a format in the data file like :
SH::SEAA:N0436:P3:CI:W0260:RUSA /AFP-EC08:KUS-Asia-radio
If the FipHdr field is two letters, it is that actual field such as EK in
this example.
eg: 1 EK=88 >2FatLadies
Routing may also be on data in the text using the '$$'as the base. eg:
2 $$="New York Times" >nyt
For data, the condition is not EQUALS but CONTAINS.
ie The following will be true :
AbcdefThe New York Timeszzzzzzz
and New York Timesn
Actual type-of-file or the 'magic number' can be tested with '$f'
eg ; Is it a Word 2000 file ?
1 $f=xCFxD0xE0x11 >translateWord2000
If a Keyword file is being searched, Routing can be done on a match.
In this case '%%' is used and the name of the Keyword file.
(The %KEY parameter is ignored in this case).
In this example a file is held in 'tables/keyword/FOREIGN_LIST'.
1 c=INT ; for Category = International ..
2 %%=FOREIGN_LIST >mainforeign ; Main For queue for matches
2 c=* >junkforeign ; Rest to junk
- Condition is either '=' to signify equals or '#' meaning NOT equal. eg:
1 c=INT >allints
1 c#INT >noints
- in the case where several tests need to be made to determine a destination
then the preceeding lines have no destination
; If category starts 's' ..
1 c=s*
; send a COPY of all high priority files to the editor, and keep searching
2 p=1 +hied
; if Keyword is SOCCER - send to the soccer destination
2 k=SOCCER >soccer ; send race cards
... etc
There can be up to 100 levels if you really need.
- Use wild strings and chrs using (by default) '*' for strings and '?' for a
single chr.
(These can be changed using %WILD or %SINGLEWILD as above)
eg 7 k=*SOCCER*
will be true for XK fields such as :
SOCCER_RESULTS
These are the Soccer Headlines
And so for soccer
- Case - upper or lowercase - are NORMALLY ignored so that SOCCER' and
'soccer' are equal
unless specifically noted
either with the '%CONDCASE' flag for FipHdr fields
or with the '%CASE' flag for Text and Keyword matches.
- Note that if any spaces are embedded, double quotes should surround the
condition.
3 k="Liverpool Beatles"
- Testing for something or nothing !
The following test will always be true, no matter whether there
is a FipHdr L1, a blank L1 or an L1 with data.
1 L1=* >IamTrue
So to test if there IS something there :
1 L1#"" >thereIsSomut
Or nothing (ie no L1 or L1 with no data) :
1 L1="" >thereisNothing
- Special Case - you can also add extra FipHdr information on the end of a
destination
(or destinations)
1 p=1 >w4rns+rnsEd#JH:RNS-PRIORITY
or and example where you want to continue
1 p=1
2 c=HHH #JH:RNS-PRIORITY
2 c=HHH +w4rns+rnsEd
Note that the '#' sign marks the end of any destinations and the begining of
FipHdr.
The FipHdr bit may be in FipSeq :
1 p=1 >w4rns+rnsEd#JH:RNS-TIME-$d
For example :
; routing table for AP
; default destination is junk
%DU:junk
; chrset is 'anpa'
%SC:anpa
;
; If category starts 's' ..
1 c=s*
; level 2 - so for all category 's' etc, if the priority is '1' ..
; send to sptpri1 and do not look any further down the table.
2 p=1 >spthipri ; Priority 1 to sports hi pri
2 SU=AP >rac ; send race cards
1 c=t >tv ; tv pages
Note that if you use one of the $1 to $9 FipSeq functions to get the 1st to
9th line of text, any HASH is replaced by x9D.
Note also that a couple of extra builtin FipHdrs exist
%x - generate a 32bit hash of the first 64k of the data
Input parameters are (all optional) :
-1 : (minus one) filename for single-shot default: folder is spooled
-b : Balance ALL data files - balance queue default: (if -B specified)
spool/2balance
-B : Balance ALL data files - balance group default: not balanced
-c : default SC - source chrset default: ascii
-d : default DU - destination default: woops
-D : display contents of header fields as they are checked def.no
This is ONLY valid when using '-1' single shot and NO other
output is generated - ie no files, just the check.
eg iproute -1/fip/spool/test/abc.wire.9001.1234 -D
-f : done queue for the input file default: file is deleted
This can be overridden by the DONE-QUEUE parameter
-h : hostname to use def: name of this host
This is used for the XP flag for redundant feeds and should
only be necessary on systems connected to more than one
ethernet where the base hostname is NOT that to be used.
-G : WINNT/2k - grab the first file only default: no
-H : display contents of header fields in HTML def.no
This also requires the -D input switch
-i : input queue in /fip/spool default: 2brouted
-o : output queue in /fip/spool default: 2go
-l : (minus el) do NOT log default: log
-METAONLY : special case of iproute - pls see below. default: no
-m : name of the top MetaData file default: METADATA or (WinNT -
METADATA.fip)
-n : default SN - filename default: UNK000
-Q : all destinations are really folders NOT DU default: DU
This can be selectively turned off for a single parameter file
with DEST-IS-FOLDER:no
-r : Do NOT search for a routing if a DU exits. def: always use table
With this switch, if a DU field exists, it is used and the
routing table ignored. Normally the contents of the DU are
ignored and are overwritten with whatever destination
is selected via the table. This switch is for all incoming
services. If only one or two services need this, use the
%REUSEDU flag in the routing table for that service (see above).
-R : Running Round Robin on the output folder default: no
-s : default SU - source (usually the agency) default: noname
-S : force the parameter file to ALWAYS be this. default: FipHdr SR or SU is
used
-t : interval between scans of the queue default: 3 secs
-X : ignore SZ redundancy default: use SZ
This stops a copy of an incoming file being sent automatically
to the redundant system as in the SZ line of the routing file.
-W : 10 ths of a second - MUST wait if nothing processed default: 0
-8 : input is NOT speedy (valid on a speedy system only)
ie the input folder is NOT piped but the rest of the system is.
-9 : run in Speedy mode default: no
-v : print version number and exit
-- The above defaults are ONLY used if any defaults are NOT set in the
parameter file (%DU, %SC etc).
-- Note that if 'iproute' is used for files which are not directly from the
wires - perhaps files which have to pass through twice, A selection of existing
FipHdr fields are zapped before processing if present.
These are :
- XZ (resent file flag - do NOT archive)
- XS (do not send to other systems flag)
- CX and DC (for xchg)
- CQ change queue for xchg
- SZ (to denote redundant systems)
- XP and CP : source and destination FIP Hosts
This has no effect on FipHdr fields being ADDED at this stage (such as SZ being
added by the %SZ line) - only to delete any old FipHdr fields inserted by
another program.
The XZ flag stops 'ipwheel' from archiving the data.
The XS flag stops 'ipwheel' from sending to other systems.
CX and DC are usually added by 'ipwheel' to denote which XCHG to use for
'ipxchg'.
SZ is used to send a copy of the file over to the redundant system.
XP and CP are the source and destination FIP Hosts (ie system file arrived on
and where it was sent to)
-- If you do not want to log every time, then either use the parameter to STOP
ALL logging :
'NOLOG:'
or, just a single, incoming file, add the FipHdr field ROUTE_LOG_EVERY:(999)
or, in the parameter file, for a filter :
1 NN=grabeachsecond #ROUTE_LOG_EVERY:600
1 NN=grabeachsecond >delete
The value for ROUTE_LOG_EVERY is the number of seconds to skip before loggin
the next occurance OR a negetive numbre for No logging at all. In the example
above, the first file AFTER 600 secs/5mins is logged.
-- Why would you send a file through 'iproute' twice ??
Perhaps you have several publications. The main wire feeds come into one (pair
of) systems and most of the data is sorted and processed on that system for the
main publication.
But a smaller publication - or a specialist publication such as a Sports or
Financial magazine wants a subset of data with totoally different sorting
requirements.
It is easy to split a copy of all the wire files using a line like :
route file :
1 c=* +otherpub ; copy all to destination otherpub
USERS file
otherpub DP:smlpubhost
and send to the 2nd pubications servers when 'iproute' would resort properly.
USERS file on 'smlpubhost' :
otherpub DP:localhost DQ:2brouted SC:no DC:no
-- File tracing thru the Fip
There are a series of parameters common to serveral Fip Programs which will log
what is happening to all files, a service or a single file.
file-trace: (valid actions - yes/no, + hdr + in + out)
Actions are 'Yes' - log, 'No' - dont; if NOT NO, then in addition to logging,
we can save a copy of the FipHdr, Input file and/or Output
(delete is ignored in iproute))
file-trace-name: (UniqueId in FipSeq if not default - SU-SN-HR)
file-trace-log: (extra File Tracing logging in FipSeq)
The Hdr and files are left in /fip/log/file_trace (which should be purged in
the nightly maintenance zapfiplog)
These can be overridden by FipHdr fields in each file if a particular file -
or service needs tracing.
(notice the '_' if using in the FipHdr but '-' if parameters)
eg ZFILE_TRACE: hdr
ZFILE_TRACE_NAME: louislouis
ZFILE_TRACE_LOG: seqno.ZQ
---------------- META ONLY ------------------------
Normally 'iproute' is used as the routing stage for all wire files arriving on
a system.
There is a special case where a second copy of the program may be used to add
MetaData (in the form of FipHdr fields) only to any incoming files. This is
flagged by the input switch -METAONLY. In addition the input switches '-i' and
'-o' for input and output folders MUST be specified. This is to make sure that
the 2nd copy does NOT interfere with the main copy flow.
You MUST also define an input and output folders (-i and -o).
eg for the SYSTEM file :
meta iproute -METAONLY -m testmetadata.fip -i 2meta -o 2edsys
The parameter to the -m switch is a file in tables/metadata whose syntax is
different from the normal route file :
; comment
check-file:(name of file)
Example of a MetaData file
; Sports - look in /fip/tables/metadata/PSP.FIP
check-file:PSP.fip
There can be several check-file(s) which all reside in tables/metadata.
The syntax of the check-file is the normal 'ROUTE' as described above except
1. 3 new keywords
ADDHDR-IF-VALID:(FipHdr)
ADDHDR-IF-INVALID:(FipHdr)
These give the strings of FipHdr to add to the file if the tests are true or
false.
DEFAULT-RESULT: (valid/invalid)
The default result if all conditions are met is either 'valid' or 'notvalid'
or 'invalid' (the last 2 are the same !)
2. The only valid destinations are
stop-valid
Stop looking in this MetaData file as the result is valid
Continue with any further Metadata files specified in the top file
stop-invalid
Stop looking in this MetaData file as the result is invalid
Continue with any further Metadata files specified in the top file
stop-all-valid
Stop looking in this MetaData file as the result is valid
Do NOT continue looking any further
stop-all-invalid
Stop looking in this MetaData file as the result is valid
Do NOT continue looking any further
continue
Stop checking at this level in this MetaData file
Continue checking at the next (level - 1)
ie
1 SU=PA
2 XC=HHH
; if Prioriy is 1 or 2, drop down to the next level 2
3 XP=1 >continue
3 XP=2 >continue
3 XP=* >stop-invalid
2 XC=PPP
etc etc etc
Example of a check-file:
; ------------------------------------------------------
; if still valid at the end, add VT:PSC
%ADDHDR-IF-VALID:nNO:NO PSCn
; we have wilds
%WILD:*
; we assume that it is valid...
%default-result:Valid
%word-boundary:|
; ------------------------------------------------------
; If Status is NOT New or Priority 1 - always ignore
1 NV#New >stop-all-invalid
1 IP=1 >stop-all-invalid
; ------------------------------------------------------
; want Product Codes G or SPO or E (continue to next level 1 or end)
1 NO=*
2 NO=|G| >continue
2 NO=|SPO| >continue
2 NO=|E| >continue
2 NO=* >stop-invalid
; ------------------------------------------------------
; *** End ***
Version Control
;162f52 23oct07 added extra-fiphdr-script
;b 14jan08 allowed DE_DUPLICATE-DIFF:-1 for NO checking at all
;c 26apr08 keyword stripped ';' as comment - needed for 'å'
;d 31jul08 -b, -B added for dataBalGroup, balQue and SX
;e 27aug08 bugette with Reuters which are ONLY Xref/Signoff
;f1-4 18feb09 allow speedy with DEST-IS-FOLDER ;2 allow de-duplicate-time of
0=dont test time;3 NOLOGDELETE:
;4-5 3jun09 allow SD to tbe multiples SD:AP1,AP2,AP3
;6 22oct09 CHG - redid addhdr-script so it is the same as ipspool (with
E3/E4)
;7 16sep10 added DE-DUPLICATE-BALANCE
;8-9 24sep10 added ADDHDR-BEFORE-SCRIPT:..
;10 10oct10 ignore dedup if DU=logdel or delete !
;11 23oct10 bugette with DEDUP and no file
;12-14 3mar11 wrinkle with speedy and do NOT allow ';' to mean comments on
non-route lines and added %x for CRC
;15 10may11 bugette in chk_route (WIN only)
;16-18 11jul11 added -8 for NOT speedy in but speedy out and $1 back; added
HASH-IN-HDR as -$1 or-$t
;19-20 7dec11 added time for done queue duplicates
;21-26 30jan12 allow xml in Reuters-dateline and added HASH-KEY-STOP
;27 4jul12 added KEY-TEXT-MAX
;28-32 19jul12 do NOT add DU for DEST-IS-FOLDER and added ROUTE_LOG_EVERY as
FipHdr field
;33-35 20nov12 bugette of addhdr-before-route with no routing lines
;36 8mar13 added big-file-limit and max-round-robin and allow parsed routing
line : 1 c=* >2xmlRR
;37-38 7may13 wrinkle with ignoring the FipHdr in '1 c=* >abc+def#SC:ghi'
;39 6jun13 allow SH to change in ADDHDR, ADDHDR-BEFORE-ROUTING
;40 12sep13 added DE-DUPLICATION-IGNORE-DU
;41-43 1oct13 bugette in DE-DUP balance
;44 3jan14 added SHOW-DEDUP:yes/no
;45-48 16jan14 added SCRIPT and TRACKER-SCRIPT and refresh SN if it has
changed with ADDhdr etc
;49-52 added SEQNO_BALG/BALQ/NAME and REUTERS-SIGNOFF-SCRIPT ;52 added
file-trace
;161z2 03dec04 bugette with more than one kwd search (1 %%=KWDFILE >abc)
;a 15dec04 added -f done queue and DONE-QUEUE:
;b-h 17dec04 bugette with >a#SC:a and %SC:def (f-Roy xref)
;i-j 28apr05 added REPLACE-TICKER
;k-l 16may05 mod for mtime/nfs waits
;m-p 05jul05 use AFP <org> markup in text and buglette in size of 1st dedup
;q-r 03sep05 added euro-tickers and euro-names
;s-v 18jan06 for THROTTLE-MAX and THROTTLE-FIPHDR and DE-DUPLICATE-SOURCE
added BEFORE and AFTER
;w 1aug06 rename copied file if already exists
;x-y2 17nov06 bugette in test if singlewild '?' is first chr
;z2 08may07 added CREATE-ZQ
;160z 14nov03 tuning dedups (size is now JUST the data part)
;a 12dec03 added reuters-lines:
;b 12jan04 added anpa-headline:
;c-g 16feb04 cleanup HTML display and added expand-code
;h 01apr04 added RTR code for Advisory->Emb or Corr
;i 25apr04 added stop-all-valid/invalid for Metadata
;j-o 06jun04 bugette in AFPXML plus added REUTERS-BUI-DATELINE
;p-q 27jul04 no BUI in Dateline pls and single spaces in Dateline
;r-s 01sep04 -9 speedy (s for R-Adv)
;t-v 22sep04 added REUTERS-CHK4BUIS and tweak to Advisories
;w 04oct04 extra FipHdr lines are now added to FipHdr immediately
.. plus Pilcrow again
;x-y 03nov04 RTRS - case of Xref after a Signoff
;z 17nov04 added dest-is-folder: and default-dest and -w filewait
;159z 06jan03 minor mod to Reuters/Byline
;a 13jan03 RTRS - added UPDATE/WRAPUP revision
;b-g 30jan03 RTRS - added Genre
;h 04mar03 RTRS - Signoff
;i 25may03 RTRS - get/zap HASHes in MetaFields
;j 29may03 RTRS - add xB6-Pilcrow for extra lines in Advisories
;k-l 04jun03 RTRS - strip trailing spaces/lines from Advisory
and bugette for signoff
;m 29jun03 VON as a By plus poss correction advisory
;n 04jul03 added REUTERS-PILCROW
;o 23jul03 added PRNEWSWIRE-HEADER
;p 24jul03 redid $f for filetype - which was NOT working well
;q-v 04aug03 added de-duplicates
;w-y 06oct03 stripp and <Fip...> markers for Datelines
;z 31oct03 added timings
;158zi 28dec01 added TYPE-MARKERS and LOCALE and US-TICKERS
;a 22jan02 restrict size of market for US ticker to 20
;b 05feb02 added -1 for single shot
;c 12feb02 bugette with wildstrings
ie k=*sport* sometimes true erroneously
;d/e 27feb02 added $t and $1-9 for ADDHDR
;f/g 03apr02 added BIZWIRE-HEADER
;h 14apr02 ADDHDR can now contain FipSeq for other FipHdrs
;i 18apr02 widgets put in - fixed, partial, combie etc
;j/k/l 21apr02 added check-seqno, seqno-resets-at, check-seqno-max,
check-seqno-min
;m 22may02 allow CC!="abc" as well as CC#"abc" and defaultSU put back in
;n 28may02 bugette in word boundary code
;o 5jun02 bugette in ADDHDR - not allowing $1 etc
;p 13jun02 allow more than 2 source redundancy hosts (SZ).
;r 18jun02 added -D and -H for display only
;s/t/u 24jun02 added addhdr-before-routing
and -G to grab-first-file
;v 16sep02 added $f for checking the file magic number
;w 17oct02 use calculated DU even if there is an incoming DU
;x/y 26nov02 added %REUTERS-DATELINE
;zi 11dec02 added more BIZWIRE kwds at end
;157b 12nov01 added -METAONLY/-m and word-boundary
;156p 16jul99 added %SEND-SINGLE-COPY:
;b zap FipHdr DP if in the incoming file.
;c 02jul00 added %AFPXML
;d 07aug00 bugette with multiple DUs which blow the filename.
;e 06nov00 added ExtraFipHdr info on the end of a dest.
;f 22nov00 bugette - when using duplicates (SD), sometimes overwrote original
file !
;g 02apr01 bugette - SendSingleCopy sometimes duplicated the 1st DU
;h/j 21may01 redid ExtrafipHdr
;k/l/m 25aug01 strip NULs from data part of file for comparison
;o 05sep01 redid ability to run external program
;p 28sep01 AFP changed their XML .... for %AFPXML
(copyright) 2014 and previous years FingerPost Ltd.