iproute IPROUTE may be used in three areas : - normal use as the routing stage for all wire files arriving on a system and passing the files to 'ipwheel' to distribute. Two lessers uses are : - applying extra Metadata according selections of FipHdr fields This is triggered by the '-METAONLY' input switch - moving incoming files to folders in /fip/spool depending on FipHdr fields This is triggered by the 'DEST-IS-FOLDER:' keyword in a single parameter file OR the -Q input switch. Normal routing is : IPROUTE reads a file generated by one of the WIRE programs and attempts to decide where to route it. This it does by looking at the FIP Header fields DU (destination or end user), SU (logical source or user i.e. wire name) or SR for the alternate routing filename, SC (source character set) and SH (source header). A lookup table with the name of SU header field (in uppercase) (or SR if it is optionally there) is inspected to find where the file should be sent. This parameter file can be forced to be always the same if the -S input switch is used to specify it. If the file has a SD (duplicate wire) then the file is copied (well, linked) with the new name. Optionally if the file has a DU then no further processing needs to be done. This is flagged by the -r or %REUSEDU flags. This program does NOT send it - it merely creates a link in the output queue 2GO in the form : #SU:srcuser#DU:destuser#SN:filename#SC:chrset_date_time[_optionals] On input you can change the default settings of DU, SU, SN and SC. All parameters are optional - although there MUST be routing file for each incoming feed even if it contains a single comment line. Normally the minimum file will contain : - comments (naturally!) - a default destination (using the default-dest: or %DU keyword) - a source character set (using the default-chrset or %SC keyword) Other normal keywords are : - a number of routing lines - redun-hosts or %SZ field if there is a redundant feed - %LOG to display the headline or other meaningful fields. Note that the '%' in front of each keyword in the parameter file is no longer required. So each of these keywords may be specified as either: %HDRKEY:mykeywordfile or hdrkey:mykeywordfile The syntax of the routing file is : ; comment line (comments are also valid on the rest of line). DU: (default destination setting if not matched) eg: %DU:logdel which will send all unrouted copy to the archive and no further eg: %DU:apspo+spoprt+chrisq which will send unrouted copy to the three destinations SC: (set source character set) SZ: (primary,secondary host for redundant wires) SZ:fip1, gate2 Note BOTH systems must be specified, neither need be the system that the file first came in on KEY: (keyword file containing keywords to search for) %KEY:DAILY_WORDS A file in tables/keyword/DAILY_WORDS exists with a list of keywords to search for on separate lines. If such a Keyword is found it is added to the FipHdr as the HK field. A match is made is a string is found in the stream of data. To look only for words, use the PIPE, '|' to specify the begining/end. ie mac will find 'macintosh', 'machine', 'applemac' |mac will find the first two only mac| will find 'applemac' only |mac| will find 'mac' only A word is one that starts/ends with a space or ends with ',.;?!:)([]{}'. Apostrophes are ignored. KEY-TEXT-MAX: (number or FipSeq) Maximum number of chrs to check in the Keyword file Use this to trim the search to the first 400 chrs for example default: up to 64k HDRKEY: (Also look in these fields for the Keywords) HDRKEY:XK AB Look in the K field of the SH (source header) plus the AB field for the same keywords as the text. The syntax is denote each field with a preceeding backslash plus a space to separate : "AB (space) QQ (space) FQ" CASE: (keywords and text searches are case SENSITIVE) Normally words specified in the Keyword file are case insensitive. So an entry like 'Humbug' will find : HUMBUG, humbug, hUmBuG Set this flag to force the program to only search for an EXACT match - 'Humbug' in this example. CONDCASE: (Condition lines are case SENSITIVE) The test in a condition line is normally caseINsensitive. So the following line will trigger on any case like 'TV', 'tv', 'tV' or 'Tv' : 1 c=tv >qpstv ; send tv pages to qps Almost all wire services send codes which are case-insensitive. AFP's EAA service however has category codes like 'FsI' or 'Sss' where 's' and 'S' have different meanings. Use '%CONDCASE' to flag this. LOG: (Custom logging : text, unix and octal chrs, FIP hdr fields) Each file though is logged as a one line entry in the Item Log (log/ALL) The default format is : date time iproute !i Source(SU) Dest(DU) FileName(SN) Keyword(HK) eg: Sat Jan 22 15:15:55 iproute !i : AP wsport APN0174 eggs which is represented by %LOG:SUtDUsSNsHK NOLOGCTL: Strip all control chrs - such as BELL - from the log file default is to display all chrs NOLOG: Do NOT log each item through at all NOLOGDELETE: Do NOT log any item which is sent to 'delete' WILD: (Change the wild Chr from '*' to this chr) SINGLEWILD: (Change the single wild Chr from '?' to this chr) wild and single wild chrs may be used in any condition below to signify any chr or any string of chrs eg: %WILD:% 1 c=Z%L will prove true for any category code (XC field of the Source Header (SH)) that begins Z and ends L.` WORD-BOUNDARY: (Chr in FipSeq) This is a single punctuation chr which can be used to mean the START or END of a Word or field eg %WORD-BOUNDARY:| 1 WT=|ABC* will look for the contents of the WT Fiphdr field STARTING ABC 1 WT=*XYZ| will look for the contents of the WT Fiphdr field ENDING XYZ The following chrs may NOT be used + > " # _ - = There is no default and so word/field boundaries are NOT tracked normally. TEXT: (no of chrs to look to routing information) This defaults to 8k and is 64k maximum. MAX-DOLLAR-T: (no of chrs for $t - first bit of text) This can be up to a maximum of 2000 chrs - default is 80. HASH-IN-HDR: (FipSeq) or HASH-IN-DOLLAR-T: (FipSeq) or HASH-IN-DOLLAR-1: (FipSeq) Hash character ('#') to be used in $t and $1 to $9. (Remember '#' is an end of FipHdr character, so it needs to be mapped out of the way) HASH-KEY-IGNORE-TOP: (number) Number of lines at the top of the file to ignore BEFORE starting the hash Use this if a date/time is on the first line or so. default: none HASH-KEY-START: (FipSeq) Start generating the hash-key (%x) AFTER this character/string. default:none - ie no start string meaning ALL data from the start. HASH-KEY-STOP: (FipSeq) Stop generating the hash-key (%x) on this character/string. default:none - ie no end string meaning data continues to the end ; stop hashing at the ^C/end-of-text as the datetime is ALWAYS different HASH-KEY-STOP:03 HASH-KEY-DATA: (FipSeq) Use this string to generate the hash default use the data in the file eg hash-key-data:XS-XN-XC-XP-XK TYPE-MARKERS: (2 punctuation chrs to be used as markers) For Tests, types-of-characters may also be used. There MUST be two, non-space characters used for the start and end of the types which must be punctuation and NOT the same you have used for wither wilds or word boundarys. eg look for (space) (3 uppercase letters) (4 numbers) (spc) %TYPE-MARKERS:<> 1 XK=*<SUUUNNNNS>* +someDestination Note the start and end markers may be the same chr. You can also mix exact strings with types : eg look for 'ABC' (4 numbers) (spc) %TYPE-MARKERS:| 1 XK=*ABC<NNNNS>* >weHaveABCfolks DE-DUPLICATE-DEST: (DU) Destination (DU) in sys/USERS to send 2nd (and other) duplicates Strip re-route duplicate files which do NOT have the same ItemNumber/FileName. default is - no deduplicate. A common de-dup destination might be 'logdel' to log a copy only. The incoming files list is in log/route/DEDUP_(SU) DE-DUPLICATE-HEAD: (FipSeq field for the Headline) (this depends on DE-DUPLICATE-DEST) THis can be a combie or some other sequence of FipSeq to generate a unique string. eg ; dedup on name - catcode - priority - take number - taketime fixed:Q1 SN-NC-NP-NQ-NT de-duplicate-head:Q1 default is 'XK' To use the hash key '%x' in a De-Dup head, unfortunately you cannot add it in a 'fixed:' So add it to ADDHDR-BEFORE-ROUTING eg ADDHDR-BEFORE-ROUTING:Q0:%x fixed:Q1 XK-XC-Q0 DE-DUPLICATE-HEAD:Q1 DE-DUPLICATE-DIFF: (number of bytes) This allows for slight differences in the text of the files. This fudge factir is the allowable difference. So DE-DUPLICATE-DIFF:100 will catch files that have the same Headline (and Source) and the file length is within 100 plus/minus. ie if a file is 987 bytes long and the original was 1086, it is de-duped. BUT if it is 1187, it will be considered a new file and allowed thru. If you do NOT want to dedup on size, set the value to -1 eg: de-duplicate-diff:-1 default: 0 for zero bytes difference. DE-DUPLICATE-MAX: (number of Items) (this depends on DE-DUPLICATE-DEST) Total number of entries to be tracked in the time period default: 100 As duplicate files normally follow each other, generally this total dows not have to be very big. DE-DUPLICATE-TIME: (number of seconds) Time period for duplicates in seconds. default: 3600 for 1 hour DE-DUPLICATE-SOURCE: (name) This is the name of the 'collection' of files to check for duplicates and/or throttling. It is normally the Source (SU) but can be FipSeq. This is need to define the uniqueness of the data. It is used to create the name of the log file in log/route. DE-DUPLICATE-BALANCE: (balance group) default: none Name of a Group in the sys/BALANCE file to syncronise the DEDUP file across systems Note there will normally be up to a second delay in balancing, so duplicate files arriving on both system within the baalnce time may be un-duped. The new items are APPENDED to the DEDUP file on the remotes. As cleanup (iw trimming according to time) is only done on a new file on that system, remote systems which never receive files (ie pure DR sites) will need an occasional full copy - or nightly maintenance should be used to zap the files in log/route on those systems. DE-DUPLICATE-IGNORE-DU: (yes/no) Ignore the Destination/DU for this traffic default: do NOT ignore Use this where the two files may have different DUs because of round-robin THROTTLE-MAX: (number of Items in FipSeq) Only allow this number of files through for the time period. Use this where you not only want to deduplicate but also restrict data. Note that throttling also uses de-duplications. In particular, DD-DEST, HEAD, MAX and/or TIME must be specified. The parameter can either be a number OR a FipSeq Fiphdr field : In this example the max is normally 10 files during the period unless the FipHdr SU is 'AFX' which allows 99 files. THROTTLE-MAX:Q3 option:Q1 SU,,,,AFX fixed:Q2 Q199$o combie:Q3 Q2,10 default: no max to allow ALL files through THROTTLE-FIPHDR: (FipSeq) Extra FipHdr information to add to any throttled copy default: none This could be a banner or some extra Meta data for the next process. LOCALE: (locale for this source) Use a different 'locale' ( look at the man pages for locale) Most computers are set to US English and never changed. Use this parameter to customise any date/time or Chr translations The parameter MUST be a valid locale on your system! To find out what valid locale exist, type 'locale -a' on unix. Eg ; Set for for Brazil, portugese locale:pt_BR ADDHDR: Extra, optional, fixed FipHdr information to added AFTER any routing and before the file is passed on. Often this is add extra fixed fields or to change the unique name for redundancy. eg: %ADDHDR:QQ:APTEST#ZH:SNXV$T#QZ:temp ADDHDR-BEFORE-ROUTING: Extra, optional, fixed FipHdr information added BEFORE any routing is checked. ; map the extra fields to LCC ; eg of incoming : KI:LCC MRG LCCX EMRG ; we want KI:LCC MRG EMRG replace:Q1 KI LCCX=LCC LCCY=LCC LCCZ=LCC unique:Q2 Q1 ADDHDR-BEFORE-ROUTING :XX:oldKI=KImKI:Q2n If using any FipSeq - like in the example -, they MUST be specified ABOVE/BEFORE the addhdr-before-routing line in the parameter file. Any specified below will be ignored. ADDHDR-BEFORE-SCRIPT: Extra, optional FipHdr information generated by an external program or script ADDHDR-SCRIPT: Extra, optional FipHdr information generated by an external program or script ADDHDR-SCRIPT:/fip/local/find_iim.pl E4 > E3 E3 hold the name of a TMP file to create that will be read for the list. E4 hold the name of a TMP file that holds the contents of the incoming file this is a copy of the incoming file but with a safe filename. ** if you are going to use the FipHdr fields from the script for routing or logging, pls use the '-BEFORE-' versions This is a 'must use' if using the Fip version of rdjpgcom. eg ADDHDR-BEFORE-SCRIPT:/fip/3rdparty/rdjpgcom -fiphdr E4 > E3 NOTE - that the script uses Forward Slashes for the file names. So Windows uses may need to convert in the script. REUSEDU: If the DU exists for this service, use it and ignore the routing table. Normally if a DU exists, it is ignored and the tables searched. US-TICKERS: Look for tickers in text - (NASDAQ:MSFT) (NYSE:FPOST) and stick in the R* FipHdr fields. R1:NYSE:FPOST R2:NASDAQ:MSFT up to 9 codes can be saved. format is (Exchange) (colon) (Ticker) This may be of value for Business Wire and PRnewswire services. (Note for versions 158z - currently please do NOT specify BOTH BIZWIRE-HEADER (or PRNEWSWIRE_HEADER) and US-TICKERS) TICKER-TAG:<.> start, separater and end chrs for US-Tickers. default is (:) for (NYSE:F) REPLACE-TICKER: Replace a US-Ticker in text (as bracketed by (:) or <.>) with this FipSeq Several agencies - Reuters, DPA - send a Ticker code after the company name : BP Plc <BP.LSE> Use this to replace the ticker with other links The ticker data is available as FipHdr Z1 eg REPLACE-TICKER:<a href="http://www.nasty.metabank.org/tickers=Z1>Z1</a> EURO-TICKERS: EURO-NAMES: These are lists of FipHdr fields with tickers and names to search in text. euro-tickers:EG,EL,EQ,EV euro-names:EF,EK,EP,EU us-tickers: ticker-tag:<.> replace-ticker:T9 fixed:T9 V3 (<a href="T0T4:T5">T5</a>) $o BIZWIRE-HEADER: Interpret the Business Wire Header at the top of text. ( BW)(TEST-TEST-2)(ABC) This is a Test; Please Disregard This Test These are put in B* FipHdr fields. BU for Source 'BW' BK for Keywords 'TEST-TEST-2' B1 to B9 for Ticker codes 'ABC' BH for Headline 'This is a Test; Please Disregard This Test' The headline MAY be multiline, any end-of-lines are forced to a space. BD: Keywords 'EUROPE' BI: Industry Keywords 'MULTIMEDIA' BF: for Supplier Source 'Ardvark Associates' For RNS work, BS for ShortName 'Abbey National PLC' BC for FSA Category 'SUS' BZ for Sequence Number 00000007 BR for Time of Receipt PRNEWSWIRE-HEADER: Interpret the PR Newswire Header at the top of text. These are put in B* FipHdr fields. BR for Ticker codes 'ABC' BH for Headline 'This is a Test; Please Disregard This Test' The headline MAY be multiline, any end-of-lines are forced to a space. BD: Subject Keywords 'ASI PDT' BI: Industry Keywords 'ERN TLS' AFPXML: Look for AFP tagged keywords - <org>DASSAULT AVIATION<orgid idsr="ISIN" etc and stick in the R* FipHdr fields. R1 - First TickerCode - ISIN*FR0000121725*DASSAULT AVIATION R2 - Second TickerCode - SICOVAM*12172*DASSAULT AVIATION .. R9 - Ninth Ticker Code format is (ORG) * (Code) * (Company Name) There is a variant - AFPXML-link which also looks for the first occurrence of the name of the company in the text and adds BLOOMKWD: Look for Bloomberg-style keywords - NI EUROPE NI M&A NI FIP etc and stick in the HX FipHdr field. ANPA-HEADLINE:(FipHdrField) Look for the first ANPA-style Bold line and put in this FipHdr field. The Anpa bold is started with a '^'. In theory the headline is only one line, but many suppliers make it over two or more. So this flag will take data until a '^ @ < > =' REUTERS-DATELINE:KM,MH,MN,B8,B7 Look for Reuters-style Dateline, Advisory and Bylines and put in the 5 FipHdr fields noted ie L1 implies : KM DateLine MH ByLine MN Advisory B8 Correction Advisory B7 Embargo Advisory REUTERS-HEADLINE:KK,M9 REUTERS-SLUGLINE:MD,M8 REUTERS-LANGUAGE:KF REUTERS-SIGNOFF:K9 REUTERS-SIGNOFF-SCRIPT:/fip/local/r_naomi_contacts.pl lang=QL fh=bu,bx output=E3 data='E5' Script for pulling apart the Reuters Signoff REUTERS-PRIORITY:KH Look in the KK FipHdr field for UPDATE or WRAPUP and create a Revision FipHdr (M9) M9 This Revision|Last|Next|Y/N REUTERS-BUI-DATELINE:KM Put ShiftOut and ShiftIn around the Dateline for BUI REUTERS-CHK4BUIS: Cleanup any extraneous Bui chrs in the text REUTERS-TOTAL:15 Number of lines to look for a Reuters Dateline - default 15 Minimum is 1, max is 25 ALLOWSPACES: Leave all spaces in the data field for comparison. Normally a field will have leading, trailing and multiple spaces stripped. eg WITHOUT this option a field containing (pipes show start and end of data) | This is a line with tabs and spaces in | will be tested as : |This is a line with tabs and spaces in| SEND-SINGLE-COPY: If the same destination is specified more than once, this option will only send a single copy of the file rather than as many times as it is specified. SEQNO-BALANCE-GROUP: (balance group) SEQNO-BALANCE-QUEUE: (balance queue) SEQNO-NAME: (Name for a Seqno) This will save and balance a seqno to this balance group If no QUEUE is specified, the default is spool/2balance If no NAME is specified, the default is the name of the SU (or SR) FipHdr field CHECK-SEQNO: (fiphdr field) Check this FipHdr field contains a value which increments This is used to log any missing seqnos CHECK-SEQNO-MIN: (minimum value) CHECK-SEQNO-MAX: (maximum value) Set bounds for the sequence number checking - defaults are 1 and 9999 SEQNO-RESETS-AT: (Hour:minute) Sequence number is reset at this time - default is 0000 for midnight FIPHDR-FOR-EXPAND: (FipHdr field) The fiphdr field which needs to be expanded - no default EXPAND-CODE: (code) (codes to add) Add extra codes to a FipHdr field if the 'code' exists. The 'code's are space separated. There can be one or many (up to 300) 'expand-code' lines. eg: fiphdr-for-expand:NO expand-code:A C D E M O T U MTL GRO SOF OIL expand-code:C C MTL GRO SOF expand-code:D D T CREATE-ZQ: Use this to create a fake output sequence number in the FipHdr field ZQ. If there is a number to this parameter it will be used as the amount to increment - default is 2. eg default create ZQ:3-4, the next ZQ:5-6 if create-ZQ:12, file will have ZQ:22-33 then ZQ:34-45 etc. if create-ZQ:1, file will have ZQ:2-2 then ZQ:3-3 etc. DEST-IS-FOLDER: (yes/no) Destination names are actually folders under /fip/spool default is DU are destinations in the sys/USERS file. This overrides the -Q switch Note all folder names are forced lowercase. These folders are now speedy-fip if that option is running. The DU is NOT appended in this case - so if you are putting into spool/2go, add fiphdr fields 1 SU=SPTZ #DU:dev2sc+sptz2saves#SC:no 1 SU=SPTZ >2go DONE-QUEUE: (Folder name in FipSeq) Put the input file into this folder after processing This overrides the -f Input switch. BEFORE: (FipSeq) AFTER: (FipSeq) Text to conditionally add at the top or bottom of text SCRIPT: Optional Script to run AFTER the file has been sent successfully. eg script:/fip/local/FTP_NEXT_STAGE YI-YS TRACKER-SCRIPT: Optional Tracking script - run AFTER script (if any) default: none MAX-ROUND-ROBIN: (no or 1-49) default: no add a round-robin number to the output queue (same as -R) ROUND-ROBIN-FIPHDR: (2 letter FipHdr code) default: none both these parameters need to be specified BIG-FILE-LIMIT: (size in bytes) default: none BIG-FILE-FIPHDR: (2 letter FipHdr code) default: none Flag YES or NO if the file is bigger than the limit. both these parameters need to be specified This is added as a FipHdr before routing and can be used to Route on. FILE-TRACE: yes/no, + hdr + in + out Trace all files using this parameter file default: no FILE-TRACE-NAME: (FipSeq) unique name of this file default: SU-SN-HR FILE-TRACE-LOG: (FipSeq) additional logging default: none These can be individually turned on by FipHdr fields in the incoming data : ZFILE_TRACE: ZFILE_TRACE_NAME: ZFILE_TRACE_LOG: (notice the '_' if using in the FipHdr but '-' if parameters) ---- The rest of the file is a series of conditions or rules are applied to each incoming file. The format of these is : tab (level_no) tab (condition) tab (level_no) tab (condition) tab > (destination) tab (level_no) tab (condition) tab (+opt additional DU) tab (level_no) tab (condition) tab (>) DU1+DU2+DU3..+DUn) tab (level_no) tab (condition) tab (+DU1+DU2+DU3..+DUn) - Each rule is checked starting from the top of the file. - where tab can be any combination of tabs and spaces. - where the level_no is a number. If it is greater then 1, then this line is only checked IF the preceeding level was matched. - each rule is one and only one line. - destinations - which MUST be entries in the tables/sys/USERS file - are specified as : >dest1 or >dest1+dest2 or +dest3 - Multiple destinations are separated by a '+'. - If a line has NO destination, processing continues with the next line. - If a line has one or more destinations starting with a '>', Searching STOPS and the file sent to those destinations (without the '>' or '+'). NO more tests are processed. - If a line has one or more destinations starting with a '+', Searching CONTINUES while those destinations are saved for sending later (without the '+'). - Syntax of a condition is : (FipHdr Field) = (data) eg: 1 p=3 >prio3 If the FipHdr field is a single letter it is from the Source Header (SH). eg for standard wires s is service designator n is item number p is priority c is category w is number of words k is keyword or headline etc which has a format in the data file like : SH::SEAA:N0436:P3:CI:W0260:RUSA /AFP-EC08:KUS-Asia-radio If the FipHdr field is two letters, it is that actual field such as EK in this example. eg: 1 EK=88 >2FatLadies Routing may also be on data in the text using the '$$'as the base. eg: 2 $$="New York Times" >nyt For data, the condition is not EQUALS but CONTAINS. ie The following will be true : AbcdefThe New York Timeszzzzzzz and New York Timesn Actual type-of-file or the 'magic number' can be tested with '$f' eg ; Is it a Word 2000 file ? 1 $f=xCFxD0xE0x11 >translateWord2000 If a Keyword file is being searched, Routing can be done on a match. In this case '%%' is used and the name of the Keyword file. (The %KEY parameter is ignored in this case). In this example a file is held in 'tables/keyword/FOREIGN_LIST'. 1 c=INT ; for Category = International .. 2 %%=FOREIGN_LIST >mainforeign ; Main For queue for matches 2 c=* >junkforeign ; Rest to junk - Condition is either '=' to signify equals or '#' meaning NOT equal. eg: 1 c=INT >allints 1 c#INT >noints - in the case where several tests need to be made to determine a destination then the preceeding lines have no destination ; If category starts 's' .. 1 c=s* ; send a COPY of all high priority files to the editor, and keep searching 2 p=1 +hied ; if Keyword is SOCCER - send to the soccer destination 2 k=SOCCER >soccer ; send race cards ... etc There can be up to 100 levels if you really need. - Use wild strings and chrs using (by default) '*' for strings and '?' for a single chr. (These can be changed using %WILD or %SINGLEWILD as above) eg 7 k=*SOCCER* will be true for XK fields such as : SOCCER_RESULTS These are the Soccer Headlines And so for soccer - Case - upper or lowercase - are NORMALLY ignored so that SOCCER' and 'soccer' are equal unless specifically noted either with the '%CONDCASE' flag for FipHdr fields or with the '%CASE' flag for Text and Keyword matches. - Note that if any spaces are embedded, double quotes should surround the condition. 3 k="Liverpool Beatles" - Testing for something or nothing ! The following test will always be true, no matter whether there is a FipHdr L1, a blank L1 or an L1 with data. 1 L1=* >IamTrue So to test if there IS something there : 1 L1#"" >thereIsSomut Or nothing (ie no L1 or L1 with no data) : 1 L1="" >thereisNothing - Special Case - you can also add extra FipHdr information on the end of a destination (or destinations) 1 p=1 >w4rns+rnsEd#JH:RNS-PRIORITY or and example where you want to continue 1 p=1 2 c=HHH #JH:RNS-PRIORITY 2 c=HHH +w4rns+rnsEd Note that the '#' sign marks the end of any destinations and the begining of FipHdr. The FipHdr bit may be in FipSeq : 1 p=1 >w4rns+rnsEd#JH:RNS-TIME-$d For example : ; routing table for AP ; default destination is junk %DU:junk ; chrset is 'anpa' %SC:anpa ; ; If category starts 's' .. 1 c=s* ; level 2 - so for all category 's' etc, if the priority is '1' .. ; send to sptpri1 and do not look any further down the table. 2 p=1 >spthipri ; Priority 1 to sports hi pri 2 SU=AP >rac ; send race cards 1 c=t >tv ; tv pages Note that if you use one of the $1 to $9 FipSeq functions to get the 1st to 9th line of text, any HASH is replaced by x9D. Note also that a couple of extra builtin FipHdrs exist %x - generate a 32bit hash of the first 64k of the data Input parameters are (all optional) : -1 : (minus one) filename for single-shot default: folder is spooled -b : Balance ALL data files - balance queue default: (if -B specified) spool/2balance -B : Balance ALL data files - balance group default: not balanced -c : default SC - source chrset default: ascii -d : default DU - destination default: woops -D : display contents of header fields as they are checked def.no This is ONLY valid when using '-1' single shot and NO other output is generated - ie no files, just the check. eg iproute -1/fip/spool/test/abc.wire.9001.1234 -D -f : done queue for the input file default: file is deleted This can be overridden by the DONE-QUEUE parameter -h : hostname to use def: name of this host This is used for the XP flag for redundant feeds and should only be necessary on systems connected to more than one ethernet where the base hostname is NOT that to be used. -G : WINNT/2k - grab the first file only default: no -H : display contents of header fields in HTML def.no This also requires the -D input switch -i : input queue in /fip/spool default: 2brouted -o : output queue in /fip/spool default: 2go -l : (minus el) do NOT log default: log -METAONLY : special case of iproute - pls see below. default: no -m : name of the top MetaData file default: METADATA or (WinNT - METADATA.fip) -n : default SN - filename default: UNK000 -Q : all destinations are really folders NOT DU default: DU This can be selectively turned off for a single parameter file with DEST-IS-FOLDER:no -r : Do NOT search for a routing if a DU exits. def: always use table With this switch, if a DU field exists, it is used and the routing table ignored. Normally the contents of the DU are ignored and are overwritten with whatever destination is selected via the table. This switch is for all incoming services. If only one or two services need this, use the %REUSEDU flag in the routing table for that service (see above). -R : Running Round Robin on the output folder default: no -s : default SU - source (usually the agency) default: noname -S : force the parameter file to ALWAYS be this. default: FipHdr SR or SU is used -t : interval between scans of the queue default: 3 secs -X : ignore SZ redundancy default: use SZ This stops a copy of an incoming file being sent automatically to the redundant system as in the SZ line of the routing file. -W : 10 ths of a second - MUST wait if nothing processed default: 0 -8 : input is NOT speedy (valid on a speedy system only) ie the input folder is NOT piped but the rest of the system is. -9 : run in Speedy mode default: no -v : print version number and exit -- The above defaults are ONLY used if any defaults are NOT set in the parameter file (%DU, %SC etc). -- Note that if 'iproute' is used for files which are not directly from the wires - perhaps files which have to pass through twice, A selection of existing FipHdr fields are zapped before processing if present. These are : - XZ (resent file flag - do NOT archive) - XS (do not send to other systems flag) - CX and DC (for xchg) - CQ change queue for xchg - SZ (to denote redundant systems) - XP and CP : source and destination FIP Hosts This has no effect on FipHdr fields being ADDED at this stage (such as SZ being added by the %SZ line) - only to delete any old FipHdr fields inserted by another program. The XZ flag stops 'ipwheel' from archiving the data. The XS flag stops 'ipwheel' from sending to other systems. CX and DC are usually added by 'ipwheel' to denote which XCHG to use for 'ipxchg'. SZ is used to send a copy of the file over to the redundant system. XP and CP are the source and destination FIP Hosts (ie system file arrived on and where it was sent to) -- If you do not want to log every time, then either use the parameter to STOP ALL logging : 'NOLOG:' or, just a single, incoming file, add the FipHdr field ROUTE_LOG_EVERY:(999) or, in the parameter file, for a filter : 1 NN=grabeachsecond #ROUTE_LOG_EVERY:600 1 NN=grabeachsecond >delete The value for ROUTE_LOG_EVERY is the number of seconds to skip before loggin the next occurance OR a negetive numbre for No logging at all. In the example above, the first file AFTER 600 secs/5mins is logged. -- Why would you send a file through 'iproute' twice ?? Perhaps you have several publications. The main wire feeds come into one (pair of) systems and most of the data is sorted and processed on that system for the main publication. But a smaller publication - or a specialist publication such as a Sports or Financial magazine wants a subset of data with totoally different sorting requirements. It is easy to split a copy of all the wire files using a line like : route file : 1 c=* +otherpub ; copy all to destination otherpub USERS file otherpub DP:smlpubhost and send to the 2nd pubications servers when 'iproute' would resort properly. USERS file on 'smlpubhost' : otherpub DP:localhost DQ:2brouted SC:no DC:no -- File tracing thru the Fip There are a series of parameters common to serveral Fip Programs which will log what is happening to all files, a service or a single file. file-trace: (valid actions - yes/no, + hdr + in + out) Actions are 'Yes' - log, 'No' - dont; if NOT NO, then in addition to logging, we can save a copy of the FipHdr, Input file and/or Output (delete is ignored in iproute)) file-trace-name: (UniqueId in FipSeq if not default - SU-SN-HR) file-trace-log: (extra File Tracing logging in FipSeq) The Hdr and files are left in /fip/log/file_trace (which should be purged in the nightly maintenance zapfiplog) These can be overridden by FipHdr fields in each file if a particular file - or service needs tracing. (notice the '_' if using in the FipHdr but '-' if parameters) eg ZFILE_TRACE: hdr ZFILE_TRACE_NAME: louislouis ZFILE_TRACE_LOG: seqno.ZQ ---------------- META ONLY ------------------------ Normally 'iproute' is used as the routing stage for all wire files arriving on a system. There is a special case where a second copy of the program may be used to add MetaData (in the form of FipHdr fields) only to any incoming files. This is flagged by the input switch -METAONLY. In addition the input switches '-i' and '-o' for input and output folders MUST be specified. This is to make sure that the 2nd copy does NOT interfere with the main copy flow. You MUST also define an input and output folders (-i and -o). eg for the SYSTEM file : meta iproute -METAONLY -m testmetadata.fip -i 2meta -o 2edsys The parameter to the -m switch is a file in tables/metadata whose syntax is different from the normal route file : ; comment check-file:(name of file) Example of a MetaData file ; Sports - look in /fip/tables/metadata/PSP.FIP check-file:PSP.fip There can be several check-file(s) which all reside in tables/metadata. The syntax of the check-file is the normal 'ROUTE' as described above except 1. 3 new keywords ADDHDR-IF-VALID:(FipHdr) ADDHDR-IF-INVALID:(FipHdr) These give the strings of FipHdr to add to the file if the tests are true or false. DEFAULT-RESULT: (valid/invalid) The default result if all conditions are met is either 'valid' or 'notvalid' or 'invalid' (the last 2 are the same !) 2. The only valid destinations are stop-valid Stop looking in this MetaData file as the result is valid Continue with any further Metadata files specified in the top file stop-invalid Stop looking in this MetaData file as the result is invalid Continue with any further Metadata files specified in the top file stop-all-valid Stop looking in this MetaData file as the result is valid Do NOT continue looking any further stop-all-invalid Stop looking in this MetaData file as the result is valid Do NOT continue looking any further continue Stop checking at this level in this MetaData file Continue checking at the next (level - 1) ie 1 SU=PA 2 XC=HHH ; if Prioriy is 1 or 2, drop down to the next level 2 3 XP=1 >continue 3 XP=2 >continue 3 XP=* >stop-invalid 2 XC=PPP etc etc etc Example of a check-file: ; ------------------------------------------------------ ; if still valid at the end, add VT:PSC %ADDHDR-IF-VALID:nNO:NO PSCn ; we have wilds %WILD:* ; we assume that it is valid... %default-result:Valid %word-boundary:| ; ------------------------------------------------------ ; If Status is NOT New or Priority 1 - always ignore 1 NV#New >stop-all-invalid 1 IP=1 >stop-all-invalid ; ------------------------------------------------------ ; want Product Codes G or SPO or E (continue to next level 1 or end) 1 NO=* 2 NO=|G| >continue 2 NO=|SPO| >continue 2 NO=|E| >continue 2 NO=* >stop-invalid ; ------------------------------------------------------ ; *** End *** Version Control ;162f52 23oct07 added extra-fiphdr-script ;b 14jan08 allowed DE_DUPLICATE-DIFF:-1 for NO checking at all ;c 26apr08 keyword stripped ';' as comment - needed for 'å' ;d 31jul08 -b, -B added for dataBalGroup, balQue and SX ;e 27aug08 bugette with Reuters which are ONLY Xref/Signoff ;f1-4 18feb09 allow speedy with DEST-IS-FOLDER ;2 allow de-duplicate-time of 0=dont test time;3 NOLOGDELETE: ;4-5 3jun09 allow SD to tbe multiples SD:AP1,AP2,AP3 ;6 22oct09 CHG - redid addhdr-script so it is the same as ipspool (with E3/E4) ;7 16sep10 added DE-DUPLICATE-BALANCE ;8-9 24sep10 added ADDHDR-BEFORE-SCRIPT:.. ;10 10oct10 ignore dedup if DU=logdel or delete ! ;11 23oct10 bugette with DEDUP and no file ;12-14 3mar11 wrinkle with speedy and do NOT allow ';' to mean comments on non-route lines and added %x for CRC ;15 10may11 bugette in chk_route (WIN only) ;16-18 11jul11 added -8 for NOT speedy in but speedy out and $1 back; added HASH-IN-HDR as -$1 or-$t ;19-20 7dec11 added time for done queue duplicates ;21-26 30jan12 allow xml in Reuters-dateline and added HASH-KEY-STOP ;27 4jul12 added KEY-TEXT-MAX ;28-32 19jul12 do NOT add DU for DEST-IS-FOLDER and added ROUTE_LOG_EVERY as FipHdr field ;33-35 20nov12 bugette of addhdr-before-route with no routing lines ;36 8mar13 added big-file-limit and max-round-robin and allow parsed routing line : 1 c=* >2xmlRR ;37-38 7may13 wrinkle with ignoring the FipHdr in '1 c=* >abc+def#SC:ghi' ;39 6jun13 allow SH to change in ADDHDR, ADDHDR-BEFORE-ROUTING ;40 12sep13 added DE-DUPLICATION-IGNORE-DU ;41-43 1oct13 bugette in DE-DUP balance ;44 3jan14 added SHOW-DEDUP:yes/no ;45-48 16jan14 added SCRIPT and TRACKER-SCRIPT and refresh SN if it has changed with ADDhdr etc ;49-52 added SEQNO_BALG/BALQ/NAME and REUTERS-SIGNOFF-SCRIPT ;52 added file-trace ;161z2 03dec04 bugette with more than one kwd search (1 %%=KWDFILE >abc) ;a 15dec04 added -f done queue and DONE-QUEUE: ;b-h 17dec04 bugette with >a#SC:a and %SC:def (f-Roy xref) ;i-j 28apr05 added REPLACE-TICKER ;k-l 16may05 mod for mtime/nfs waits ;m-p 05jul05 use AFP <org> markup in text and buglette in size of 1st dedup ;q-r 03sep05 added euro-tickers and euro-names ;s-v 18jan06 for THROTTLE-MAX and THROTTLE-FIPHDR and DE-DUPLICATE-SOURCE added BEFORE and AFTER ;w 1aug06 rename copied file if already exists ;x-y2 17nov06 bugette in test if singlewild '?' is first chr ;z2 08may07 added CREATE-ZQ ;160z 14nov03 tuning dedups (size is now JUST the data part) ;a 12dec03 added reuters-lines: ;b 12jan04 added anpa-headline: ;c-g 16feb04 cleanup HTML display and added expand-code ;h 01apr04 added RTR code for Advisory->Emb or Corr ;i 25apr04 added stop-all-valid/invalid for Metadata ;j-o 06jun04 bugette in AFPXML plus added REUTERS-BUI-DATELINE ;p-q 27jul04 no BUI in Dateline pls and single spaces in Dateline ;r-s 01sep04 -9 speedy (s for R-Adv) ;t-v 22sep04 added REUTERS-CHK4BUIS and tweak to Advisories ;w 04oct04 extra FipHdr lines are now added to FipHdr immediately .. plus Pilcrow again ;x-y 03nov04 RTRS - case of Xref after a Signoff ;z 17nov04 added dest-is-folder: and default-dest and -w filewait ;159z 06jan03 minor mod to Reuters/Byline ;a 13jan03 RTRS - added UPDATE/WRAPUP revision ;b-g 30jan03 RTRS - added Genre ;h 04mar03 RTRS - Signoff ;i 25may03 RTRS - get/zap HASHes in MetaFields ;j 29may03 RTRS - add xB6-Pilcrow for extra lines in Advisories ;k-l 04jun03 RTRS - strip trailing spaces/lines from Advisory and bugette for signoff ;m 29jun03 VON as a By plus poss correction advisory ;n 04jul03 added REUTERS-PILCROW ;o 23jul03 added PRNEWSWIRE-HEADER ;p 24jul03 redid $f for filetype - which was NOT working well ;q-v 04aug03 added de-duplicates ;w-y 06oct03 stripp and <Fip...> markers for Datelines ;z 31oct03 added timings ;158zi 28dec01 added TYPE-MARKERS and LOCALE and US-TICKERS ;a 22jan02 restrict size of market for US ticker to 20 ;b 05feb02 added -1 for single shot ;c 12feb02 bugette with wildstrings ie k=*sport* sometimes true erroneously ;d/e 27feb02 added $t and $1-9 for ADDHDR ;f/g 03apr02 added BIZWIRE-HEADER ;h 14apr02 ADDHDR can now contain FipSeq for other FipHdrs ;i 18apr02 widgets put in - fixed, partial, combie etc ;j/k/l 21apr02 added check-seqno, seqno-resets-at, check-seqno-max, check-seqno-min ;m 22may02 allow CC!="abc" as well as CC#"abc" and defaultSU put back in ;n 28may02 bugette in word boundary code ;o 5jun02 bugette in ADDHDR - not allowing $1 etc ;p 13jun02 allow more than 2 source redundancy hosts (SZ). ;r 18jun02 added -D and -H for display only ;s/t/u 24jun02 added addhdr-before-routing and -G to grab-first-file ;v 16sep02 added $f for checking the file magic number ;w 17oct02 use calculated DU even if there is an incoming DU ;x/y 26nov02 added %REUTERS-DATELINE ;zi 11dec02 added more BIZWIRE kwds at end ;157b 12nov01 added -METAONLY/-m and word-boundary ;156p 16jul99 added %SEND-SINGLE-COPY: ;b zap FipHdr DP if in the incoming file. ;c 02jul00 added %AFPXML ;d 07aug00 bugette with multiple DUs which blow the filename. ;e 06nov00 added ExtraFipHdr info on the end of a dest. ;f 22nov00 bugette - when using duplicates (SD), sometimes overwrote original file ! ;g 02apr01 bugette - SendSingleCopy sometimes duplicated the 1st DU ;h/j 21may01 redid ExtrafipHdr ;k/l/m 25aug01 strip NULs from data part of file for comparison ;o 05sep01 redid ability to run external program ;p 28sep01 AFP changed their XML .... for %AFPXML (copyright) 2014 and previous years FingerPost Ltd.