All files have some metadata - filename, formatting information, source/feed codings.
In Fip, this is held at the top of a file topped and tailed by a tilde on a line on its own.
Often this metadata needs to be used to fill in output fields in a formatted feed, and more often you will need defaults if that data is wrong or missing, or only partial information is needed or lookups to find other details from standing lookup files
FipSeq is a parameterised means of extracting and using this data.
The FipSeq parameters can be used in (almost) all Fip programs.
One note - there is a program 'sffparse' which can be used to test any of the FipSeq in any Fip parameter file - without making changes to (and possibly damaging) a live feed !
Six Types of FipSeq
FipSeq parameters
Quite often the information in FipHdr fields needs to be presented, formatted or styled differently on output.
Sometimes you need to find out times and dates when the files were processed.
Maybe you need to embeded non-printable characters at the begining or end of a file to start so anterior process.
FipSeq is the notation used to select and format these types of actions.
If you are testing difficult bits of FipSeq, we recommend you use the program 'sffparse' which reads a parameter file of FipSeq, an data file with a FipHdr or a FipHdr string plus an output expression.
All you would ever want in the world of FipSeq !
B. Six Types of FipSeq
There are six basic types of FipSeq which can all be mixed and matched where necessary :
B.1 Ordinary visible data: The sort of stuff you can type and see !
The Slow-challenged coloured cunning-quadruped jumped over ... etc etc etcB.2 Control or unprintable characters are preceded by backslash and are 3 numbers such as \040 \333
Note that Hex numbers can also be specified as \xA9 at any time - (that is little 'x' after the '\' and two hex digits)
The number system used can be octal (default), decimal or hex depending on the 'number' keyword specified in your parameter file.
a space is \040 (octal), \032 (decimal) and \020 (hex) Jimmy \321lsenB.3 High-end Unicode Characters
(UTF-8) - backslash and 4 hex numbers : \uXXXX eg \u12BC
(UTF-8) - backslash and 4 dec numbers : \e9999 eg \e8194B.4 Common Escaped control characters
A single lowercase letter following a backslash:
Valid ones are :
B.5 Contents of FIP header fields
a '\' with two UPPERCASE letters or a Letter and Number
eg
Remember that the Source Header field (SH) is a special case and to get the data specify \X where is the single SH field (note that actual FipHdr fields starting 'X' like XS are NOT accessible).
For example:
SH::Sreu:N1234:P3:CSSS:KSports RoundupIn this case :
B.6 System Variables
| Field | Remarks | Example |
|---|---|---|
| \$A | NAME;dd/mm | JAMES;03/12 |
| \$S | 3 figure sequence number This is normally increments on each file | 123 |
| \$Z | 4 figure sequence number This is normally increments on each file | 0037 |
| \$V | 4 figure sequence number incrementing on each call | 97 |
| \$Y | year (99) | 23 |
| \$E | century (19 or 20) | 20 |
| \$I | month (mm) | 11 |
| \$M | month (Mmm) | Mar |
| \$D | day (99) | 23 |
| \$H | hour (99) | 10 |
| \$N | min (99) | 54 |
| \$G | offset from GMT | +2 |
| \$F | GMT date and time in CCYYMMDDHHMMSS format | 20241021181920 |
| \$B | sec (99) | 45 |
| \$J | julian day of the year | 111 |
| \$K | Day of the week (abbreviated to 3 letters) | Mon |
| \$U | No of seconds since the 'epoch' 1st Jan 1970 - UnixTime or C Time | |
| \$R | random alpha numeric character between a-z, 0-9 | |
| \$Q | (Special) strip followingspaces from this point to the next non-space | |
| \$X | (Special) strip trailing spaces from this point to the last non-space | |
| \$O | (Special) if Optional field failed, strip to this point (see OPTION below) | |
| \$L | no of lines for file | 112 |
| \$W | no of words for file | 887 |
| \$C | no of chrs for file | 7262 |
Plus for certain programs :
How do you insert all or part of some metadata into the text stream - or any zone in your output format - or change the filename etc etc ??
All the different types of FipSeq can be mixed in whatever you are outputting, And that FipHdrs can be 'cascaded' where the result of one FipSeq is used for the next one.
For example, IPPRINT page heading might look like :
header:File:\SN\tFrom:\SU\r\nPrinted at:\t\$h:\$n \$d-\$m-\$c\$y\r\n\r\nFor a file AFP0223 from Source AFP would give 3 lines at the top of the page :
File:AFP0223 From:AFP
Printed at: 12:20 23-Mar-1996But what if ......:
So now we have to start playing with the FipHdr data to create new fipHdr fields for that one file in that single fip program.
There are are a number of functions for reformatting the data:
fixed - create a new FipHdr field from existing data
fixed:QZ 1234543\$j..\SN
partial - create a new FipHdr field from a subset of an existing FipHdr field
partial:QT ST,3,2,U,<,>
repeat - create a new FipHdr field from a subset of an existing FipHdr field based on a field delimiter like a dash :
repeat:QR XK,-,3
or based on a type like NOT a number or a SPACE
repeat:QR XK,,3,U,N
combie - create a FipHdr field which is either the contents of another FipHdr field, or if that does not exist, the contents of a second, or if not there a default string :
combie:QZ ep|na,(0000000)a
option - test for an existing FipHdr field in order to output some relevant FipSeq IF the condition is TRUE.
option:QT ep,11,7,s
style - create a new FipHdr field from an existing field and either truncate or pad left or right. This to pad XM upto 25 spaces
style:QH XM,%.25s
replace - copy an existing FipHdr fields and replace some of the strings.
replace:LC XC SPO=s INL=i UDL=u ECO=f
newdate - create a new, temporary FipHdr field containing date and time fields - possibly reworked to a different date/time timezone (or GMT)
newdate:JS min-38 hour+8 day+\MD "\ZD-\ZI-\ZZ"
unique - create a new FipHdr with the contents of another where the words are unique and separated by a single space or other character. - Use this for controlling Metadata, routing codes and Stox tickers.
valid - create a new FipHdr by checking the contents of another against a standing list of valid entries. If there is no match, the first entry is used as the default.
lookup - create a new FipHdr by matching the contents of another against a standing lookup file.
perl - create a new FipHdr from running a Perl Regular expression or any script/program
merge - create a new FipHdr by merging all the occurances of another fipHdr field
sum -create a new FipHdr from a calculation
filter - create a new FipHdr by filtering the contents of another against a mask or a list of valid entries.
recode - create a new FipHdr by converting the contents of another into base64, quoted printable, hex, percent-encoding/url-encoding
Note
The new fields can be linked together up to 20 times or levels
Example of a snippet of parameter file for IPEDSYS to generate a consistent filename which can be sorted easily on the day of the year :
Lets have a FipHdr containing 3 normal FipHdr fields : SN for storyname, HS for History and SH for the Source Header
SN:dpa6722
HS:wire_9090_99-6-20_13:15:33_5_070
SH::Sdpa:N6722:P1:CPIG:KPigs-In-Space..and the relevant part of the IPEDSYS parameter file ..
; Create a new field called QJ using REPEAT
; Get the Day of the year from the HS field (which is the 6th field)
; Create a new field called QR using COMBIE
; this will be the same as QJ or 0 if either HS did not exist or there was no 6th field
; Create a new field called QE using STYLE
; which will force it to 3 digits padded zero.
;
; Use QE to create a new filename for the output file which will be :
; 3 digit Julian data, Storyname, Category code (C field in SH) and Keyword (K in SH)
repeat:QJ HS,_,6
combie:QR QJ,0
style:QE QR,%.03d
; Filename
name:\QE.\SN.\XC.\XKFor our example FipHdr, this would generate a filename of :
070.dpa6722.PIG.Pigs-In-SpaceThe syntax used is :
(keyword:) (New FipHdr field) (tabs/spaces) (parameters) (eoln)
fixed:QT 1234543
ie If \QT is specified, replace with 1234543
Syntax fixed: [newfield] [tab/space] [fixed text]
Use escape chrs like \s to embed spaces, tabs, newlines etc
If you use a sequence number several times for the same file, (\$Z or \$S) then normally the seqno is incremented EACH time you call it.
However if you specify it once in a fixed field, this new fipHdr field can then be used as many times as you wish !
Example :
before:\$Z : is the seqno
after:\$Z : is now the next seqnoif the seqno is 24, this will stuff :
'24 : is the seqno' at the top of the fileand '25 : is now the next seqno' at the bottom
... Probably NOT what you wanted!
Using fixed we get the same result for this file, next seqno for the next etc.:
fixed:QZ \$Z
before:\QZ : is the seqno
after:\QZ : is still the seqnoif the seqno is 24, this will stuff :
'24 : is the seqno' at the top of the fileand '24 : is still the seqno' at the bottom
An example:
partial:QT ST,3,2,U,+,|ie Create QT using the contents of the ST header field starting at byte 3 for 2 characters and, if alphabetic, force UPPERCASE.
Syntax partial: [newfield] [tab/space] [existing field] [comma] [startposn] [opt comma] [opt length] [opt comma] [opt processing] [opt comma] [opt start character] [opt comma] [opt end character]
where : Start and Length are counters from 1 not 0. Length can be zero or not defined for all characters in the field Processing can be one or two chrs (or none for no change)
If you want to filter a TYPE of chr :
Filters on negatives are also valid using a '#' before the type. eg partial:ZE SN,,,#c will give all the chrs which are NOT control
This will ONLY allow characters matching this type.
For alphabetic chrs, you may also force the case :
There is also a special command - R - which will trim all leading, training and multiple spaces from a FipHdr field This may be used inconjunction with any other processing and/or alphatising (as above).
eg partial:QE SN,1,6,#sL
will give the first 6 NONspace chrs forced lower
Start chr is a start of valid data chr
End chr is an end of string when this chr is encounted
Combies are 'use this it it exists, if not use the 2nd, then the 3rd etc etc, and finally use a fixed string if none of the previous fields hold any data
combie:QZ ep|na,(0000000)aie Stuff QZ with the contents of the EP header field BUT if EP does not exist or has no data then there use the NA field BUT if NA does not exist or has no data then there use the fixed text '(0000000)a'.
Syntax combie: [newfield] [tab/space] [existing field1] [|] [existing field2] [opt comma] [opt default fixed text]
PIPE '|' is used for separating HdrFlds while the first comma means that the rest of the data is fixed text to use as a default.
Note that if a FipHdr field is present BUT has no data, it is considered NOT present. eg
Hdr field BB 'BB:' will fail 'BB:1' will pass
Note also that while the fixed, default CAN have fixed control chrs, it canNOT have anything FipHdr or system variable - like filesize :
; filesize in F0
fixed:F0 \$c
; if QB is empty/missing, use the filesize in F0
combie:QZ QB|F0Optionally output data only if some other metadata exists. This is used with the \$O flag which terminates the optional part
option:QT ep,11,7,sie If EP header field exists and has a space in the 7th position, send this text else strip text until the \$O flag.
Syntax option: [newfield] [tab/space] [newfield] [?] [existing field] [comma] [size] [opt comma] [opt posn of test chr] [opt comma] [opt 'S' to output data otherwise nothing is output] [opt comma] [opt String to test for]
where size is minimum size of field.
Normally this is just a TEST - ie nothing is output - but the 'send' parameter can be used to output the contents from the position specified IF THERE.
Note that both size and test are counters from 1 not 0.
A single chr can be tested to be non-space as in the example above.
If either the size or the test is FALSE, all text and subsequent data whether fixed or variable (including more Optionals) is ignored until the EndOpt flag is met - '\$O' (oh, not zero).
How is this used ?
option:JN SN,10
before:StoryName is \SN \JN..and it is 10 or more chrs long at \$h:\$n \$Owhile the ..If the SN field is less than 10 chrs in length - say SN:lily - the 'before' string will output :
StoryName is lily while the ..ie JN fails so all (text, FipHdr, Sys Variables etc) is stripped to the \$O
However an SN:bigbigbigbig gives a 'before' string of :
StoryName is bigbigbigbig ..and it is 10 or more chrs long at 12:22 while the ..(See a more involved example at the end of the REPEAT section.)
Taking four cases where an example field XB is :
| Specify | (no XB) | XB: | XB:111 | XB:11 11 |
|---|---|---|---|---|
| option:QH XB | fail | fail | pass | pass |
| option:QH XB,1 | fail | fail | pass | pass |
| option:QH XB,4 | fail | fail | fail | pass |
| option:QH XB,1,1 | fail | fail | pass | pass |
| option:QH XB,,1 | fail | fail | pass | pass |
| option:QH XB,,3 | fail | fail | fail | fail |
| option:QH XB,,4 | fail | fail | fail | pass |
In a further example we can test the content of a field
option:QP PR,,,,21Here we do not care about size, position and output; only whether the field starts with the value '21'. Case (lower or upper) is always ignored. To test fields with spaces, special chrs etc, put double quotes around.
option:PF XF,,,,\023\021
option:PZ XR,,,,'PASNAP \$d'Note that the data is compared ONLY for the length of the string specified in the 'option'. So if the FipHdr 'SS' contains '12345678'
option:PS SS,,,,1 - will be TRUE
option:PS SS,,,,12345678 - will be TRUE
option:PS SS,,,,123456789 - will be FALSESo to test for an EXACT string, add a comma at the end of the field. (if you want a comma as part of the string, use FipSeq to encode it - \054 (octal))
If the string to test is NOT the first chr of the FipHdr field, put the stat posn in the 'test' field.
option:PS SS,,3,,34567Test a string is NOT equal using the following syntax:
option:PS #SS,,,,12345ie is SS is NOT 12345, PS will be valid
Test is a number is greater or less than a given amount - use 6th (a greater number) and 7th parameters (less than) to check
option:Q5 N1,,,,,21,14N1 must be between 14 and 21 for Q5 to be true
Extract just a single section of a structured bit of data - such as an URL which has '/' separating the different bits
There are two different types of syntax for 'repeat' :
with a particular chr as separator :
repeat:QR XK,-,3
From FipHdr field XK, using '-' as sep, find the 3rd sub-field
Use this for FipHdr fields like :
RR:engine-train-coach-seat-trolleySyntax 1 repeat: [newfield] [tab/space] [newfield] [?] [existing field] [opt punct/comma] [single sep chr - possibly in FipSeq eg \002] [opt punct/comma] [opt field number from 1 or -1] [opt A for date-from-ALL-remaining-fields] [opt R for reverse]
Examples To get 'coach' field 3 repeat:R3 RR,-,3 To get the last-but-one field 'seat' (where you might have a variable number of fields, use '-' or 'r' repeat:R9 RR,-,-2 repeat:R9 RR,-,2r To get all fields from and including 4 ie R7:seat-trolley repeat:R7 RR,-,4a
in this case the sep is left blank, and valid types are
Syntax 2 repeat: [newfield] [tab/space] [newfield] [?] [existing field] [opt punct/comma] [blank, nothing] [opt punct/comma] [opt field number from 1 or -1] [opt punct/comma] [opt type of field]
eg repeat:QR XK,,3,U
To negate a type of field (ie NOT a number), use the '#' before the type - when you want to test if a byte is NOT a certain type:
repeat:QR XK,,3,#XHere, QR is the 3rd field in the K field of SH separated by a non-alphanumeric chr
Use this for FipHdr fields like :
repeat:QR RR,,5,S
RR:a319 Boeing-747 TigerMoth MothsBros *&^%$ BingBangSo QR would be '*&^%$'
If there are a variable number of repeatable fields and you only want the last one, add 'REV' or a minus sign to the 3rd field to mean :
; to get the last field ...
; .. in W$:http://mighty-ducks.com/super-kwak/legs/tails/123456.html
repeat:AB W$,/,-1Take the last field of SN split by '.' : so if the filename (in the SN field) is ruby.tuesday.may.30.pls.ignore.oink, QX = oink
There is actually a third type of REPEAT code for use with OPTION where it is possible to output all codes :
eg In IPEDSYS, the keyword 'before' adds a String at the top of the data file :
before:<MODVER ID="1:00" VER="03">\n\ZU<DEST ID="1:05">\QU</DEST>\n\$O<FIFO ID="1:20"FFNO="25">\n
; add our option/repeat with a star ..
; if Fip Hdr field \ZU is specified,
; ..look for field \QU has a least 1 chr in it
; .. if there is, output to the \$O flag and then start again with a test for the next field
; .. if there is nothing, strip until the \$O.
option:ZU QU,1
; Hdr Field QU is really EACH sub-field of \NU separated by a comma.
repeat:QU NU+,+*So if Fip Hdr field NU is : NU:HOK,DDD,NNR,MSV,SMF,XXX
The top of the file will have :
<MODVER ID="1:00" VER="03">
<DEST ID="1:05">HOK</DEST>
<DEST ID="1:05">DDD</DEST>
<DEST ID="1:05">NNR</DEST>
<DEST ID="1:05">MSV</DEST>
<DEST ID="1:05">SMF</DEST>
<DEST ID="1:05">XXX</DEST>
<FIFO ID="1:20" FFNO="25">... and the rest of the data
NOTE : '\$' holds the contents of the current Repeat Value - (which should be the same as the FipHdr field in the repeat '\$%' holds the current Repeat seqno from 1 : 1st repeat = 1, 2nd is 2, 3rd is 3 etc '\$+' ONLY allow Repeat value and seqno for this level; ie the string containing (option)....(\$O) (default) Any repeats called inside thuis string should work fine - BUT \$ and \$% will be the NEW repeat '\$-' Allow Repeat value and seqno for this and all subsequent FipSeq calls until the \$O If you use this, any other Repeats which get called before the \$O will go wrong To use, place immediately after the option : repeat:G0 SN,_,* option:V1 G0 fixed:G1 \V1\$-seqno:\$%=\G0\$o '\$|' FlipFlow Repeat value and seqno for this level; ie the string containing (option)....(\$O) (default)
Note normally you cannot have more fixed, replace etc inside a Repeat unless the toggle '\$+' is specified. This is a toggle - so specify once to allow sublevels and then a 2nd time to disallow. Example: where K0=a+b+f+a+A ; unique - dedup the incoming data unique:K1 K0 ; repeat '' and option for the Loop repeat:K2 K1,+, option:V2 K2 ; 3 lookups on the value lookup:KA K2 file:F2MS key:3 value:7 wild: lookup:KB K2 file:F2MS key:3 value:10 wild: lookup:KC K2 file:F2MS key:3 value:6 wild:*
; use \$+ to flag we want the 2nd, 3rd etc ; use \$ to show THIS value fixed:KZ \V2\$+|\$^\KA^\KB^\KC\$o
++ NOTE - NEVER use \$+ with a 2nd repeat as it may lead to odd results ++
Used for padding zeros or spaces (or any other padding chr)
Syntax style: [newfield] [tab/space] [existing field] [style definition]
eg to make a 4 digit number padded with leading zeros
style:ZZ XN,%.04dThis is really just a passthru for the 'sprintf' command ! which is nasty but a standard (of sorts).
Do a man printf for fuller information if you need.
Note the single parameter after the old FipHdr ALWAYS starts with a '%'
If the expression does not end with an 's' ('d' for integer for example), then the string in the header field is first converted to that type.
Specify One and ONLY one expression (can not have %s%d%f) - as is takes the first only
Do NOT use for fixed data - use keyword 'fixed:' (as explained above)
Types are :
Common Examples :
| Description | style defn |
|---|---|
| To trim a string, use a dot | %.5s |
| To pad a string with spaces | %5s |
| To pad a string with spaces (left justified) | %-5s |
| To pad a number with up to 6 leading zeros | %.06d |
Copy the contents of one FipHdr field into another and then search and replace characters or strings.
Syntax replace:[New FipHdr] [spc/tab] [Existing FipHdr] [optional comma then flags to ForceCase, encode etc] [spc/tab] (Search String) (= or :) (Replace)
replace:LC XC SPO=s INL=i UDL=u ECO=fThere can be up to 50 searches/replace tuplets on a single line.
Flags -
By default case i ignored - but if the 'C' flag is specified, the Search String MUST always be in the right case. So always specify all combinations : SPO=s spo=s
The output can be forced upper or lowercase by specifying 'U' or 'L' after the existing FipHdr field. eg replace:LC XC,u SPO=s INL=i UDL=u ECO=f
Other flags can be set to:
Use a start-of-block or end-of-block chr : b=|
; this will strip all spaces from the star and end of the zone, replace any combination of Yuummmy by YUM and heXXo with hello replace:LC AB,s=?,r=+,b=| \s+|="" |\s+="" Yu+m+y=YUM he??o=hello
Strings with spaces and punctuation MUST be enclosed by double quotes.
replace:K2 YR,u ".Y"=yellow ".C"=cyan ".M"=magenta ".K"=blackThe replacement string may be in FipSeq - but NOT the search.
You must take care to only use small amounts of data - 1000 chrs max
Remember you must Never call a FipHdr field from itself !!
Create a new internal FipHdr field with a date relative to when the program runs
Syntax newdate: [new FipHdr] [datetime differences] [style]
Style MUST be in double quotes. Style can contain both ordinary words and FipSeq.
The syntax for differences is
eg
hours+3 for plus 3 hours
days-7 for the same day a week agoIf you need to use a date which is relative to a date/time which is NOT the current date, put 'basedate=YYYYMMDDHHNNBB'. By adding a 'Z' for Zulu time on the end of the basedate string, then this basedate will be considered to be GMT; otherwise the time will be in the localtime of whatever the server is set to.
eg Y0:20211130112233
newdate:A1 basedate:\Y0Z "\ZH:\ZF:\ZE"If you need to output the date/time in GMT/Zulu time, add 'gmt'. newdate:A1 gmt "\ZH:\ZF:\ZE" newdate:A2 gmt hour-10 "\ZH:\ZF:\ZE"
The GMT parameter can be specified for input or output (default is output only if specified WITHOUT) but do NOT specify both in and out as they will cancel each other out !
newdate:A1 gmt:in "\ZH:\ZF:\ZE"
newdate:A1 gmt:out "\ZH:\ZF:\ZE"If you want to recalculate against a time zone shift - such as GMT from 2012-09-28T11:33:52-05:00 - use BOTH basedate=(he date) and 'offset=yes' to take the offset into account. (Be warned that Fip - and a well-known supplier managed to get the sign of the zone wrong, so this can be used to sort it out !
There are no spaces between the number and the hours etc.
Several differences can be specified, each separated by a SPC or TAB.
The contents of ANOTHER fiphdr field may be used for the number :
hours+\JR
Only the first (or for 'm' first two) letters are considered.
The default is a PLUS sign meaning in the future.
'h3' is the same as 'hours+3'
The Style MUST be in double quotes. They can contain both ordinary words and FipSeq. Relevant header fields are
* ZD - 2 digit day of month (leading space)
* ZG - 2 digit day of month (leading zero)
* ZM - 2 digit month
* ZY - 2 digit Year 92
* ZZ - 4 digit Year 1992
* ZW - Day of week as in Monday, Tuesday etc
* ZS - 3 chr Day of week as in Mon, Tue etc
* ZN - Month as in January, February
* ZT - 3 chr Month as in Jan, Feb, Mar etc
* ZJ - Julian day of year
* ZH - Hour 00-23
* ZI - Hour 00-12
* ZF - Minute 00-59
* ZE - Second 00-59
* ZU - 1st, 2nd, 3rd, 24th for the day of
* ZA,ZB,ZK - The three different week-of-year numbers (ZA is 0-53 starting Sunday, ZB is 0-53 starting Monday, ZK is ISO8601 1-53 where the first week has 4 or more days starting Monday!)
* ZC - Day of week
* ZP - AM or PM
* ZL - Daylight saving offset HHMM
* ZQ - Daylight saving difference in hours between system time and time calculated in newdate
* ZR - UTC offset from GMT in +/-HHMM format
* ZX - Unix time in Secs|Unix time in MinsNote that actual Day and Month names depend on your LOCALE Default is "\ZW, \ZD \ZN \ZZ"
Example
newdate:JS min-38 hour+8 day+\MD "xx\ZD-\ZS-\ZZ or \ZY \ZH:\ZTxx"Where MD is a FipHdr field in the incoming file. Gives a result like xx01-Sat-2025 or 00 00:17xx
newdate:J7 "\ZS, \ZD \ZT \ZZ \ZH:\ZM:\ZE +0100"Will give the current system date/time as an RFC2822 string Gives a result like Fri, 15 May 2019 17:58:28 +0100
Control the contents of a single FipHdr field so that :
Syntax unique:(new FipHdr Fld) (space) (old FipHdr Field) [opt (punct/comma) (separator) ] [opt (punct/comma) (force Upper or Lower case) ] [opt (punct/comma) (output separator) ]
The Separator defaults to a plus sign '+'.
unique:Q1 F1,|
unique:Q2 XC,=So if XC (ie the C field of SH) was :
SH:N1234:Sabc:Ccat1=cat1=cat3=cat2=cat3=cat4:P3Q2 now becomes
Q1:cat1=cat2=cat3=cat4If a separator is specified, ONLY that is used to divide the zones. Otherwise, any space, plus sign '+' or comma ',' is considered a divider. Multiple separators between zones are just ignored. If you want the Sep to be a comma, use another punctuation chr after the FipHdr field. eg
unique:Q1 XC,+,If you want a different Output separator, specify as the third parameter eg :
unique:Q1 XC,,,$So if XC (ie the C field of SH) was :
SH:N1234:Sabc:Ccat1 cat1,cat3+ cat2 cat3 , cat4:P3As no input separator has been specified, the elements are split by Space, comma or plussign Q1 now becomes
Q1:cat1$cat2$cat3$cat4Create a new FipHdr by checking the contents of another against a standing list of valid entries.
*Syntax** valid:(new FipHdr Fld) (space) (old FipHdr Field) [(opt) (comma) (flag - force Upper or Lower case) ] (list of values)
The list of values are separated by one (or more) spaces/tabs.
So if any of the values contain a space or comma, it must be inside double quotes
Optionally a flag can be specified after the Old FipHdr field which will force the resulting data :
- U uppercase
- L lowercase
- I idicase - or camel-case where the first letter of each word is capitalised
- F upperFirst or initial case
- C data to match is case-sensitive.
- ? a single wild character (this can be any punctuation except '",)
- X if there is NO matching value, set the FipHdr to nothing
- (Normally If there is NO match, the first value in the list is used as a default.)
- V if the data in the FipHdr has multiple values, output then in Valid-List order, NOT the current order of fields in the FipHdr
- (a number from 1 to 9) if the data in the FipHdr has multiple values, output JUST that number of items
eg valid:Q1 XP 4 1 2 3 5
Create Q1 from XP which must be a value of 1 to 5 and defaults to 4
eg valid:Q1 XP,X 4 1 2 3 5
Create Q1 from XP which must be a value of 1 to 5 and there is no default value, so Q1 will be blank in that case
If you do NOT specify ',X' and there is no match, the first entry is used as the default unless the FipHdr field does NOT exist or has no data - in which case there is no default. So if you ALWAYS need something, use a 'combie' to make sure there is some content in the data-to-be-validated.
eg combie:Q0 AY,XX
valid:Q1 Q0 XX SE DK NO FI BX "Z "
; allow any value starting MCC: with one or three trailing chrs
valid:P1 TR,? MCC:? MCC:???
; note that the values may also be a variable in FipSeq - good practice is to make sure there is ONE value in the list !
combie:P0 PZ,first
valid:P1 TX second \P0
Note that the original VALID only allowed for a single item of data-to-be-validated. But from April2014, the data zone can have zero, one or many items - but only the valid ones are left afterwards.Create a new FipHdr by matching the contents of another against a standing lookup file.
Note this is for simple, small lookup files NOT massive tables of thousands of entries.
Syntax lookup:(new FipHdr Fld) (space) (old FipHdr Field) (series of sub-parameters ... )
The old FipHdr field will contain the key to search for. SubParameters are :
Limit example - use 'limit' where the separator chr may be in the last field eg
; data in lookup file
text-top:Some :rubbish: to be added at the "top: of the file.. and in the parameter file
fixed:YC text-top
lookup:AL YC file:/fip/tables/setup/RUBBISH key=1 value=2 limit=2 sep=:Wild example for short keys :
In the parameter file wild:%
In the file car%|4wheels|petrol
will match data with 'car alarm', 'carriage', 'cart'Error-code example :
error-code:w will give a '!w' type in the item log
Fri Jun 26 12:26:40 ipxchg !w : ** FipSeq - No Such LOOKUP file /fip/web/preferences/SPECIALISTGeneral examples :
lookup:Y1 YP file:Categories sep:| comment:# key:3 value:5
Create Y1 from the entry in file /fip/tables/setup/Categories which matches the contents of XP where the key is in field 3 and the value to use is in field 5.
If using Fixed width fields, the width of each field follows the 'F' in type-format.
eg lookup:Y2 SU file:services key:1 value:4 type-format:F,6,6,2,2,3
Create Y2 from the entry in file /fip/tables/setup/services which matches the contents of SU where the key is in field 1 (which are characters 1 to 6) and the value to use is in field 4 (characters 15 and 16) .
Where the old FipHdr has more than one key - and you need to lookup ALL of them, then specify the Rpt-Separator - usually a spc (as it is unprintable, use \s) The result is a single FipHdr with all the new codes withe the same separator.
eg F1:AAA BBB CCC DDD
and the lookup file maps
; comment here
AAA^ZZZ
BBB^YYY
DDD^XXX
lookup:F2 F1 file:/fip/tables/setup/MYMAP sep:^ comment:; key:1 value:2 rpt-sep:\s
gives F2:ZZZ YYY XXX
(note CCC is missing, so only 3 are returned)Create a new FipHdr from a Perl Regular expression or an external script (or program)
Syntax 1 perl:(new FipHdr Fld) (space) (Perl RegExp)
Syntax 2 perl:(new FipHdr Fld) (space) (path to any script or program plus any subparameters in FipSeq)
Note that all '\' should be doubled as the RegExp string is considered to be FipSeq, so other FipHdr fields may be included.
The RegExp MUST be on a single line and you are advised to use a 'print' to display the result you want.
Any NewLines and/or Carriage Returns are mapped to Spaces.
Examples
perl:E1 $seqno = "\VD"; $bigSeq = 24*$seqno; print "Seqno is $bigSeq";
Create E1 with (if FipHdr VD:3) "Seqno is 72".
the second format is to run a script (of any variety)
perl:AB /fip/local/redoAB.php \QU \L0Take care that the syntax is correct and does not have an infinite loop!
Take care to respect Perl quotes - especially for Winnt where for some bizarre reason we need two double quotes around it !
eg for Winnt
perl:Z4 ""@hoho = 29 * \F4;""Create a new FipHdr from a on or more occurances of another FipHdr field.
Normally, it a FipHdr field is repeated, only the last FipHdr fields is accessible - which allows you to overwrite entries quite happily.
But there are some (few) occasions where in fact you want the first or second or maybe all of them concatenated together in one long string. This could be when you run 'ipxchg' against another formatted file which has duplicated entries for some fields.
Syntax merge:(new FipHdr Fld) (space) (old FipHdr Fld) [(optional) , Separator] [(optional) , First-occurance] [(optional) , NumberOfTimes] [(optional) , T-op, B-ottom, R-everse]
In the above syntax the comma may be replaced by any other punctuation except '$'. The Separator is to divide two or more fields and can be in FipSeq and can be a TAB (\t), SPC (\s) or any printable chr. The First occurance is counted from the begining or top of the FipHdr. The NumberOfTimes is the number of fields to take into account.
The Top or Bottom flag denoted if the occurance is from the Top of the FipHdr (default - first ones) or the Bottom (last ones). While Reverse will sort the output backwards (LIFO rather than the default of FIFO). Specify ONLY a single chr T or B - or the R or nothing.
eg if the FipHdr had :
* AB:blue
* AB:green
* AB:red
merge:BC AB,+
gives a new Fiphdr field of BC:blue+green+redCreate a new FipHdr from the result of a calculation and/or output in decimal, hex, octal, base36 or base64
Syntax sum:(new FipHdr Fld) [: optional flags] (space) (the calculation)
The calculation is in FipSeq - so can include other FipHdr fields
Calculations can NOT be split over several lines.
The default precision for a calculation is 2 decimal places. This can be overridden using the syntax :
sum:AB 100*(\Q1/\Q2)Flags can be :
Operators can be :
Take care when dividing by zero ! so use a 'COMBIE' and 'PARTIAL' to make sure.
Use round brackets to denote how the calculation should be worked out. Ie the deepest level is calculated first, and then the calculation is worked from left to right.
For example : (\Q1/\Q2)*(\Q3/(\Q5+\Q4))/100
.. will add run through the following order :
* Step 1 (\Q5+\Q4)
* Step 2 (\Q1/\Q2)
* Step 3 (\Q3/result of Step 1)
* Step 4 (result of Step 2 * result of Step 3)
* Step 5 (result of Step 4 / 100)Create a new FipHdr by filtering the contents of another against a mask or a list of valid entries.
Syntax filter:(new FipHdr Fld) (tab/spcs) (existing FipHdr field) [: optional flags] (tab/spcs) (list and/or mask)
Some XML schemas like NewsML G2 allow for a qualified metadata.
Use FILTER to reduce the number of multiple entries in a FipHdr file eg
; allow any value starting MCC: with one or three, but NOT two, trailing chrs
filter:P1 TR,! MCC:! MCC:!!!
filter:P2 AC Sports News BusinessEntries may be dbl quoted if spaces are needed.
Flags are specified in order
To Exclude items (filter normally just INcludes), add a HASH before the old Fiphdr field
filter:P3 #TR,,%,+ qcode:%
This will filter all in the TR FipHdr field, (separated by '+') which do NOT start 'qcode:...'
Create a new FipHdr by recoding the contents of another
Syntax recode:(new FipHdr Fld) [: optional flags] (tabs/spaces) (existing FipHdr field),(input coding),(output coding),[optional output size],[opt pctencoding Safe chrs]
- a blank coding means take as ordinary 8 bits chrs
- for input, it is the contents of the string
- for output the default is UTF8NOTE this is really for strings so beware about binary data. eg
; take the contents of P0 and put a base64 encoded version in P1
recode:P1 P0,,base64,32
; take the contents if W$ and urlencode/percent-encode it for P2
recode:P2 W$,,pctencode
; urlencode Q5, EXCEPT these safe chrs, and populate Q6
recode:Q6 Q5,,pctencode,,,-_.~=&
; convert the contents of AN - which is in base64 - and create P3 with the same contents in hex
recode:P3 AN,base64,hex
; J1 is the length of JG
recode:J1 JG,lengthCodings are :
A special 'coding' is L - Length Only - which returns just the length of the string
These 5 can be optionally Upper or Lowercase
Force the input string to be a Number: These 4 can be optionally big or little endian or 'whatever the system is' endian
For Pct or Url encoding, the default 'safe chrs' (ie ones that are NOT mapped to %XX format) are
Use the last parameter to overwrite the default - you must specify all in this case !
(see https://tools.ietf.org/html/rfc3986 or https://en.wikipedia.org/wiki/Percent-encoding for more information)
Include data from an external file when this FipHdr field is specified
Syntax include:(new FipHdr Fld) (existing FipHdr field) [optional (tab/spaces) pathname:(fipSeq pathandfilename)] [optional (tab/spaces) pathname:(fipSeq pathandfilename)] [optional (tab/spaces) pathname:(fipSeq pathandfilename)]
- if the new FipHdr field is encountered - in a string or a parsed file - it will include the contents of the file pointed to.
- maximum size of the data is under 62k
- if a pathname is specified, then it holds the full path and filename of the data
- if NO Pathname is specified, then the OLD FipHdr field should have the full path and filename
pathname:(filename in fipseq)
If the old FipHdr does NOT have the full pathname for the data file
default:(filename in fipseq)
replacement data if there NO oldFipHdr or NO file
missing-msg:"Missing include file for SPORTS TEAMS "
log message if there is no entry for this value
error-msg: (flag chr) or '-' for no error default is x for !x error messages
eg: FS:SOCCER_TEAMS.LOOKUP
include:F1 FS template:/fip/tables/setup/\FS
FF:/fip/tables/setup/SOCCER_TEAMS.LOOKUP
include:F2 FF
- specifying \F1 or \F2 in a fipseq string will output the contents of that file at that point
before:Soccer Teams is :\n\F1\n
Version Control
; 38k5 ;a 20may19 bug in partial/UTF8 ;b TooMuchData (> 64k - current max) ;c minor
;d allow fipseq_extras - % ;e 26oct20 minor prototypes ;f 21dec20 added partial : json and xml
;g 10mar21 filter/NOT issuette plus lookup/only-allow multiple chrs
;h added partial type SLUG 'g'
;j 20mar22 allow \%x for hash
;i-k 28feb22 BUGlet lookup rpt-sep output sep being ignored ;k1 macosx ;k2 started FIPSTORE
; 37z ;a 25may15 added Lookup blank-value and allow range value=3-7
;b 19jun15 perl/script and Win2012 ! why do Microsoft have to change things ?
;c 15jul15 added lookup error-code:w to make it a warning
;d 5apr16 maxFilters 100->250 as Reuters are adding 160+
;e 16may16 bug in unique when single FipHdr can be > STDBUF
;fg 22aug16 issuette in option (not serious)
;h-k 12nov16 added lookup/row and lookup/fixed with spaces
;l-n 27jul17 added 'recode' and \\$p = FIP_startTime (lib/prog)
;o 5feb18 added newdate .. tz=Australia/Brisbane ..
;p 1apr18 added recode-string to number and bugette in new 'tz' code
;q 7may18 added fipseq_extras for $T and $1 and ...
;r 5sep18 buglette in lookup - rpt-sep
;s 10sep18 added \@x as a Program-dependent variables (mainly sr_PSYS calls)
;t-v 25sep18 added recode/pctencode - safe
;wxy 6jan19 added include:
;z pp_trim_spaces - trim dbl/single quotes around plus better filter control
; 36z ;a 12mar12 PARSE_FipHdrSize->HDRBUF-STDBUF ;b allow valid items to be parseable
;c PARSE_FipHdrSize->FIP_limitHdrFieldSize
;d tuned unique
;e 21dec12 strip some duplicating error messages for lookup etc
;f 26mar13 allow A$ in combie
;g 5jun13 added 8bit and 7bit for partial
;h 11jul13 added urlencode and urldecode to replace
;i 23aug13 added REPLACE - singlewild, repeatChr, blockstart/end
;j 30aug13 PARTIAL - 8=utf8
;kl 16sep13 bugettes in Filter when item found at end of line with multiple seps after
and Replace if nofh and added negate to Filters : filter:W1 #RI,* R:*
;m 7oct13 added 'all' to repeat meaning this field and the rest of the string - ; to get the path of a file : repeat:F3 FN,/,-2all
;n 28jan14 woops bugette in unique
;o 26feb14 bugette in replace if found at end of buffer
;p 4apr14 tuned perl:
;q-s 14apr14 Valid - allow more than one element, added output order and number, newdate vc2005
;t-u 10jun14 buggettes in newdate and valid
;v 10oct14 tune replace (do NOT allow a:b)
;w 22oct14 added ZL ZQ ZR to newdate for DST and UTC offsets
;x 2dec14 added output sep for UNIQUE
;y 31mar15 redid merge and added reverse
;z 28apr15 added limit (ie max no of fields for lookup
(copyright) 2024 and previous years FingerPost Ltd.