FipSeq for All

Overview

All files have some metadata - filename, formatting information, source/feed codings.

In Fip, this is held at the top of a file topped and tailed by a tilde on a line on its own.

Often this metadata needs to be used to fill in output fields in a formatted feed, and more often you will need defaults if that data is wrong or missing, or only partial information is needed or lookups to find other details from standing lookup files

FipSeq is a parameterised means of extracting and using this data.

The FipSeq parameters can be used in (almost) all Fip programs.

One note - there is a program 'sffparse' which can be used to test any of the FipSeq in any Fip parameter file - without making changes to (and possibly damaging) a live feed !

Six Types of FipSeq

  • Ordinary visible data
  • Control or Unprintable characters.
  • High-end Unicode characters (UTF-8)
  • Common escaped characters.
  • Contents of FIP header fields
  • System Variables

FipSeq parameters

  • fixed - create a new FipHdr field from existing data
  • partial - create a new FipHdr field from a subset of an existing FipHdr field
  • combie - create a FipHdr field which is either the contents of another FipHdr field, or if that does not exist, the contents of a second, or if not there a default string
  • option - test for an existing FipHdr field in order to output some relevant FipSeq IF the condition is TRUE.
  • repeat - create a new FipHdr field from a subset of an existing FipHdr field
  • style - create a new FipHdr field from an existing field and either truncate or pad left or right.
  • replace - copy an existing FipHdr fields and replace some of the strings.
  • newdate - create a new, temporary FipHdr field containing date and time fields
  • unique - create a new FipHdr with the contents of another where the words are unique and separated by a single space or other character. - Use this for controlling Metadata, routing codes and Stox tickers.
  • valid - create a new FipHdr by checking the contents of another against a standing list of valid entries. If there is no match, the first entry is used as the default.
  • lookup - create a new FipHdr by matching the contents of another against a standing lookup file.
  • perl - create a new FipHdr from running a Perl Regular expression
  • merge - create a new FipHdr by merging all the occurances of another fipHdr field
  • sum - create a new FipHdr of the result of a calculation
  • filter - create a new FipHdr by filtering the contents of another against a mask or a list of valid entries.
  • recode - create a new FipHdr by converting the contents of another into base64, quoted printable, hex, percent-encoding/url-encoding
  • include - include data from an external file when this FipHdr field is specified

A. Introduction to FipSeq

Quite often the information in FipHdr fields needs to be presented, formatted or styled differently on output.

Sometimes you need to find out times and dates when the files were processed.

Maybe you need to embeded non-printable characters at the begining or end of a file to start so anterior process.

FipSeq is the notation used to select and format these types of actions.

If you are testing difficult bits of FipSeq, we recommend you use the program 'sffparse' which reads a parameter file of FipSeq, an data file with a FipHdr or a FipHdr string plus an output expression.

All you would ever want in the world of FipSeq !

B. Six Types of FipSeq

There are six basic types of FipSeq which can all be mixed and matched where necessary :

B.1 Ordinary visible data: The sort of stuff you can type and see !

The Slow-challenged coloured cunning-quadruped jumped over ... etc etc etc

B.2 Control or unprintable characters are preceded by backslash and are 3 numbers such as \040 \333

Note that Hex numbers can also be specified as \xA9 at any time - (that is little 'x' after the '\' and two hex digits)

The number system used can be octal (default), decimal or hex depending on the 'number' keyword specified in your parameter file.

a space is \040 (octal), \032 (decimal) and \020 (hex) Jimmy \321lsen

B.3 High-end Unicode Characters

(UTF-8) - backslash and 4 hex numbers : \uXXXX eg \u12BC
(UTF-8) - backslash and 4 dec numbers : \e9999 eg \e8194

B.4 Common Escaped control characters

A single lowercase letter following a backslash:

Valid ones are :

  • \r CarriageReturn / hex 0D, octal 015
  • \n Newline / hex 0A octal 012
  • \l CR NL Octal 015 012
  • \s Space Octal 040
  • \f FormFeed Octal 013
  • \t Tab Octal 011
  • \b Backspace Octal 010
  • \ Backslash Octal 134

B.5 Contents of FIP header fields

a '\' with two UPPERCASE letters or a Letter and Number

eg

  • \SN - output the filename
  • \SU - output the Wire Agency name
  • \H9 - webwire HTTP result code and message

Remember that the Source Header field (SH) is a special case and to get the data specify \X where is the single SH field (note that actual FipHdr fields starting 'X' like XS are NOT accessible).

For example:

SH::Sreu:N1234:P3:CSSS:KSports Roundup

In this case :

  • \XK is 'Sports Roundup'
  • \XP is '3'

B.6 System Variables

Field Remarks Example
\$A NAME;dd/mm JAMES;03/12
\$S 3 figure sequence number This is normally increments on each file 123
\$Z 4 figure sequence number This is normally increments on each file 0037
\$V 4 figure sequence number incrementing on each call 97
\$Y year (99) 23
\$E century (19 or 20) 20
\$I month (mm) 11
\$M month (Mmm) Mar
\$D day (99) 23
\$H hour (99) 10
\$N min (99) 54
\$G offset from GMT +2
\$F GMT date and time in CCYYMMDDHHMMSS format 20241021181920
\$B sec (99) 45
\$J julian day of the year 111
\$K Day of the week (abbreviated to 3 letters) Mon
\$U No of seconds since the 'epoch' 1st Jan 1970 - UnixTime or C Time
\$R random alpha numeric character between a-z, 0-9
\$Q (Special) strip followingspaces from this point to the next non-space
\$X (Special) strip trailing spaces from this point to the last non-space
\$O (Special) if Optional field failed, strip to this point (see OPTION below)
\$L no of lines for file 112
\$W no of words for file 887
\$C no of chrs for file 7262

Plus for certain programs :

  • \$T first 80 chrs of text - ipedsys, ipsgml
  • \$1 first line of text - ipedsys, ipsgml
  • \$9 ninth line of text - ipedsys, ipsgml

Reformatting FIP header data, system variables and other metadata

How do you insert all or part of some metadata into the text stream - or any zone in your output format - or change the filename etc etc ??

All the different types of FipSeq can be mixed in whatever you are outputting, And that FipHdrs can be 'cascaded' where the result of one FipSeq is used for the next one.

For example, IPPRINT page heading might look like :

header:File:\SN\tFrom:\SU\r\nPrinted at:\t\$h:\$n \$d-\$m-\$c\$y\r\n\r\n

For a file AFP0223 from Source AFP would give 3 lines at the top of the page :

File:AFP0223    From:AFP
Printed at: 12:20 23-Mar-1996

But what if ......:

  • you only want PART of a header field?
  • you have a field that contains several sub-fields?
  • you want to test if a field exists or not?
  • you need to make sure the data is an exact number of bytes, possibly padding left or right?

So now we have to start playing with the FipHdr data to create new fipHdr fields for that one file in that single fip program.

There are are a number of functions for reformatting the data:

  • fixed - create a new FipHdr field from existing data

    fixed:QZ 1234543\$j..\SN

  • partial - create a new FipHdr field from a subset of an existing FipHdr field

    partial:QT ST,3,2,U,<,>

  • repeat - create a new FipHdr field from a subset of an existing FipHdr field based on a field delimiter like a dash :

    repeat:QR XK,-,3

    or based on a type like NOT a number or a SPACE

    repeat:QR XK,,3,U,N

  • combie - create a FipHdr field which is either the contents of another FipHdr field, or if that does not exist, the contents of a second, or if not there a default string :

    combie:QZ ep|na,(0000000)a

  • option - test for an existing FipHdr field in order to output some relevant FipSeq IF the condition is TRUE.

    option:QT ep,11,7,s

  • style - create a new FipHdr field from an existing field and either truncate or pad left or right. This to pad XM upto 25 spaces

    style:QH XM,%.25s

  • replace - copy an existing FipHdr fields and replace some of the strings.

    replace:LC XC SPO=s INL=i UDL=u ECO=f

  • newdate - create a new, temporary FipHdr field containing date and time fields - possibly reworked to a different date/time timezone (or GMT)

    newdate:JS min-38 hour+8 day+\MD "\ZD-\ZI-\ZZ"

  • unique - create a new FipHdr with the contents of another where the words are unique and separated by a single space or other character. - Use this for controlling Metadata, routing codes and Stox tickers.

  • valid - create a new FipHdr by checking the contents of another against a standing list of valid entries. If there is no match, the first entry is used as the default.

  • lookup - create a new FipHdr by matching the contents of another against a standing lookup file.

  • perl - create a new FipHdr from running a Perl Regular expression or any script/program

  • merge - create a new FipHdr by merging all the occurances of another fipHdr field

  • sum -create a new FipHdr from a calculation

  • filter - create a new FipHdr by filtering the contents of another against a mask or a list of valid entries.

  • recode - create a new FipHdr by converting the contents of another into base64, quoted printable, hex, percent-encoding/url-encoding

  • include - include data from an external file when this FipHdr field is specified

Note

  • These FipHdr fields are ONLY for use internally to that one program and are NOT added to the existing FipHdr UNLESS you specify they should be
  • Any FipHDr field created by FipSeq can be used in exactly the same way as any FipHdr created by any program automatically.
  • If an existing FipHdr field has the same name as a newly created FipSeq, the OLD is ignored and the NEWLY created one used. So please make sure you do NOT use existing FipHdr field names with valid data in them as they will NOT be accessible.)
  • So FipHdr fields are 'reserved' for internal use - (see the program doc for 'ipwheel') especially Sx Dx Hx Zx Ex Wx
  • Others are NEVER used internally such as fields starting Qx, Nx, Tx, Ux, Vx where x is A-Z, 0-9 and $ (dollar)

The new fields can be linked together up to 20 times or levels

Example of a snippet of parameter file for IPEDSYS to generate a consistent filename which can be sorted easily on the day of the year :

Lets have a FipHdr containing 3 normal FipHdr fields : SN for storyname, HS for History and SH for the Source Header

SN:dpa6722
HS:wire_9090_99-6-20_13:15:33_5_070
SH::Sdpa:N6722:P1:CPIG:KPigs-In-Space

..and the relevant part of the IPEDSYS parameter file ..

; Create a new field called QJ using REPEAT
;   Get the Day of the year from the HS field (which is the 6th field)
; Create a new field called QR using COMBIE
;   this will be the same as QJ or 0 if either HS did not exist or there was no 6th field
; Create a new field called QE using STYLE
;   which will force it to 3 digits padded zero.
;
; Use QE to create a new filename for the output file which will be :
;   3 digit Julian data, Storyname, Category code (C field in SH) and Keyword (K in SH)
repeat:QJ   HS,_,6
combie:QR   QJ,0
style:QE    QR,%.03d
; Filename
name:\QE.\SN.\XC.\XK

For our example FipHdr, this would generate a filename of :

070.dpa6722.PIG.Pigs-In-Space

In More Detail ...........

The syntax used is :

(keyword:) (New FipHdr field) (tabs/spaces) (parameters) (eoln)

FIXED fields

fixed:QT 1234543

ie If \QT is specified, replace with 1234543

Syntax fixed: [newfield] [tab/space] [fixed text]

Use escape chrs like \s to embed spaces, tabs, newlines etc

If you use a sequence number several times for the same file, (\$Z or \$S) then normally the seqno is incremented EACH time you call it.

However if you specify it once in a fixed field, this new fipHdr field can then be used as many times as you wish !

Example :

 before:\$Z : is the seqno
 after:\$Z : is now the next seqno

if the seqno is 24, this will stuff :

 '24 : is the seqno'        at the top of the file

and '25 : is now the next seqno' at the bottom

... Probably NOT what you wanted!

Using fixed we get the same result for this file, next seqno for the next etc.:

 fixed:QZ   \$Z
 before:\QZ : is the seqno
 after:\QZ : is still the seqno

if the seqno is 24, this will stuff :

 '24 : is the seqno'        at the top of the file

and '24 : is still the seqno' at the bottom


PARTIAL fields

An example:

partial:QT  ST,3,2,U,+,|

ie Create QT using the contents of the ST header field starting at byte 3 for 2 characters and, if alphabetic, force UPPERCASE.

Syntax partial: [newfield] [tab/space] [existing field] [comma] [startposn] [opt comma] [opt length] [opt comma] [opt processing] [opt comma] [opt start character] [opt comma] [opt end character]

where : Start and Length are counters from 1 not 0. Length can be zero or not defined for all characters in the field Processing can be one or two chrs (or none for no change)

If you want to filter a TYPE of chr :

  • a alphabetic letters
  • x alphanumeric - letters and numbers
  • e alphanumeric plus dot
  • z anpa type - alphanumeric plus dash
  • d decimal numbers plus optional dot, commas
  • m money plus optional dot, commas, plus, minus and dollar
  • n number 0,1,2,3,4,5,6,7,8,9
  • t punctuation (!'\043$%^&*()_+=-:;@'~#<>,.|)
  • s space (space, tab, ff, cr, nl but NOT backspace)
  • p printable (including space)
  • w printable (excluding space)
  • c control

Filters on negatives are also valid using a '#' before the type. eg partial:ZE SN,,,#c will give all the chrs which are NOT control

This will ONLY allow characters matching this type.

For alphabetic chrs, you may also force the case :

  • u uppercase
  • l lowercase
  • i idicase - or camel-case where the first letter of each word is capitalised
  • f upperFirst or initial case
  • 7 only allow 7 bit chrs (and strip high end utf8/ucs2 etc
  • 8 allow and check for valid utf8 chrs

There is also a special command - R - which will trim all leading, training and multiple spaces from a FipHdr field This may be used inconjunction with any other processing and/or alphatising (as above).

eg partial:QE SN,1,6,#sL

will give the first 6 NONspace chrs forced lower

Start chr is a start of valid data chr

End chr is an end of string when this chr is encounted


COMBINATIONS

Combies are 'use this it it exists, if not use the 2nd, then the 3rd etc etc, and finally use a fixed string if none of the previous fields hold any data

combie:QZ   ep|na,(0000000)a

ie Stuff QZ with the contents of the EP header field BUT if EP does not exist or has no data then there use the NA field BUT if NA does not exist or has no data then there use the fixed text '(0000000)a'.

Syntax combie: [newfield] [tab/space] [existing field1] [|] [existing field2] [opt comma] [opt default fixed text]

PIPE '|' is used for separating HdrFlds while the first comma means that the rest of the data is fixed text to use as a default.

Note that if a FipHdr field is present BUT has no data, it is considered NOT present. eg

Hdr field BB 'BB:' will fail 'BB:1' will pass

Note also that while the fixed, default CAN have fixed control chrs, it canNOT have anything FipHdr or system variable - like filesize :

; filesize in F0
fixed:F0    \$c
; if QB is empty/missing, use the filesize in F0
combie:QZ   QB|F0

OPTION

Optionally output data only if some other metadata exists. This is used with the \$O flag which terminates the optional part

option:QT   ep,11,7,s

ie If EP header field exists and has a space in the 7th position, send this text else strip text until the \$O flag.

Syntax option: [newfield] [tab/space] [newfield] [?] [existing field] [comma] [size] [opt comma] [opt posn of test chr] [opt comma] [opt 'S' to output data otherwise nothing is output] [opt comma] [opt String to test for]

where size is minimum size of field.

Normally this is just a TEST - ie nothing is output - but the 'send' parameter can be used to output the contents from the position specified IF THERE.

Note that both size and test are counters from 1 not 0.

A single chr can be tested to be non-space as in the example above.

If either the size or the test is FALSE, all text and subsequent data whether fixed or variable (including more Optionals) is ignored until the EndOpt flag is met - '\$O' (oh, not zero).

How is this used ?

 option:JN  SN,10
 before:StoryName is \SN \JN..and it is 10 or more chrs long at \$h:\$n \$Owhile the ..

If the SN field is less than 10 chrs in length - say SN:lily - the 'before' string will output :

 StoryName is lily while the ..

ie JN fails so all (text, FipHdr, Sys Variables etc) is stripped to the \$O

However an SN:bigbigbigbig gives a 'before' string of :

 StoryName is bigbigbigbig ..and it is 10 or more chrs long at 12:22 while the ..

(See a more involved example at the end of the REPEAT section.)

Taking four cases where an example field XB is :

  1. NOT present
  2. Present with no data
  3. Present with data
  4. Present with data including a space at position 3
Specify (no XB) XB: XB:111 XB:11 11
option:QH XB fail fail pass pass
option:QH XB,1 fail fail pass pass
option:QH XB,4 fail fail fail pass
option:QH XB,1,1 fail fail pass pass
option:QH XB,,1 fail fail pass pass
option:QH XB,,3 fail fail fail fail
option:QH XB,,4 fail fail fail pass

In a further example we can test the content of a field

option:QP   PR,,,,21

Here we do not care about size, position and output; only whether the field starts with the value '21'. Case (lower or upper) is always ignored. To test fields with spaces, special chrs etc, put double quotes around.

option:PF   XF,,,,\023\021
option:PZ   XR,,,,'PASNAP \$d'

Note that the data is compared ONLY for the length of the string specified in the 'option'. So if the FipHdr 'SS' contains '12345678'

 option:PS  SS,,,,1  - will be TRUE
 option:PS  SS,,,,12345678  - will be TRUE
 option:PS  SS,,,,123456789 - will be FALSE

So to test for an EXACT string, add a comma at the end of the field. (if you want a comma as part of the string, use FipSeq to encode it - \054 (octal))

If the string to test is NOT the first chr of the FipHdr field, put the stat posn in the 'test' field.

option:PS SS,,3,,34567

Test a string is NOT equal using the following syntax:

option:PS #SS,,,,12345

ie is SS is NOT 12345, PS will be valid

Test is a number is greater or less than a given amount - use 6th (a greater number) and 7th parameters (less than) to check

option:Q5   N1,,,,,21,14

N1 must be between 14 and 21 for Q5 to be true


REPEAT

Extract just a single section of a structured bit of data - such as an URL which has '/' separating the different bits

There are two different types of syntax for 'repeat' :

  1. with a particular chr as separator :

    repeat:QR XK,-,3

From FipHdr field XK, using '-' as sep, find the 3rd sub-field

Use this for FipHdr fields like :

 RR:engine-train-coach-seat-trolley

Syntax 1 repeat: [newfield] [tab/space] [newfield] [?] [existing field] [opt punct/comma] [single sep chr - possibly in FipSeq eg \002] [opt punct/comma] [opt field number from 1 or -1] [opt A for date-from-ALL-remaining-fields] [opt R for reverse]

Examples To get 'coach' field 3 repeat:R3 RR,-,3 To get the last-but-one field 'seat' (where you might have a variable number of fields, use '-' or 'r' repeat:R9 RR,-,-2 repeat:R9 RR,-,2r To get all fields from and including 4 ie R7:seat-trolley repeat:R7 RR,-,4a

  1. using a type of chr (or NOT a type of chr)

in this case the sep is left blank, and valid types are

  • u - uppercase
  • l - lowercase
  • x - alphanumeric
  • e - alphanumeric plus dot
  • t - punctuation
  • s - space (space, tab, ff etc)
  • n - number
  • p - printable
  • c - control

Syntax 2 repeat: [newfield] [tab/space] [newfield] [?] [existing field] [opt punct/comma] [blank, nothing] [opt punct/comma] [opt field number from 1 or -1] [opt punct/comma] [opt type of field]

eg repeat:QR XK,,3,U

To negate a type of field (ie NOT a number), use the '#' before the type - when you want to test if a byte is NOT a certain type:

 repeat:QR  XK,,3,#X

Here, QR is the 3rd field in the K field of SH separated by a non-alphanumeric chr

Use this for FipHdr fields like :

 repeat:QR  RR,,5,S
 RR:a319 Boeing-747 TigerMoth MothsBros *&amp;^%$ BingBang

So QR would be '*&^%$'

If there are a variable number of repeatable fields and you only want the last one, add 'REV' or a minus sign to the 3rd field to mean :

; to get the last field ...
; .. in W$:http://mighty-ducks.com/super-kwak/legs/tails/123456.html
repeat:AB   W$,/,-1

Take the last field of SN split by '.' : so if the filename (in the SN field) is ruby.tuesday.may.30.pls.ignore.oink, QX = oink

There is actually a third type of REPEAT code for use with OPTION where it is possible to output all codes :

eg In IPEDSYS, the keyword 'before' adds a String at the top of the data file :

before:<MODVER ID="1:00" VER="03">\n\ZU<DEST ID="1:05">\QU</DEST>\n\$O<FIFO ID="1:20"FFNO="25">\n
; add our option/repeat with a star ..
; if Fip Hdr field \ZU is specified,
; ..look for field \QU has a least 1 chr in it
; .. if there is, output to the \$O flag and then start again with a test for the next field
; .. if there is nothing, strip until the \$O.
option:ZU QU,1
; Hdr Field QU is really EACH sub-field of \NU separated by a comma.
repeat:QU NU+,+*

So if Fip Hdr field NU is : NU:HOK,DDD,NNR,MSV,SMF,XXX

The top of the file will have :

<MODVER ID="1:00" VER="03">
<DEST ID="1:05">HOK</DEST>
<DEST ID="1:05">DDD</DEST>
<DEST ID="1:05">NNR</DEST>
<DEST ID="1:05">MSV</DEST>
<DEST ID="1:05">SMF</DEST>
<DEST ID="1:05">XXX</DEST>
<FIFO ID="1:20" FFNO="25">

... and the rest of the data

NOTE : '\$' holds the contents of the current Repeat Value - (which should be the same as the FipHdr field in the repeat '\$%' holds the current Repeat seqno from 1 : 1st repeat = 1, 2nd is 2, 3rd is 3 etc '\$+' ONLY allow Repeat value and seqno for this level; ie the string containing (option)....(\$O) (default) Any repeats called inside thuis string should work fine - BUT \$ and \$% will be the NEW repeat '\$-' Allow Repeat value and seqno for this and all subsequent FipSeq calls until the \$O If you use this, any other Repeats which get called before the \$O will go wrong To use, place immediately after the option : repeat:G0 SN,_,* option:V1 G0 fixed:G1 \V1\$-seqno:\$%=\G0\$o '\$|' FlipFlow Repeat value and seqno for this level; ie the string containing (option)....(\$O) (default)

Note normally you cannot have more fixed, replace etc inside a Repeat unless the toggle '\$+' is specified. This is a toggle - so specify once to allow sublevels and then a 2nd time to disallow. Example: where K0=a+b+f+a+A ; unique - dedup the incoming data unique:K1 K0 ; repeat '' and option for the Loop repeat:K2 K1,+, option:V2 K2 ; 3 lookups on the value lookup:KA K2 file:F2MS key:3 value:7 wild: lookup:KB K2 file:F2MS key:3 value:10 wild: lookup:KC K2 file:F2MS key:3 value:6 wild:*

; use \$+ to flag we want the 2nd, 3rd etc ; use \$ to show THIS value fixed:KZ \V2\$+|\$^\KA^\KB^\KC\$o

++ NOTE - NEVER use \$+ with a 2nd repeat as it may lead to odd results ++


STYLE

Used for padding zeros or spaces (or any other padding chr)

Syntax style: [newfield] [tab/space] [existing field] [style definition]

eg to make a 4 digit number padded with leading zeros

style:ZZ    XN,%.04d

This is really just a passthru for the 'sprintf' command ! which is nasty but a standard (of sorts).

Do a man printf for fuller information if you need.

Note the single parameter after the old FipHdr ALWAYS starts with a '%'

If the expression does not end with an 's' ('d' for integer for example), then the string in the header field is first converted to that type.

Specify One and ONLY one expression (can not have %s%d%f) - as is takes the first only

Do NOT use for fixed data - use keyword 'fixed:' (as explained above)

Types are :

  • string : s
  • single char : c
  • integer number : d,i,o,p,u,x,X
  • floating point number : f,e,E,g,G
  • % : print a % !

Common Examples :

Description style defn
To trim a string, use a dot %.5s
To pad a string with spaces %5s
To pad a string with spaces (left justified) %-5s
To pad a number with up to 6 leading zeros %.06d

REPLACE

Copy the contents of one FipHdr field into another and then search and replace characters or strings.

Syntax replace:[New FipHdr] [spc/tab] [Existing FipHdr] [optional comma then flags to ForceCase, encode etc] [spc/tab] (Search String) (= or :) (Replace)

replace:LC  XC  SPO=s INL=i UDL=u ECO=f

There can be up to 50 searches/replace tuplets on a single line.

Flags -

By default case i ignored - but if the 'C' flag is specified, the Search String MUST always be in the right case. So always specify all combinations : SPO=s spo=s

The output can be forced upper or lowercase by specifying 'U' or 'L' after the existing FipHdr field. eg replace:LC XC,u SPO=s INL=i UDL=u ECO=f

Other flags can be set to:

  • Url Encode and/or Url Decode the data : u or e
  • Use a r'repeat character' to strip multiple consequetive occurances of the same chr : r=+
  • Use a single-character wild card : s=?
  • Use a start-of-block or end-of-block chr : b=|

    ; this will strip all spaces from the star and end of the zone, replace any combination of Yuummmy by YUM and heXXo with hello replace:LC AB,s=?,r=+,b=| \s+|="" |\s+="" Yu+m+y=YUM he??o=hello

Strings with spaces and punctuation MUST be enclosed by double quotes.

replace:K2 YR,u ".Y"=yellow ".C"=cyan ".M"=magenta ".K"=black

The replacement string may be in FipSeq - but NOT the search.

You must take care to only use small amounts of data - 1000 chrs max

Remember you must Never call a FipHdr field from itself !!


NEWDATE

Create a new internal FipHdr field with a date relative to when the program runs

Syntax newdate: [new FipHdr] [datetime differences] [style]

Style MUST be in double quotes. Style can contain both ordinary words and FipSeq.

The syntax for differences is

  • years + or - a number
  • months
  • days
  • hours
  • mins
  • secs

eg

hours+3     for plus 3 hours
days-7      for the same day a week ago

If you need to use a date which is relative to a date/time which is NOT the current date, put 'basedate=YYYYMMDDHHNNBB'. By adding a 'Z' for Zulu time on the end of the basedate string, then this basedate will be considered to be GMT; otherwise the time will be in the localtime of whatever the server is set to.

eg  Y0:20211130112233
newdate:A1  basedate:\Y0Z "\ZH:\ZF:\ZE"

If you need to output the date/time in GMT/Zulu time, add 'gmt'. newdate:A1 gmt "\ZH:\ZF:\ZE" newdate:A2 gmt hour-10 "\ZH:\ZF:\ZE"

The GMT parameter can be specified for input or output (default is output only if specified WITHOUT) but do NOT specify both in and out as they will cancel each other out !

    newdate:A1   gmt:in "\ZH:\ZF:\ZE"
    newdate:A1   gmt:out "\ZH:\ZF:\ZE"

If you want to recalculate against a time zone shift - such as GMT from 2012-09-28T11:33:52-05:00 - use BOTH basedate=(he date) and 'offset=yes' to take the offset into account. (Be warned that Fip - and a well-known supplier managed to get the sign of the zone wrong, so this can be used to sort it out !

There are no spaces between the number and the hours etc.

Several differences can be specified, each separated by a SPC or TAB.

The contents of ANOTHER fiphdr field may be used for the number :

    hours+\JR

Only the first (or for 'm' first two) letters are considered.

The default is a PLUS sign meaning in the future.

    'h3' is the same as 'hours+3'

The Style MUST be in double quotes. They can contain both ordinary words and FipSeq. Relevant header fields are

    * ZD    - 2 digit day of month (leading space)
    * ZG    - 2 digit day of month (leading zero)
    * ZM    - 2 digit month
    * ZY    - 2 digit Year 92
    * ZZ    - 4 digit Year 1992
    * ZW    - Day of week as in Monday, Tuesday etc
    * ZS    - 3 chr Day of week as in Mon, Tue etc
    * ZN    - Month as in January, February
    * ZT    - 3 chr Month as in Jan, Feb, Mar etc
    * ZJ    - Julian day of year
    * ZH    - Hour 00-23
    * ZI    - Hour 00-12
    * ZF    - Minute 00-59
    * ZE    - Second 00-59
    * ZU    - 1st, 2nd, 3rd, 24th for the day of
    * ZA,ZB,ZK - The three different week-of-year numbers (ZA is 0-53 starting Sunday, ZB is 0-53 starting Monday, ZK is ISO8601 1-53 where the first week has 4 or more days starting Monday!)
    * ZC    - Day of week
    * ZP    - AM or PM
    * ZL    - Daylight saving offset HHMM
    * ZQ    - Daylight saving difference in hours between system time and time calculated in newdate
    * ZR    - UTC offset from GMT in +/-HHMM format
    * ZX    - Unix time in Secs|Unix time in Mins

Note that actual Day and Month names depend on your LOCALE Default is "\ZW, \ZD \ZN \ZZ"

Example

newdate:JS  min-38 hour+8 day+\MD "xx\ZD-\ZS-\ZZ or \ZY \ZH:\ZTxx"

Where MD is a FipHdr field in the incoming file. Gives a result like xx01-Sat-2025 or 00 00:17xx

newdate:J7  "\ZS, \ZD \ZT \ZZ \ZH:\ZM:\ZE +0100"

Will give the current system date/time as an RFC2822 string Gives a result like Fri, 15 May 2019 17:58:28 +0100


UNIQUE

Control the contents of a single FipHdr field so that :

  • each element is unique
  • each element is separated by a separator

Syntax unique:(new FipHdr Fld) (space) (old FipHdr Field) [opt (punct/comma) (separator) ] [opt (punct/comma) (force Upper or Lower case) ] [opt (punct/comma) (output separator) ]

The Separator defaults to a plus sign '+'.

unique:Q1   F1,|
unique:Q2   XC,=

So if XC (ie the C field of SH) was :

SH:N1234:Sabc:Ccat1=cat1=cat3=cat2=cat3=cat4:P3

Q2 now becomes

 Q1:cat1=cat2=cat3=cat4

If a separator is specified, ONLY that is used to divide the zones. Otherwise, any space, plus sign '+' or comma ',' is considered a divider. Multiple separators between zones are just ignored. If you want the Sep to be a comma, use another punctuation chr after the FipHdr field. eg

 unique:Q1  XC,+,

If you want a different Output separator, specify as the third parameter eg :

 unique:Q1  XC,,,$

So if XC (ie the C field of SH) was :

SH:N1234:Sabc:Ccat1 cat1,cat3+ cat2 cat3 , cat4:P3

As no input separator has been specified, the elements are split by Space, comma or plussign Q1 now becomes

 Q1:cat1$cat2$cat3$cat4

VALID

Create a new FipHdr by checking the contents of another against a standing list of valid entries.

*Syntax** valid:(new FipHdr Fld) (space) (old FipHdr Field) [(opt) (comma) (flag - force Upper or Lower case) ] (list of values)

The list of values are separated by one (or more) spaces/tabs.
So if any of the values contain a space or comma, it must be inside double quotes

Optionally a flag can be specified after the Old FipHdr field which will force the resulting data :
- U uppercase
- L lowercase
- I idicase - or camel-case where the first letter of each word is capitalised
- F upperFirst or initial case
- C data to match is case-sensitive.
- ? a single wild character (this can be any punctuation except '",)
- X if there is NO matching value, set the FipHdr to nothing
    - (Normally If there is NO match, the first value in the list is used as a default.)
- V if the data in the FipHdr has multiple values, output then in Valid-List order, NOT the current order of fields in the FipHdr
- (a number from 1 to 9) if the data in the FipHdr has multiple values, output JUST that number of items

 eg valid:Q1    XP 4 1 2 3 5
 Create Q1 from XP which must be a value of 1 to 5 and defaults to 4
 eg valid:Q1    XP,X 4 1 2 3 5
 Create Q1 from XP which must be a value of 1 to 5 and there is no default value, so Q1 will be blank in that case

If you do NOT specify ',X' and there is no match, the first entry is used as the default unless the FipHdr field does NOT exist or has no data - in which case there is no default. So if you ALWAYS need something, use a 'combie' to make sure there is some content in the data-to-be-validated.
eg combie:Q0    AY,XX
valid:Q1    Q0  XX SE DK NO FI BX "Z "

; allow any value starting MCC: with one or three trailing chrs
valid:P1    TR,?    MCC:? MCC:???

; note that the values may also be a variable in FipSeq - good practice is to make sure there is ONE value in the list !
combie:P0   PZ,first
valid:P1    TX  second \P0

Note that the original VALID only allowed for a single item of data-to-be-validated. But from April2014, the data zone can have zero, one or many items - but only the valid ones are left afterwards.

LOOKUP

Create a new FipHdr by matching the contents of another against a standing lookup file.

Note this is for simple, small lookup files NOT massive tables of thousands of entries.

  • Use fipseq REPLACE for a small number of fixed entries - say up to 30.
  • Use fipSeq LOOKUP for between 30 and 100 entries (or if the file needs to be maintained externally).
  • Use program IPLOOKUP for anything else.

Syntax lookup:(new FipHdr Fld) (space) (old FipHdr Field) (series of sub-parameters ... )

The old FipHdr field will contain the key to search for. SubParameters are :

  • file:(filename - no default,; this field is mandatory)
  • either key-field: (field number of the key field - starting at 1 - default 1 for 1st)
  • or row-number (number row or line - starting at 1 - default : use key-field)
  • value-field: (field number of the data or value field - starting at 1 - default 2 for 2nd) the value field may be a range for consequetive fields value=3-7
  • default-value: (FipSeq - default is nothing) use this if the key is NOT found
  • blank-value: (FipSeq - default is nothing) use this if the key is found, but there is no data in the value field
  • separator: (Fipseq chr - default is '|')
  • type-format: (filetype - CSV, TAB-sep, Fixed width - default is blank for TEXT)
  • comment-line: (character which signifies a comment line - default is ';')
  • numeric-key: signifies the key is numeric.
  • sorted: yes/no is the file sorted ?
  • limit: (2 to 20) max number of fields on a single lookup line
  • case-sensitive-key: yes/no ie Cardiff or cardiff
  • allow-spaces: normally leading and trailing spaces in both the key and searched-for zone are removed before testing.
  • missing-msg:"Missing value " : log message if there is no entry for this value
  • only-allow:(char which starts line of only those lines to check) This is the opposite of comment-line as ONLY lines starting with this chr will be scanned.
  • wild:(chr) allow keys to end with this wildcard chr. This allows the data to have short keys.
  • rpt-sep:(FipSeq chr - no default) Separator between multiple input keys
  • error-code: (single lowercase letter for log type if the lookup file does NOT exist, flag the message as this type; a dash will mean : do NOT log)

Limit example - use 'limit' where the separator chr may be in the last field eg

; data in lookup file
text-top:Some :rubbish: to be added at the "top: of the file

.. and in the parameter file

fixed:YC    text-top
lookup:AL   YC  file:/fip/tables/setup/RUBBISH key=1 value=2 limit=2 sep=:

Wild example for short keys :

In the parameter file   wild:%
In the file car%|4wheels|petrol
will match data with 'car alarm', 'carriage', 'cart'

Error-code example :

error-code:w will give a '!w' type in the item log
Fri Jun 26 12:26:40 ipxchg !w : ** FipSeq - No Such LOOKUP file /fip/web/preferences/SPECIALIST

General examples :

lookup:Y1 YP file:Categories sep:| comment:# key:3 value:5
Create Y1 from the entry in file /fip/tables/setup/Categories which matches the contents of XP where the key is in field 3 and the value to use is in field 5.

If using Fixed width fields, the width of each field follows the 'F' in type-format.
 eg lookup:Y2 SU file:services key:1 value:4 type-format:F,6,6,2,2,3
 Create Y2 from the entry in file /fip/tables/setup/services which matches the contents of SU where the key is in field 1 (which are characters 1 to 6) and the value to use is in field 4 (characters 15 and 16) .

Where the old FipHdr has more than one key - and you need to lookup ALL of them, then specify the Rpt-Separator - usually a spc (as it is unprintable, use \s) The result is a single FipHdr with all the new codes withe the same separator.
eg  F1:AAA BBB CCC DDD
    and the lookup file maps
        ; comment here
        AAA^ZZZ
        BBB^YYY
        DDD^XXX
    lookup:F2 F1 file:/fip/tables/setup/MYMAP sep:^ comment:; key:1 value:2 rpt-sep:\s
    gives   F2:ZZZ YYY XXX
    (note CCC is missing, so only 3 are returned)

PERL

Create a new FipHdr from a Perl Regular expression or an external script (or program)

Syntax 1 perl:(new FipHdr Fld) (space) (Perl RegExp)

Syntax 2 perl:(new FipHdr Fld) (space) (path to any script or program plus any subparameters in FipSeq)

Note that all '\' should be doubled as the RegExp string is considered to be FipSeq, so other FipHdr fields may be included.

The RegExp MUST be on a single line and you are advised to use a 'print' to display the result you want.

Any NewLines and/or Carriage Returns are mapped to Spaces.

Examples

 perl:E1 $seqno = "\VD"; $bigSeq = 24*$seqno; print "Seqno is $bigSeq";
 Create E1 with (if FipHdr VD:3) "Seqno is 72".

the second format is to run a script (of any variety)
perl:AB /fip/local/redoAB.php \QU \L0

Take care that the syntax is correct and does not have an infinite loop!

Take care to respect Perl quotes - especially for Winnt where for some bizarre reason we need two double quotes around it !

eg for Winnt

perl:Z4 ""@hoho = 29 * \F4;""

MERGE

Create a new FipHdr from a on or more occurances of another FipHdr field.

Normally, it a FipHdr field is repeated, only the last FipHdr fields is accessible - which allows you to overwrite entries quite happily.

But there are some (few) occasions where in fact you want the first or second or maybe all of them concatenated together in one long string. This could be when you run 'ipxchg' against another formatted file which has duplicated entries for some fields.

Syntax merge:(new FipHdr Fld) (space) (old FipHdr Fld) [(optional) , Separator] [(optional) , First-occurance] [(optional) , NumberOfTimes] [(optional) , T-op, B-ottom, R-everse]

In the above syntax the comma may be replaced by any other punctuation except '$'. The Separator is to divide two or more fields and can be in FipSeq and can be a TAB (\t), SPC (\s) or any printable chr. The First occurance is counted from the begining or top of the FipHdr. The NumberOfTimes is the number of fields to take into account.

The Top or Bottom flag denoted if the occurance is from the Top of the FipHdr (default - first ones) or the Bottom (last ones). While Reverse will sort the output backwards (LIFO rather than the default of FIFO). Specify ONLY a single chr T or B - or the R or nothing.

eg if the FipHdr had :

 * AB:blue
 * AB:green
 * AB:red

merge:BC    AB,+
gives a new Fiphdr field of BC:blue+green+red

SUM

Create a new FipHdr from the result of a calculation and/or output in decimal, hex, octal, base36 or base64

Syntax sum:(new FipHdr Fld) [: optional flags] (space) (the calculation)

The calculation is in FipSeq - so can include other FipHdr fields

Calculations can NOT be split over several lines.

The default precision for a calculation is 2 decimal places. This can be overridden using the syntax :

sum:AB  100*(\Q1/\Q2)

Flags can be :

  • (digit from 0-6) precision ie no of decimal places (default is 2)
  • R round to nearest decimal (default is NO rounding)
  • O output in Octal (default is decimal)
  • H output in Hex (default is decimal)
  • B output in Base36 (default is decimal)

Operators can be :

  • plus '+'
  • minus '-'
  • multiply '*'
  • divide '/'
  • mod '%'

Take care when dividing by zero ! so use a 'COMBIE' and 'PARTIAL' to make sure.

Use round brackets to denote how the calculation should be worked out. Ie the deepest level is calculated first, and then the calculation is worked from left to right.

For example : (\Q1/\Q2)*(\Q3/(\Q5+\Q4))/100

.. will add run through the following order :

 * Step 1   (\Q5+\Q4)
 * Step 2   (\Q1/\Q2)
 * Step 3   (\Q3/result of Step 1)
 * Step 4   (result of Step 2 * result of Step 3)
 * Step 5   (result of Step 4 / 100)

FILTER

Create a new FipHdr by filtering the contents of another against a mask or a list of valid entries.

Syntax filter:(new FipHdr Fld) (tab/spcs) (existing FipHdr field) [: optional flags] (tab/spcs) (list and/or mask)

Some XML schemas like NewsML G2 allow for a qualified metadata.

Use FILTER to reduce the number of multiple entries in a FipHdr file eg

; allow any value starting MCC: with one or three, but NOT two, trailing chrs
filter:P1   TR,!    MCC:! MCC:!!!
filter:P2   AC  Sports News Business
  • Entries may be dbl quoted if spaces are needed.

  • Flags are specified in order

    • first is a single wild character (default is ?)
    • second is a wild string (default is *)
    • third is the separator - default is tab or space(s) - this may be FipSeq eg \034
    • fourth are case flags : 'U' uppercase, 'L' lowercase, 'F'-firstCap, 'I'
  • To Exclude items (filter normally just INcludes), add a HASH before the old Fiphdr field

    filter:P3 #TR,,%,+ qcode:%

This will filter all in the TR FipHdr field, (separated by '+') which do NOT start 'qcode:...'


RECODE

Create a new FipHdr by recoding the contents of another

Syntax recode:(new FipHdr Fld) [: optional flags] (tabs/spaces) (existing FipHdr field),(input coding),(output coding),[optional output size],[opt pctencoding Safe chrs]

- a blank coding means take as ordinary 8 bits chrs
- for input, it is the contents of the string
- for output the default is UTF8

NOTE this is really for strings so beware about binary data. eg

; take the contents of P0 and put a base64 encoded version in P1
recode:P1   P0,,base64,32
; take the contents if W$ and urlencode/percent-encode it for P2
recode:P2   W$,,pctencode
; urlencode Q5, EXCEPT these safe chrs, and populate Q6
recode:Q6   Q5,,pctencode,,,-_.~=&
; convert the contents of AN - which is in base64 - and create P3 with the same contents in hex
recode:P3   AN,base64,hex
; J1 is the length of JG
recode:J1   JG,length

Codings are :

  • ascii (or binary)
  • utf8
  • base64
  • qp (quoted printable)
  • percent-encoded (also called url encoding)
  • hex (2 byte encoding of a single chr)

A special 'coding' is L - Length Only - which returns just the length of the string

These 5 can be optionally Upper or Lowercase

  • B base64
  • H hex
  • P percent encoded (same as url)
  • Q quoted printable
  • U url encoded (same as percent)

Force the input string to be a Number: These 4 can be optionally big or little endian or 'whatever the system is' endian

  • int8
  • int16
  • int32
  • int64

For Pct or Url encoding, the default 'safe chrs' (ie ones that are NOT mapped to %XX format) are

  • dash '-'
  • underscore '_'
  • dot '.'
  • tilde '~'

Use the last parameter to overwrite the default - you must specify all in this case !

(see https://tools.ietf.org/html/rfc3986 or https://en.wikipedia.org/wiki/Percent-encoding for more information)

INCLUDE

Include data from an external file when this FipHdr field is specified

Syntax include:(new FipHdr Fld) (existing FipHdr field) [optional (tab/spaces) pathname:(fipSeq pathandfilename)] [optional (tab/spaces) pathname:(fipSeq pathandfilename)] [optional (tab/spaces) pathname:(fipSeq pathandfilename)]

    - if the new FipHdr field is encountered - in a string or a parsed file - it will include the contents of the file pointed to.
    - maximum size of the data is under 62k
    - if a pathname is specified, then it holds the full path and filename of the data
    - if NO Pathname is specified, then the OLD FipHdr field should have the full path and filename
    pathname:(filename in fipseq)
        If the old FipHdr does NOT have the full pathname for the data file
    default:(filename in fipseq)
        replacement data if there NO oldFipHdr or NO file
    missing-msg:"Missing include file for SPORTS TEAMS "
        log message if there is no entry for this value
    error-msg: (flag chr) or '-' for no error default is x for !x error messages
    eg: FS:SOCCER_TEAMS.LOOKUP
        include:F1  FS  template:/fip/tables/setup/\FS
        FF:/fip/tables/setup/SOCCER_TEAMS.LOOKUP
        include:F2  FF
    - specifying \F1 or \F2 in a fipseq string will output the contents of that file at that point
        before:Soccer Teams is :\n\F1\n

Version Control
; 38k5  ;a 20may19 bug in partial/UTF8 ;b TooMuchData (> 64k - current max) ;c minor
    ;d allow fipseq_extras - % ;e 26oct20 minor prototypes ;f 21dec20 added partial : json and xml
    ;g 10mar21 filter/NOT issuette plus lookup/only-allow multiple chrs
    ;h added partial type SLUG 'g'
    ;j 20mar22 allow \%x for hash
    ;i-k 28feb22 BUGlet lookup rpt-sep output sep being ignored ;k1 macosx ;k2 started FIPSTORE
; 37z   ;a 25may15 added Lookup blank-value and allow range value=3-7
    ;b 19jun15 perl/script and Win2012 ! why do Microsoft have to change things ?
    ;c 15jul15 added lookup error-code:w to make it a warning
    ;d  5apr16 maxFilters 100->250 as Reuters are adding 160+
    ;e 16may16 bug in unique when single FipHdr can be > STDBUF
    ;fg 22aug16 issuette in option (not serious)
    ;h-k 12nov16 added lookup/row and lookup/fixed with spaces
    ;l-n 27jul17 added 'recode' and \\$p = FIP_startTime (lib/prog)
    ;o 5feb18 added newdate .. tz=Australia/Brisbane ..
    ;p 1apr18 added recode-string to number and bugette in new 'tz' code
    ;q 7may18 added fipseq_extras for $T and $1 and ...
    ;r 5sep18 buglette in lookup - rpt-sep
    ;s 10sep18 added \@x as a Program-dependent variables (mainly sr_PSYS calls)
    ;t-v 25sep18 added recode/pctencode - safe
    ;wxy  6jan19 added include:
    ;z pp_trim_spaces - trim dbl/single quotes around plus better filter control
; 36z   ;a 12mar12  PARSE_FipHdrSize->HDRBUF-STDBUF ;b allow valid items to be parseable
    ;c PARSE_FipHdrSize->FIP_limitHdrFieldSize
    ;d tuned unique
    ;e 21dec12 strip some duplicating error messages for lookup etc
    ;f 26mar13 allow A$ in combie
    ;g  5jun13 added 8bit and 7bit for partial
    ;h 11jul13 added urlencode and urldecode to replace
    ;i 23aug13 added REPLACE - singlewild, repeatChr, blockstart/end
    ;j 30aug13 PARTIAL - 8=utf8
    ;kl 16sep13 bugettes in Filter when item found at end of line with multiple seps after
        and Replace if nofh and added negate to Filters : filter:W1 #RI,*   R:*
    ;m  7oct13 added 'all' to repeat meaning this field and the rest of the string - ; to get the path of a file : repeat:F3 FN,/,-2all
    ;n 28jan14 woops bugette in unique
    ;o 26feb14 bugette in replace if found at end of buffer
    ;p 4apr14 tuned perl:
    ;q-s 14apr14 Valid - allow more than one element, added output order and number, newdate vc2005
    ;t-u 10jun14 buggettes in newdate and valid
    ;v 10oct14 tune replace (do NOT allow a:b)
    ;w 22oct14 added ZL ZQ ZR to newdate for DST and UTC offsets
    ;x  2dec14 added output sep for UNIQUE
    ;y 31mar15 redid merge and added reverse
    ;z 28apr15 added limit (ie max no of fields for lookup

(copyright) 2024 and previous years FingerPost Ltd.