ipxchg
IPXCHG translates the Character Set and any Style changes that are needed to
process any incoming file. It uses a parameter file, in tables/xchg, to define
what the processing should be and how it should be done.
The incoming file is normally passed on by 'ipwheel' as part of the pipline to
get the data to the destination. IPXCHG can also be run as a once-hit against a
particular file.
In order to determine what parameter file to use, 'ipxchg' uses FipHdr fields,
if they exist.
This can be overridden by the '-D' input switch which will force the xchg to be
that name.
Otherwise the 'CX' field is first looked for.
If there is no 'CX', the 'SC' field is compared to the 'DC' field. If they are
different, the xchg file used will be SC2DC.
ie if 'SC:AP' and 'DC:HTML', then the xchg parameter file will be 'AP2HTML'.
If there is NO SC, DC or CX and NO -D input switch, a string at the beginning
of the filename (up to the first HASH/POUND SIGN) can be used.
The actual parameter file is in tables/xchg and the filename will be forced
Uppercase.
Syntax for the xchg parameter file held in tables/xchg :
one parameter per line
; Comments lines may be inserted at any time and are preceeded by a ';'.
: Comment lines can also be preceeded by a colon in this case
(type) [ (flag) ] (delimiter) (1st parameter) [ (delimiter) (2nd parameter)
...]
where items in [] are optional
- type is a CASE SENSITIVE type of parameter - c, D, x etc
There should be NO leading spaces/tabs. To aid legibility,
other letters and spaces may be put AFTER the type but take
care NOT to use letters or punctuation, eg these are equivalent:
chrset :ascii
and c:ascii
- A flag is any number between 1 and 100. It allows conditional
xchgs to be done. Ie the flag must have be set ON using the 'o'
command beforehand or the line is ignored.
- Delimiter can be any punctuation EXCEPT backslash or Hash.
it must also be consistent :
z:SPC:after
z_SPC_after
- Parameter string is any FIP sequence of fixed text,
Octal numbers (00 format) and Unix escape chrs (n etc).
In some cases - eg 'r' for Add FipHdr - the data can be any
FipSeq string including System Variables and existing FipHdr
fields.
The first parameter string is called the TO or SEARCH field.
The second parameter (if there) is FROM or REPLACE field.
There are no mandatory parameters - the defaults are :
c: ASCII Chrset is ASCII
b: 8 : 8 Input stream is 8 bit chr and output is 8 bit
Default is : No escapes, xchgs or other mutilations.
List of Types
_____________
Syntax Example
------ -------
x, X - An xchg; from is the source string, to is the replacement string.
The From or search string can be 1-255 chrs long; the replacement string can be
0-255 chrs long.
x:fromString:toString x:+++:bong
The difference between 'x' and 'X' is :
'x' the search string is case SENSITIVE
'X" the search string is case INsensitive
ie X:head:hhh
will replace 'HeAd' or 'HEAD' or 'head' etc with 'hhh'
c -This refers to parameter file tables/chrset/PA
(see below for format of this external chrset file)
c:(main chrset) c:PA
There are a number of hard-coded chrsets which do NOT need to be in chrset:
c:ascii 7 bit ascii
c:isoascii 8 bit ascii
c:ascii2baudot automatic conversion from ascii to baudot
c:c2let,c2num the two baudot sets
c:utf8 8 bit unicode - see doc on utf
c:utf16 16 bit unicode - see doc on utf
c:ucs2 16 bit unicode (Microsoft's notepad/Sql Server version)
Multiple chrsets can be defined as comma separated names :
c:PA,PA2,PA3
There can be up to 10 chrsets.
or if only one alternate set needs to be defined :
C - This is the alternate chrset - for 5 or 6 bit sets for example
C:alt chrset C:c2num
So these two are equivalent :
c:c2let,c2num c:c2let
C:c2num
h - This is the header chrset. Parameter z:chghdr must be specified for the
fields to translate (see below). If there is no 'h:' the main chrset is used.
h:hdr chrset h:swhdr
Please do NOT use characters NL, Tilde (octal 176) and Colon in Header fields
They will be moved to octal 371, 372 and 373 respectively if you do !
e, E - Escape chrs to the alternate and back to the main chrset for
multi-chrset
For example where multi-layer chrsets such as BAUDOT and TTS are used,
e:escape chr from main e:33
E:escape chr from alt E:37
Note the default is to pass these chrs through - use z:ZAPESC to strip.
OR these can be specified as
e:(Esc chr):(Number of chrset)
where the number is the chrset in the 'c:' line above :
c:PA,PA2,PA3 PA is 1, PA2 is 2 and PA3 is 3
there can be up to 20 escape chrs. The actual chr MUST be unique so octal 005
cannot
escape to both PA and PA2 at the same time. But more than one escape chr
can point to the same chrset.
Escapes MUST always be specified lower down the parameter file than Chrsets.
s, S - Escape chrs for a single Chr from the alternate and then revert back to
the main chrset for multi-chrset
For example where multi-layer chrsets such as ISO-7901 in its many guises.
s:single chr escape from main s:&
S:single chr escape from alt S:&
Where we want to escape to a secondary chrset for a single chr.
Note the default is to pass these chrs through - use z:ZAPESC to strip.
d, D - Delete all between from and to strings.
Lowercase d strips BOTH FROM and TO strings, Upper D strips only the FROM.
d:ignore from:to d:<:>
D:ignore from:to D:<MDNO:<MD
i, I - Ignore all from the beginning until this string
Strip all from this string to the end of file
From version 47l conditional flags are NOT ignored.
i:Ignore Upto string i:02
I:Strip all from this string I:03
Note that this string is the RAW data -
ie NO chrset translation has been done beforehand
In either case, if the string is NOT found, then data is NOT stripped.
m - MultiByte chrs for UTF/Unicode
m+u2012+-
m+u2013+{endash}
m+u2014+{emdash}
One common one is the Euro :
m+u20ac+EUROs
a, A - Add text at beginning or end
This text is ALWAYS added and any conditional flags ignored.
a:String for begining of file a:{m1Top Table{m0n
A:String to end file A:r03NAME $d-$m-$y04
From version 44m onwards, up to 100 adds may be specified.
a:Top linen
a:Line 2n
a:Line 3n
From 44w, both After and Before Adds may be dependent on a flag
A +n(always)
A #22+n(not 22)
A #33+n(not 33)
A 22+n(yes 22)
A 33+n(yes 33)
A +nbibin
f, o - Set flag OFF or ON - see below for more information on flags
To set flag on or off for conditional Xchgs.
f:Search string:Flag number f:rnREU:23
o:Search string:Flag number o:NNNN:23
The flag is turned ON/OFF immediately AFTER the string
The string itself is left untouched - use an 'x:' line to strip that.
Uppercase 'F' and 'O' may be used for case-INsensitive search strings.
p, P - flip-flop a flag ON and OFF
To set flag alternatively on then off for conditional Xchgs.
p:Search string:Flag number p:*T:44
The flag is turned ON/OFF immediately AFTER the string
The string itself is left untouched - use an 'x:' line to strip that.
Uppercase 'P' may be used for case-INsensitive search strings.
k, K - Kill or delete characters either before or after the search string
k: search string:No to delete before k:rn:5
K: search string:No to delete after K:RHUBARB:93
The string itself is left untouched - use an 'x:' line to strip that.
r - Add a new Fip Header Field after processing has finished.
The SEARCH file holds the Hdr field name and the REPLACE field
holds data. Note that any FipSeq data can be used but all
FF, CR, DEL and NL are replaced by '_'.
r-TI-Time is $h:$n:$b
Several 'r's may be specified. The new fields are added at the end
of the Fip Hdr so they may be used to override existing Hdr fields
but will in turn be overridden by any Hdr fields in the filename.
Note the '-' used as separators as ':' is used as part of the string
y, Y - Yank or save text between from and to strings
Lowercase y saves from the end of the FROM field to the BEGINNING
of the TO field, while Uppercase Y saves to the END of the TO.
y:[ptsize is:n
Y:<save>:<unsave>
u, U - UNYank or unsave text previously saved using 'y' or 'Y'
Lowercase u adds the text at the BEGINNING of the FROM field,
while Uppercase U adds it AFTER the FROM. The TO field is ignored.
u:<add before this>
U:=AddAfter=
w, W - Allow wild character and strings
The first character of the Lowercase w is the wild CHARACTER ie
a single character, while the first character of the Uppercase W
is the wild string.
w:? wild chr is '?'
W:* wild string is '*'
Note that only one wild string is allowed in the TO or SEARCH string
of any test and it can NOT be the first chr of the SEARCH string.
There is no restriction on the number of wild Chrs in either search
replace strings. When used in REPLACE the saved chrs are output
in the same order as found.
The maximum number of chrs and string is 1024.
; wild CHR is '?'
w:?
; wild STRING is '*'
W:*
x:ABCD???HIJ:ab?fg*
An incoming string of 'ABCDEFGHIJ' will be stripped to 'abEfgFG'
Note that you should make sure that the two wilds are not in
any other SEARCH or REPLACE string anywhere else in the
parameter file or you will get funny results :
W .*
x .[p*].<ptsize=*>
x .@*&%$.abcde
will probably NOT give the required values for the second, fixed string.
Note that by default the wild card/string is positional - use it and lose it.
To use the Wild string as a full string each time - as in the example :
x .[cp*].[cp*][cl*]
use the option 'z:ALLWILD'
To use the contents of the wild buffer in a later xchg line,
use the option 'z:KEEPWILD'
z - Setup Option
z:option z:FCC:
Options include:
FCC idiot upper and lower-case. Note you may force an
Uppercase letter by using a '~' as a precedent chr.
NUL do NOT strip NULs ie binary zero chrs
NULS are normally deleted from the output automatically
TOPSPC strip all spaces, tabs, carriage return and line feeds at the top of
the file
BOTTOMSPC strip all spaces, tabs, carriage return and line feeds at the
bottom of the file
EOLSPC strip all spaces and tabs BEFORE the end of a line (all other spaces
on the line are preserved)
An end-of-line is either CR and/or NL chr - but optionally a single
different character may be specified
eg z:eolspc:222
SPC or SPC:before strip DUPLICATE spaces BEFORE xchgs
SPC:after strip DUPLICATE spaces AFTER xchgs
LINE or LINE:before strip DUPLICATE newlines/line feeds BEFORE xchgs
LINE:after strip DUPLICATE newlines/line feeds AFTER xchgs
IGNORE-DATA Ignore the data part of the file.
Any fipHdr and before/after are processed but all data from the file is
stripped.
ABSTRACT-DATA:(number) Only preserve the first 'n' characters
Any fipHdr and before/after are processed but all data AFTER the first 'n'
are stripped.
FIPHDR-IS-DATA-TOO Allow xchgs on the whole of the file
including the FipHdr part!
NOFIPHDR or NOHDR Do NOT add the FIP header to the data.
CHGHDR Also replace header fields with this chrset. This
defaults to the Source Header only (SH). Other fields
can be specified by listing after this flag. eg :
z:chghdr:QA,SH,DI
The header chrset can be specified by the 'h:' parameter
or will default to the same as the primary chrset as
in the 'c:' parameter. Note that if the fields also
exist in the filename, they are also changed.
Note you cannot replace individual subfields of the SH -
like XP, XF etc. Use NEWHDR to make a copy of that
subfield first and then chghdr it.
z:newhdr:AP XP
h:anpapriority
z:chghdr:ap
CONVERT-FIPHDR: (options - comma separated if there are more than one)
eg: z:convert-fiphdr:utf8,map
convert the FipHdr field(s)
UTF8 Convert the COMPLETE FipHdr to UTF8
MAP Convert the field(s) in z:chghdr:xx using the map-unicode file specified
Eg if the incoming data is Russian (CP1251) variety, put the unicode map in
tables/unicode and :
z:chghdr:IH
z:unicode-map:CP1251.TXT
z:convert-fiphdr:utf8,map
Note you will always want to specified UTF8 too for this option
ENTITIES:Quark
ENTITIES:InDesign
ENTITIES:Dec
ENTITIES:Hex
ENTITIES:Rtf
Convert to entities.
TRIM-HDR-SPACES: yes/no
remove all multiple spaces and any spaces at the beginning and end of all
FipHdr fields
HDRONLY Only process the FipHdr fields - leave the data alone - see below.
WORDMAC If the file is in the Mac version of the Microsoft Word format, the
header and trailer are stripped, leaving just the text section. This works for
Versions 5 and 6 only.
An optional parameter may be added to STRIP the data if NOT Word (eg XLS)
WORDPC If the file is in the Mac version of the Microsoft Word format, the
header and trailer are stripped, leaving just the text section. This works for
Versions 6 thru 9 (ie all WordBinary formats from 48f35).
Optional parameters may be added to STRIP the data if NOT Word (eg XLS) and
Save Meta as MA-MS: fiphdr fields
z:wordpc:strip
z:wordpc:strip,meta
Unless z:nohdr' is specified, a FipHDr of M9 will be the numeric part of the
code page for the data eg MZ:65001 for UTF8
For the Meta-Fiphdrs :
MA - just the numeric part of the code page of metadata eg MA:1252 for
Latin1
MB - Title
MC - Subject
MD - Author
ME - Keywords
MF - Comments
MG - Template
MH - Last Saved By
MI - Revision Number
MJ - Total Editing Time
MK - Last Printed
ML - Create Time/Date
MM - Last saved Time/Date
MN - Number of Pages
MO - Number of Words
MP - Number of Characters
MQ - Thumbnail
MR - Name of Creating Application
MS - Security
- From DocumentSummaryInfo ..
NB - Category
NN - Manager
NO - Company
ZAPESC For escaped chrsets - such as 5 level baudot or 6 level
TTS, delete/ignore the escape chrs.
ALLWILD Use the whole of the wild string in a REPLACE, not just
the remainder.
KEEPWILD:
Normally the wild card buffer is only available for
the one Xchg line and then zapped after use.
This allow it to be used later on.
PARTIAL new FIP header field created from existing field
z:partial:qc XK,3,8,u
ie To make QC, take K header field from the Source Header (SH)
posn 3 for 8 chrs, UPPERCASE.
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to SysAdmin FipSeq doc for more information)
FIXHDR additional FIP header information
z:fixhdr:QZ 1234543
ie If QZ is specified, replace with 1234543
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to SysAdmin FipSeq doc for more information)
OPTION test a FipHdr field before outputting
z:option:QR ZZ,3
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to SysAdmin FipSeq doc for more information)
REPEAT create a new FipHdr field from an existing field
z:repeat:QR ZZ,-,3
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to SysAdmin FipSeq doc for more information)
COMBIE create a new FipHdr field from one of two existing fields
or use a default
z:combie:QR ZZ|ZR,(defaultpls)
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to SysAdmin FipSeq doc for more information)
STYLE: create a new FipHdr field from an existing field and
pad it with spaces, zeros or justify it using
a 'printf' type style
z:style:QZ XZ,%.04f
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to SysAdmin FipSeq doc for more information)
REPLACE: create a new FipHdr field from an existing field and
replace one or more strings/chrs.
z:replace:qK XK "BC-"=""
(refer to SysAdmin FipSeq doc for more information)
NEWDATE: create a new FipHdr field containing the date offset from
the current datee and time
z:newdate:QD days+3 "ZH - ZD"
This field is only used internally and NOT stuffed in
the output file unless saved with NEWHDR.
(refer to SysAdmin FipSeq doc for more information)
SUM:, VALID:, LOOKUP:, UNIQUE:, FILTER:, MERGE:, PERL:
(refer to SysAdmin FipSeq doc for more information)
NEWHDR: create and SAVE a new FipHdr field.
Any FipSeq can be added.
z:newhdr:JF FormatIs.XF
z:newhdr:JN XN+XS=SN
This field is added to the FipHdr at the start of
processing and can therefore be 'hdrXchged' etc.
(refer to SysAdmin FipSeq doc for more information)
Please note that any FipSeq (z:REPLACE: z:COMBIE: etc) MUST be
specified first BEFORE any NEWHDR lines in the parameter file.
TEXT2FIPHDR: Extract a line at the top of the file and make it a
FipHdr field (and normally/optionally remove from text).
A line is ended by any combination of CR and NL
This is done on the RAW, incoming data,; not the output.
At the top of text :
This is done on the RAW, incoming data,; not the output.
At the bottom of text :
Specify with a negative line number : -1 for the last, -2 2nd last etc
Lines are counted AFTER z:bottomspc and and Strip from (I:)
If there is a trailing NL, CR or CRNL, they are ignored
syntax z:text2fiphdr: (line number):(fiphdr):(optional flags)
eg z:text2fiphdr:1:DI:no
The optional flags can be :
yes - (stripfromdata) - also delete that data
or no - do NOT strip
or xml - ignore any xml tags and attributes and only get data
or html - ignore up to <body> or </body>
The default strip is YES. There can be 100 z:text2fiphdr lines
(note you cannot strip XML)
REDUCEBEFORE: (chr)
REDUCEAFTER: (chr)
Reduce all multiple occurances of a single chr to a one
The character can be in FipSeq - 31
reducebefore acts After the 'chrset' and Before any xchgs
reduceafter acts after all processing.
There can be up to 100 reducesbefores and afters.
WRAPTEXT: (line length) : (End-of-Line string)
Wrap text after all processing to a specified number
of chrs per line. Add the End-of-Line string for long
lines.
The default line length is 64.
The default end of line is NL or n
IGNORE-XML-IN-WRAP: yes/no
Ignore any tags within <...> in calculating wordwrap
default: no for leave as part of the calculation
IGNORE-ENTITIES-IN-WRAP: yes/no
Ignore any entities within &...; in calculating wordwrap
default: no for leave as part of the calculation
WRAP-LONG-WORDS:(length)
add a space after this number of characters to wrap long words
eg z:wrap-long-words:77
default: no
NUMBER: (octal/decimal/hex)
Change the number system for FipSeq numbers to this
The default is octal.
Note this takes effect for ALL parameters in the file.
FILEBEFORE: (complete pathname to a FipSeq file)
BINARY-FILEBEFORE: (complete pathname to a FipSeq file)
Add the contents of this file AFTER the FipHdr (if there
is no 'z:nohdr' and before the 'a:' text string before
and before the data.
There can be up to 100 file befores.
The file can be contain any text and FipSeq except the binary
version will copy the data byte for byte with no checking or
parsing for FipSeq.
z:filebefore:/fip/tables/customer/Header.fipseq
z:binary-filebefore:/fip/spool/savefiles/SN
z:filebefore:/fip/spool/savefiles/ThirdFile
FILEAFTER: (complete pathname to a FipSeq file)
BINARY-FILEAFTER: (complete pathname to a FipSeq file)
ditto for after.
The data from this file is add AFTER data and AFTER the
'A' text string after.
There can be up to 100 file afters.
INPUT-FILEBEFORE:
INPUT-FILEAFTER:
Special case of before and after files where the data
is the binary contents of the input data file with NO
changes. This is rarely used - as an extra attachment
with a preceeding mimetype, for example.
LOCALE: (valid locale)
Change the locale from the System Locale to this
The locale MUST be valid !
FLAG-IF-FIPHDR: (flag number) : (FipHdr field)
Turn on a flag if a FipHdr exists and has a least one character in.
z:flag-if-fiphdr:42:XC
There is an optional string after the FipHdr which is a further test that
the FipHDr equals this string.
z:flag-if-fiphdr:44:SU:AFP
or use either != or # to signify NOT
z:flag-if-fiphdr:44:SU#AFP
USA-DATE-FORMAT $A date format should be MM/DD (default is DD/MM)
usa-data-format:no will also force it DD/MM
CONVERT-TO-ENTITIES: (dec|hex|quark|indesign|rtf|json)
Convert all non-basic ascii chars to the ϧ notation.
eg a pound sign (xA6 or octal 236) becomes ž
Use this for XML feeds that do not want UTF8.
The option can be :
dec - decimal notation ì
hex - hex notation ¦
quark - quark express notation <#236>
indesign - adobe indesign notation <0x00A6>
rtf - rtf format u998? (assumes uc1 has been set in the header)
json - json format u00A6
Default is decimal
ENTITY-HASH:(fipseq chr)
Replacement chr for the HASH '#' in an entity created by CONVERT-TO-ENTITIES
for dec, hex and quark options
Use this if any FipHdr fields need converting too.
OUTPUT-ENTITY-HASH:(fipseq chr)
Replacement chr for the HASH '#' in an entity
Use this only if you are using the -O switch to convert back to HASH !
CONVERT-ENTITIES:
Convert entities in ϧ or &xAA; notation to a single chr
Normally use this with z:convert-to-utf8 as most of the numbers will be >
255 dec.
If the ENTITY-HASH is specified, that character will also be tested for
READ-ENTITY-FILE: (name of file in tables/xchg)
Many websites/xml feeds encode certain chrs as 'entities' - either named or
numbered:
ie ' for a ' € for the euro sign
numbers like £ for a PoundSign
This uses a W3C entity file to automatically resolve Html/Xml entities
before the rest of the xchg is run
If the entity is NOT in the file, it is left in the data and not converted.
so &russiandoll; would be left alone
These are exactly the same format/files as used by ipsgml and
are 'borrowed' from w3c http://www.w3.org/TR/xhtml1/DTD/ appendix 2
They look like
<!ENTITY copy "©"> <!-- copyright sign, U+00A9 ISOnum -->
Multiple files can be specified :
z:read-entity-file:xhtml-lat1.ent
z:read-entity-file:xhtml-special.ent
z:read-entity-file:xhtml-symbol.ent
CONVERT-TO-UCS2:
Convert the incoming data to the UCS2 variant of Unicode.
This process is done on ALL xchged data as an absolutely last stage.
UCS2 is the format that MicroSoft Word uses by default for example.
CONVERT-TO-UTF8:
Convert the incoming data to the UTF8 variant of Unicode.
This process is NOT done on any xchged data.
If you need the reverse process, you will have to find all the funny chrs
and map them to whatever eg
x +u2034+(euro)
CONVERT-TO-BASE64:
CONVERT-FROM-BASE64:
Convert the data to or from Base64
when converting TO base64,
- Adjust line lengths with the optional number (default 72)
- Only the data part is converted - which allows befores
and/or filebefores to add lines like:
Content-Transfer-Encoding: base64
- Note that if you are encoding PDF files, you nearly
always need to preserve NULs - using z:nul
See the example below.
CONVERT-TO-UUENCODE: (TO uuencoded - not implemented yes)
CONVERT-FROM-UUENCODE:
Convert the data to or from uuencoded
UNMATCHED-UNICODE: (FipSeq String)
Where the chrset of the incoming data is UTF8 or UTF16 or UCS2 (as specified
by 'c:utf8' or 'c:ucs2' for example), use this to map any hi-byte unicode chr
to a replacement.
z:unmatched-unicode:*
UNICODE-MAP (file of character maps in tables/unicode)
Convert the data from the source set to unicode using a Unicode Mapping
table
Use where the source encoding such as BIG5 or KOI-8 and iyu need to map to
Unicode
Note that you need also to specify how to represent the unicode chr - which
will often be > 255.
So select an output encoding such as
'z:convert-to-utf8'
'z:convert-to-ucs2'
Also the input chrset should be 'c:isoascii'
For standard maps - look at
http://www.unicode.org/Public/MAPPINGS/
or http://tcl.apache.org/sources/tcl/tools/encoding
In these lookup tables, '#' is a comment.
Each chr is on a separate line:
(raw hex chr) (spc) (unicode chr) spc [ opt comment]
eg from KOI-8.TXT
0x9F 0x00F7 # DIVISION SIGN
0xA0 0x2550 # BOX DRAWINGS DOUBLE HORIZONTAL
0xA1 0x2551 # BOX DRAWINGS DOUBLE VERTICAL
0xA2 0x2552 # BOX DRAWINGS DOWN SINGLE AND RIGHT DBL
0xA3 0x0451 # CYRILLIC SMALL LETTER IO
REVERSE-UNICODE-MAP (file of character maps in tables/unicode)
Convert the data from unicode to the source.
The syntax and explanation is the same as for UNICODE-MAP (above)
Note that normally the input chrset will be 'c:utf8' or 'c:ucs2'
If you have both a UNICODE-MAP and a REVERSE-UNICODE-MAP, the REVERSE is
alwys done first
UNICODE-MAP-IS-7BIT:yes
Some code maps - like BIG5 or SHIFTJIS - are 7 bit codes not 8 as the
mapping start at xA0 not xFF
ie anything from xA0 needs to be mapped
default is NO (or unspecified) for 8 bit
eg ; map from SHIFTJIS
z:unicode-map:SHIFTJIS.TXT
; note it is a 7bit jobbie
z:unicode-map-is-7bit:yes
; .. and remember to output in UTF8
z:convert-TO-utf8
UNICODE-MAP-BASE: (ascii | isoascii | none)
fill in any missing elements with the basic ASCII set
ascii - basic 7 bit from 0 to x7F
isoascii- basic 8 bit from 0 to xFF
none - ignore it !
default - none
MAP-DATA: (no | yes) default is yes
If you want to ONLY map one of more FipHdr fields, use this to NOT map data.
ie - convert ONLY DI and XK
c:isoascii
z:map-data:no
z:convert-to-utf8
z:chghdr:DI,XK
z:unicode-map:CP1277.txt
FLIPLINES
Flip a line of data from left2right to right2left - or viceversa
Use this for mixed data - Hebrew-English or Arabic-English
z:fliplines:ltr text default is left-to-right, so flip Hebrew
z:fliplines:rtl text is right-to-left, so flip Latin9
z:fliplines:rtl,xml text is right-to-left, so flip Latin9 and ignore any
xml
z:fliplines:xml,x40,x80 text is ltr, and flip any chrs between x40 and x80
A line end is CR, NL or CRNL
CONVERT-FIPHDR-TO-QP: (orig FipHdr field):(new):(language):(length):(WrapChr)
Convert a single FipHdr to Quoted-Printable for use
in a Mime header (as used by the 'Subject' line in smtp)
If no language is specified, iso-8859-1 is the default.
If no length is specified, the whole field is converted otherwise
this is the maximum allowed on the line (normally 76).
If length is specified and the language is 'utf-8', the chrs are
converted to and from beforehand so that complete chrs are included.
The Optional WrapChr is a FipSeq chr which signals to ipsmtp to wrap here
rather than trimming the Fiphdr to that size.
eg
z:convert-fiphdr-to-qp:DI:DJ:koi-8
DI:DJIA at 3pm is 9999.3
generates
DJ:=?koi-8?Q?=DJIA=20at=203pm=20is=209999.3?=
CONVERT-ENCODED-WORDS: (all|hdr|data)
This converts the FipHdr or the Data or Both from Mime Encoded Words format
Use this to convert Email subject lines like
Subject: =?windows-1255?B?5uQg8uXjIPDx6eXvIOH54ensIODw4/jl?=
An Encoded Word line (usually an email field) has the format '=? (chrset)
'?' (type) '?' (data) '?='
This only works within text, so if the data is in a FipHdr field, use this
and
Z:FIPHDR-IS-DATA-TOO in the xchg parameter file (and make sure you have no
other xchg lines which could
damage the FipHdr !).
ANCHOR:
Search for common web urls and wrap an <A> tag around them
Urls starting with 'http://', https://, ftp://', 'www.',
'ftp.' and 'mailto:' are converted.
eg www.fippo.org becomes
<a href="www.fippo.org">www.fippo.org</a>
This works for Text only and can have a negetive effect on wordwrap.
PRESERVE-FILENAME:
The input filename will be used as the output filename.
Normally the output filename has a few fipHdr fields added.
This is the same as the -P input switch, but will only be done
for files using this xchg.
COPY-FILE: (full path and filename in FipSeq)
Up to 10 copies of the original file can be made
eg Z:COPY-FILE:/fip/data/nefarious/ends/$d/SU_EN
REPLACE-TAG: (original xml tag) (spaces) (replacement tag)
used to replace a tag with its attributes
Also use 'z:replace-tag-attX' (where X is 1-9) to put the attributes into a
temporary, private FipHdr area, so you can use FipSeq to reouput them in the
new tag.
EG - Incoming data has format :
<ticker companyLongName="Reed Elsevier NV Cert." companyShortName="REED
ELSEVIER NV" sedol="4148810" zzcode="ELS" isin="NL0000349488"
countryCodeId="NL">Reed Elsevier</ticker>
But we need the style of :
Manpower <symbol tickercode="MAN" country="US" cusip="56418H100"
isin="US56418H1005">MAN</symbol> shares led the gainers, rising 2.5
percent to $44.52.
z:replace-tag:ticker T0 <symbol tickercode="T6" country="T5" cusip="T3"
isin="T4">T6</symbol>
z:replace-tag-att1:companyLongName
z:replace-tag-att2:companyShortName
; Sedol is T3
z:replace-tag-att3:sedol
z:replace-tag-att4:isin
z:replace-tag-att5:countryCodeId
z:replace-tag-att6:zzcode
LOG-LINE: (extra information for logging in FipSeq)
z:log-line:SN company=A1 indcode=A2 isin=A3
The default is SN CX for filename and xchg(s)
ipxchg needs the '-l' input switch for this to print
ALWAYS-LOG: Always log ignoring the -l switch
NEVER-LOG: Never log ignoring the -l switch
DONE-QUEUE: Name of a folder which the incoming file is copied to
default - incoming file is deleted if spooled
MAX-SINGLE-FIPHDR-SIZE: max length that a single fiphdr field can be.
default is 8196-sep2012 (was 2024)
The minimum is 1024 and maximum is 31000
INTERPRET-BOM: (yes/no)
Look for the Byte Order Mark and cater for this automatically
This is usually the first two or three bytes of a Windows or Adobe text
file.
(first two bytes as BigEndian 0xFE 0xFF, LittleEndian 0xFF 0xFE or 0xEF 0xBB
0xBF for UTF8)
Some common programs from Microsoft, such as Notepad and Visual C++ add BOMs
to UTF-8 files by default.
Google Docs adds a BOM when a Microsoft Word document is downloaded as a
.txt file.
default is yes (from version 48f46)
An Example of an xchg :
-----------------------
; use the MAC chrset ie /fip/tables/chrset/MAC
c:mac
; strip all text from notes mode to the next mode change
D:<MDN>:<MD
; strip formats
d:16:17
;.. replace CR CAN (Octal 15 then 30) by CR
x:r30:r
; .. otherwise make CAN a space (chg Octal 30 to 40)
x:30:s
x:<MDB>:<B>
x:<MDR>:<P>
x:<MDI>:<I>
x:<MDF>:<F>
; use a different delimiter for changes sake
x-300e-e
; strip duplicate spaces both before and after the xchgs
; default is BEFORE the xchgs
z:SPC:before
z:SPC:after
Multiple Passes
---------------
From version 47b you can run multiple xchgs in the same run by specifying in
the CX Fiphdr file (usually in the USERS file)
stocks DP:edsys1 DQ:junk-edp CX:Stocks1,Stocks2,Stocks3 SC:NO DC:YES
DF:stocks
This will run the same file through 3 xchgs successively - tables/xchg/STOCKS1,
STOCKS2 and STOCKS3 - before sending to the Editorial System.
Previously the OLD method was ...
Note that only one pass of the file can be made in one go. So to run multiple
passes, add dummy addresses in the tables/sys/USERS file eg :
stocks DP:localhost DQ:2go CX:Stocks1 SC:NO DC:YES DU:stocks1
stocks1 DP:localhost DQ:2go CX:Stocks2 SC:NO DC:YES DU:stocks2
stocks2 DP:edsys1 DQ:junk-edp CX:Stocks3 SC:NO DC:YES DF:stocks
How to use flags
----------------
- ON 'o' and OFF 'f' and FlipFlop 'p' lines do not affect the data themselves.
- The flag is turned on/off AFTER the string.
- add spaces around the flag for better legibility, These are the same :
x77:nnn:rn
x 77 :nnn:rn
- NOT all lines are affected :
x, X, I, o, f, r, k, K, d, D
- you can use flags within flags :
o 22 +n+32
Only if flag 22 is ON, a newline will turn 32 ON.
- Test for a flag being OFF with a '#' before the number :
f # 19 :abc:33
if flag 19 is OFF, string 'abc' will turn flag 33 OFF (if 33 was ON)
o # 19 |abc|33
if flag 19 is OFF, string 'abc' will turn flag 33 ON (ignoring if 33 was ON or
OFF)
- Flags 1 to 100 are acceptable.
- You want to set flags on and off before any xchgs which use them as the xchg
may zap the data you need to key off. ie Turn the first 'A' after a NL to a
'B':
o:n:32
x 32 :A:B
f 32 :A:32
will NEVER turn flag 32 OFF as the preceeding line will have modified it !
Better do :
o:n:32
f 32 :A:32
x 32 :A:B
From version 44j, you may specifiy multiple flags on/off by putting a
non-number between :
o +n+1,2,3,4
Current limitations are :
-------------------------
Max size of an XCHG parameter file is 64k chrs.
Max size of a search OR a replacement string is 255 chrs each.
Max number of flags is 100 (1-100).
Running IPXCHG from a Script
----------------------------
The program can be used in a script to run against a single file using the '-1'
input switch. In this case, if the output file exists, it is OVERWRITTEN. If
the output queue is the same as the input, the input FILE is overwritten. The
'-D/-F/-o' switches are also advised in this case to be sure where the output
file will end up. Eg
/fip/bin/ipxchg -1 formwork/SHARES.RUBY -F -o formsave -D crunchy
will process (and delete after) the file from spool/formwork using the xchg
file tables/xchg/CRUNCHY and leave the xchged file in formsave.
or ipxchg -1 Cruddo -D mangle
will process Cruddo in the current queue using xchg file MANGLE.
Note that the output queue is ALWAYS under /fip/spool.
Special Case of Running an Xchg against the FipHdr
--------------------------------------------------
Using 'z:hdronly' allows you to ONLY modify data in the FipHdr and leave the
data part completely untouched.
Only a subset of commands are useable - some are obviously ignore like Word
format. Others not taken into account are :
- 'h' for hdr field translate (use normal 'c' for chrset)
- 'a' and 'A' for adding before and after the data part
You may find it advisable to use flags to limit the damage an xchg can do to a
particular FipHdr field.
For example - we want to xchg LAG, SOC and ORO in the 'NC' FipHdr field and and
word ManU in the 'NZ' :
; no chrset chg here
c:isoascii
; turn ON HdrOnly
z:hdronly
; Flag 1 is ON during the NC field only - turned off at the end of the line
; note we use pipes as delimiters as we have a colon ':' as part of the data
o |nNC:| 1
f |n| 1
; Only search for these if flag 1 is on
x 1 |LAG|PAR:12000000:LAG
x 1 |SOC|PAR:21000000:SOC
x 1 |ORO|PAR:21000000:ORO
; Flag 2 is ON during the NZ field only - turned off at the end of the line
o |nNH:| 2
f |n| 2
; Only search for these if flag 2 is on
x 2 |ManU|Liverpool
Example of using ipxchg to encode mail attachments
--------------------------------------------------
Use this for sending PDFs for example therough 'ipsmtp'
; main parameter file
; default chrset
c:isoascii
; preserve NULs
z:nul
; convert to base64 with a line length of 75 chrs
z:convert-to-base64:75
; read and parse the mime-header template beforehand
z:filebefore:/fip/tables/xchg/MIMEHDR.BEFORE
; At the end - finish the last part
A+rn--180191FipSN180191--rn
... and MIMEHDR.BEFORE could look like:
Content-Type: multipart/mixed;r
boundary="180191FipSN180191"rnr
--180191FipSN180191r
Content-Type: text/plainrnr
The following is a PDF from the Fippo
--180191FipSN180191r
Content-Disposition: attachment; filename="SN"r
Content-Type: application/pdfr
Content-Transfer-Encoding: base64rnr
File tracing thru the Fip
-------------------------
There are a series of parameters common to several Fip Programs which will log
what is happening to all files, a service or a single file.
file-trace: (valid actions - yes/no, + hdr + in + out)
Actions are 'Yes' - log, 'No' - dont; if NOT NO, then in addition to logging,
we can save a copy of the FipHdr, Input file and/or Output
(delete is ignored in ipxchg))
file-trace-name: (UniqueId in FipSeq if not default - SU-SN-HR)
file-trace-log: (extra File Tracing logging in FipSeq)
The Hdr and files are left in /fip/log/file_trace (which should be purged in
the nightly maintenance zapfiplog)
These can be overridden by FipHdr fields in each file if a particular file -
or service needs tracing.
(notice the '_' if using in the FipHdr but '-' if parameters)
eg ZFILE_TRACE: hdr
ZFILE_TRACE_NAME: louislouis
ZFILE_TRACE_LOG: seqno.ZQ
Picking up data in the file and reusing as a FipHdr
---------------------------------------------------
Say you want to pick up the funny date/time stamp at the end of a wire file..
;
; create TI field from wire file
; needs ipxchg.44d plus (13nov00)
stuff the lot into the FipHdr 'TI'
r-TI-*
; a '*' is a wild string
W:*
; do not zapt the wild buffer after the 'x' line
z:keepwild
; strip from the ETX to the EST
x:03*EST:
; ** ENDS **
---------------------------------------------------
Input flags
-----------
Input options are (all optional) :
Either -i : queue in spool to scan default: xchg
Or -1 : name of a single file to xchg default: none
In this case the input file is NOT deleted and
the output is forced to the folder '-o' field if there.
or the filename '-N' if specified
-o : default destination queue if no CQ field is present
Normally a FIP hdr field, CQ, determines the output queue.
If that is missing, this field overwrites the default
If this does NOT starts with a '/', it is assumed to
be under /fip/spool. default: 2atex
-O : display the resultant, xchged file and do NOT save to disk.
Use this inside a CGI for example default: disk
-A : allow W4_ABSTRACT_DC default: ignore
this will run the xchg and copy the file to a temp area for ipw4
the original file is untouched, except the W4_ABTRACT_FILE points to the
xchged abstract
-d : name of the default xchg default: no default
If there is more than one,separate by a comma
eg -d DEFAULT1ST,DEFAULT2ND (48d3 onwards)
Use '-L' to NOT log when this is used.
-D : force the xchg file to this def: use SC2DC or CX
-e : Supplementary xchg filename default: none
This name is appended to the CX or SC2DC file name
This enables a 2nd group of xchges to be used.
-E : name of a single entity file that should be run on ALL data. def: none
see Z:READ-ENTITY-FILE: above
-f : file creep time default: 0
For files arriving from a network, this is the wait time
Generally this should be 5 secs for RCPed files etc.
-F : force all output to the output destination queue and ignore
the CQ field. default: use CQ
-G : WINNT/2k - grab the first file only default: no
-h : Extra FipHdrs to be added to the input fiphdr default: none
-I : ignore any include-files in the parameter file default: allow
ie ignore z:FILEBEFORE:abc.pl
-l : log items through default: no
-L : do NOT complain/log files requesting non-existent xchgs
- just use the default default: complain
-N : output filename (or path and filename) for single files (-1) only.
default: overwrite input file
-P : Preserve the filename for the output file default: if spooled, chged.
-r : move the original file to this queue default: delete it
-R : move the original file to this queue default: delete it
and Change the name to CCYYMMDD.HHMMSS.(name)
-S : display any message that is being logged. default: log only.
-t : scan time of directory default: 2 secs
-u : optional done que for input file default: input file is deleted
-x : do NOT delete the original default: delete it
This only applies to spooled files; running single mode
will NOT delete the input.
-X : do NOT allow multiple xchgs in the CX FipHdr field . default: allow
-W : 10 ths of a second - MUST wait if nothing processed default: 0
-Y : one or more flags to set ON at the start of the run default: none
-Y 33,44
-9 : do NOT run speedyFip with named pipes default: do if available
-v : print version number and exit default: no
Version Control
;48f72 ; 17apr07 added convert-to-entities:indesign
;a-b 14may07 added Z:WRAP-LONG-WORDS:(length)
;c 05nov07 added xml for TEXT2FIPHDR
;d WINNT bugette ;d1-2 poss bug if single chr X and already-zapped data)
;3 allow multixchg for default ;4 text2fiphdr html added ;5 bugette with
R:A3:abc
;67 allow quoted filenames for -1 and winnt bugette ;89 hdronly in multis
bugette
;e4 01feb09 added convert-to/from-uudecode/uuencode ;2 TOPSPC after processing
too
;f1 25mar09 added -A allowW4Abs and W4_ABSTRACT_DC: ;2 bugette with
yanl/unyank and mulitple CX
;3-4 added z:convert-hdr:utf8,unicodemap and copy-file ;5-6
IGNORE-XML-IN-WRAP for entities too !
;7 added ABSTRACT-DATA ;8 added W4_ABSTRACT_CX ;9 entities: rtf ;10 7dec09
bugette in wrap-long-words
;11 15dec09 bugette with anchor tags being xchged (they should NOT be)
;12 bugette with missing multiple xchg ;13-14 bugette - dont use BOTTOMSPC
with HDRONLY
;15 3jun10 added z:convert-from-unicode ; 16 added ALWAYS-LOG and NEVER-LOG
and EOLSPC
;17-18 5aug10 added MAP-DATA:no/yes ;18 allow blank Z:COPY-FILE using an
option .. Z:COPY-FILE:V1abc$o
;19 16sep10 CHG - now looks for SC2DC before start of filename
;20-21 15nov10 better filename sizing / Win2k isspace bug
;22-24 wrinkleette in fipSpeedy and CONVERT-ENTITIES added
;25-27 19may11 added TRIM-HDR-SPACES
;28 29nov11 make sure output folder exists for -1/-N
;29 bugette in 'X' if flag specified but flag is never turned on
;30-2 9mar12 tuned TRIM-HDR-SPACES
;33 1may12 added z:fliplines:RTL and XML
;34 14jun12 added z:done-queue:(folder)
;35-44 14jul12 redid Word binary for all versions including Meta
;45 4sep12 added MAX-SINGLE-FIPHDR-SIZE
;46 8nov12 added INTERPRET-BOM to turn it off !
;47 5dec12 woops bugette with convert-fiphdr-to-qp with multiple lines
;48 3jan13 read-unicode-map can be FipSeq ; 49-50 bugette with WORD
;51-53 bugette in QP
;54 28oct13 added convert-to-entities:JSON
;55-60 chasing bugette in Word-DocumentSummaryInfo
;61-65 27jan14 fudge/fix latin1->utf8 !
;66 29apr14 file-trace added and ignore after if added (wrongly) to EOLSPC !!
;67 20jun14 maxCopies 10->30
;68 30jun14 bugette in convert-to-entities:JSON
;69 30jul14 unicodes 100 -> 1000 ; 70 if z:hdronly do NOT convert NL
convert-to-entities:JSON
;71-72 23aug14 allow uFFFF as replace string ..
;047z bugette with multiple xchgs losing CQ
;a-b 19apr06 added z:replace-tag
;c 15may06 bugette if flag set on input or FLAG-IF-FIPHDR without other 'o'
and 'f'
;de 25may06 added z:convert-to-entities at last and -N newname for single
;fg 28may06 added z:ignore-xml-in-wrap
;h-k 22jun06 added z:convert-to-entities:quark and ENTITY-HASH
;l 25jul06 allow flags in i/I strips and fixed bugette in -Y startflags
;m-n 14aug06 wrinkle in convert264 one too many '=' at end
;o 21sep06 added WrapChr to CONVERT-FIPHDR-TO-QP
;p 23oct06 bugette with c:utf8/z:convert-to-utf8
;q-s 13nov06 added convert-encode-words:
;t 08dec06 added <@> to quark-entities and removed TAB
;u-v 08jan07 allow ^ and % in parameter file names (v bugette in wild)
;w-y 08feb07 added z:FLIPLINES
;z 17mar07 added -3 extraFH3 plus Winnt -O is now binary
;046z3 mod for unicode-map (single bytes bugette)
;b 26jan04 added INPUT-FILEBEFORE/AFTER for original, raw file
added CONVERT-FIPHDR-TO-QP
;c 30mar04 chrset can now be hex too x4E
;d-e 19apr04 bugettes with unicode in hdr and include_file
;f-i 28apr04 added z:anchor
;j 29jun04 added max length to Convert-to-QP
;k-l 16aug04 added Multiple xchgs..
;m-q 07sep04 added -9 speedy and x/m speedup and -I
;r-s 01apr05 add unicode-map-is-7bit for CP850 and KOI7
;t 06may05 bugette with wildchrs (emmettlanebug)
and the hex in CONVERT-FIPHDR-TO-QP should be UCASE
;u 30jun05 bugette in map-unicode - sometimes making a mapped chr NUL
;v 02aug05 bugette in i/I if string > 120 chrs
;w-z3 22sep05 added CONVERT-TO-UCS2 and read-entity-file - z3 thsxhg->STDBUF
;045z 16apr02 added UNMATCHED-UNICODE
;a/b 18apr02 added 'm' for multibyte chr as in m+u20ac+EUROs
;c 25jun02 added -G for WINNT/2k grab first file
;d 31jul02 z:NEWHDR - new fipHdr field may also have '$', '-' and/or '_'
;e 09aug02 z:FIPHDR-IS-DATA-TOO added
;f/g 15aug02 ':' is only matched/moved to 372 in FipHdr SH
;h 30oct02 bugette in flags where files has thousands of flags
;i 21dec02 woops - allow h:utf8 too where UTF8 chrs are in the fiphdr
;j 22dec02 allow z:text2fiphdr for lines at the END of text
;k 14jan03 bugette with h:utf8
;m 16jan03 CHANGE - d:<:> will now be ignored if the end is not found
;n 03mar03 bottomspc now zaps replaced copy too
;o 10mar03 added IGNORE-DATA
;p-q 27mar03 better filenames for Win2k plus bugette in NEWHDR with NEWDATE
;r-t 10apr03 added FLAG-IF-FIPHDR
;u-w 14apr03 added USA-DATE-FORMAT/minors
;x-y 25sep03 bottomspc now zaps beforehand too
;z 31oct03 timings
;044z 18aug00 allow multiple filebefores/afters and -h for fiphdr and -O for
stdout
;a/b 19aug00 added text2fiphdr
;c 10oct00 finally got the repeated chr bug
;d 03nov00 ..but that blew Yank/Unyank
;e 12nov00 allow SN length to be variable and allow wilds in 'r'
;f 20nov00 bug - stripped FipHdr for zero length data part
;g 06mar01 possible bug with flags and big files.
;h 27mar01 bugette - strips to/from (i/I) were parsed twice
;i 16apr01 bugette with flags again
;j/k 05jul01 allowed multiple flags on/off o +n+1,2,3,4
;m 03sep01 bugette with a/A and no data
;o/p/q 13sep01 added binary-filebefore/after and move_it added
;r 03oct01 bugette with FileBefore and no data in file.
;s 25oct01 bugette with text2fiphdr and stripping.
;t 14nov01 added FlipFlipFlags.
;u 19nov01 added z:bottomspc.
;v 26nov01 added c:utf8, c:utf16 and z:convert-to-utf8
;w 29nov01 added flags to A-add after
;x 29jan02 added convert-to/from-base64
;y 25feb02 bugette in d and D where 2nd string may be matched early
;z 15apr02 c:utf8 not working brilliantly.
(copyright) 2014 and previous years FingerPost Ltd.