IPXCHG translates the Character Set and any Style changes that are needed to
process any incoming file. It uses a parameter file, in tables/xchg, to define
what the processing should be and how it should be done.

The incoming file is normally passed on by 'ipwheel' as part of the pipline to
get the data to the destination. IPXCHG can also be run as a once-hit against a
particular file.

In order to determine what parameter file to use, 'ipxchg' uses FipHdr fields,
if they exist.

This can be overridden by the '-D' input switch which will force the xchg to be
that name.

Otherwise the 'CX' field is first looked for.

If there is no 'CX', the 'SC' field is compared to the 'DC' field. If they are
different, the xchg file used will be SC2DC.
ie if 'SC:AP' and 'DC:HTML', then the xchg parameter file will be 'AP2HTML'.

If there is NO SC, DC or CX and NO -D input switch, a string at the beginning
of the filename (up to the first HASH/POUND SIGN) can be used.

The actual parameter file is in tables/xchg and the filename will be forced

Syntax for the xchg parameter file held in tables/xchg :
    one parameter per line
    ; Comments lines may be inserted at any time and are preceeded by a ';'.
    : Comment lines can also be preceeded by a colon in this case

    (type) [ (flag) ] (delimiter) (1st parameter) [ (delimiter) (2nd parameter)

    where items in [] are optional
        - type is a CASE SENSITIVE type of parameter - c, D, x etc
        There should be NO leading spaces/tabs. To aid legibility,
        other letters and spaces may be put AFTER the type but take
        care NOT to use letters or punctuation, eg these are equivalent:
            chrset :ascii
        and c:ascii
        - A flag is any number between 1 and 100. It allows conditional
        xchgs to be done. Ie the flag must have be set ON using the 'o'
        command beforehand or the line is ignored.
        - Delimiter can be any punctuation EXCEPT backslash or Hash.
        it must also be consistent :
        - Parameter string is any FIP sequence of fixed text,
        Octal numbers (\000 format) and Unix escape chrs (\n etc).
        In some cases - eg 'r' for Add FipHdr - the data can be any
        FipSeq string including System Variables and existing FipHdr
        The first parameter string is called the TO or SEARCH field.
        The second parameter (if there) is FROM or REPLACE field.

There are no mandatory parameters - the defaults are :
    c: ASCII        Chrset is ASCII
    b: 8 : 8        Input stream is 8 bit chr and output is 8 bit
    Default is : No escapes, xchgs or other mutilations.

List of Types

    Syntax              Example
    ------              -------
x, X - An xchg; from is the source string, to is the replacement string.
The From or search string can be 1-255 chrs long; the replacement string can be
0-255 chrs long.
    x:fromString:toString       x:+++:bong
The difference between 'x' and 'X' is :
    'x' the search string is case SENSITIVE
    'X" the search string is case INsensitive
    ie  X:head:hhh
    will replace 'HeAd' or 'HEAD' or 'head' etc with 'hhh'

c -This refers to parameter file tables/chrset/PA
    (see below for format of this external chrset file)
    c:(main chrset)         c:PA

    There are a number of hard-coded chrsets which do NOT need to be in chrset:
        c:ascii         7 bit ascii
        c:isoascii      8 bit ascii
        c:ascii2baudot  automatic conversion from ascii to baudot
        c:c2let,c2num   the two baudot sets
        c:utf8          8  bit unicode - see doc on utf
        c:utf16         16 bit unicode - see doc on utf
        c:ucs2          16 bit unicode (Microsoft's notepad/Sql Server version)

    Multiple chrsets can be defined as comma separated names :
    There can be up to 10 chrsets.

    or if only one alternate set needs to be defined :
    C - This is the alternate chrset - for 5 or 6 bit sets for example
        C:alt chrset            C:c2num
    So these two are equivalent :
        c:c2let,c2num           c:c2let

h - This is the header chrset. Parameter z:chghdr must be specified for the
fields to translate (see below). If there is no 'h:' the main chrset is used.
    h:hdr chrset            h:swhdr
    Please do NOT use characters NL, Tilde (octal 176) and Colon in Header fields
    They will be moved to octal 371, 372 and 373 respectively if you do !

e, E - Escape chrs to the alternate and back to the main chrset for
For example where multi-layer chrsets such as BAUDOT and TTS are used,
    e:escape chr from main      e:\033
    E:escape chr from alt       E:\037
    Note the default is to pass these chrs through - use z:ZAPESC to strip.
    OR these can be specified as
        e:(Esc chr):(Number of chrset)
    where the number is the chrset in the 'c:' line above :
        c:PA,PA2,PA3    PA is 1, PA2 is 2 and PA3 is 3
    there can be up to 20 escape chrs. The actual chr MUST be unique so octal 005
        escape to both PA and PA2 at the same time. But more than one escape chr
        can point to the same chrset.
    Escapes MUST always be specified lower down the parameter file than Chrsets.

s, S - Escape chrs for a single Chr from the alternate and then revert back to
the main chrset for multi-chrset
For example where multi-layer chrsets such as ISO-7901 in its many guises.
    s:single chr escape from main   s:&
    S:single chr escape from alt    S:&
    Where we want to escape to a secondary chrset for a single chr.
    Note the default is to pass these chrs through - use z:ZAPESC to strip.

d, D - Delete all between from and to strings.
Lowercase d strips BOTH FROM and TO strings, Upper D strips only the FROM.
    d:ignore from:to        d:<:>
    D:ignore from:to        D:<MDNO:<MD

i, I - Ignore all from the beginning until this string
    Strip all from this string to the end of file
    From version 47l conditional flags are NOT ignored.
    i:Ignore Upto string        i:\002
    I:Strip all from this string    I:\003
    Note that this string is the RAW data -
        ie NO chrset translation has been done beforehand
    In either case, if the string is NOT found, then data is NOT stripped.

m - MultiByte chrs for UTF/Unicode
    One common one is the Euro :

a, A - Add text at beginning or end
    This text is ALWAYS added and any conditional flags ignored.
    a:String for begining of file   a:{m1Top Table{m0\n
    A:String to end file        A:\r\003NAME \$d-\$m-\$y\004
    From version 44m onwards, up to 100 adds may be specified.
            a:Top line\n
            a:Line 2\n
            a:Line 3\n
    From 44w, both After and Before Adds may be dependent on a flag
            A   +\n(always)
            A #22+\n(not 22)
            A #33+\n(not 33)
            A  22+\n(yes 22)
            A  33+\n(yes 33)
            A   +\nbibi\n

f, o - Set flag OFF or ON   - see below for more information on flags
    To set flag on or off for conditional Xchgs.
    f:Search string:Flag number f:\r\nREU:23
    o:Search string:Flag number o:NNNN:23
    The flag is turned ON/OFF immediately AFTER the string
    The string itself is left untouched - use an 'x:' line to strip that.
    Uppercase 'F' and 'O' may be used for case-INsensitive search strings.

p, P - flip-flop a flag ON and OFF
    To set flag alternatively on then off for conditional Xchgs.
    p:Search string:Flag number p:*T:44
    The flag is turned ON/OFF immediately AFTER the string
    The string itself is left untouched - use an 'x:' line to strip that.
    Uppercase 'P' may be used for case-INsensitive search strings.

k, K - Kill or delete characters either before or after the search string
    k: search string:No to delete before    k:\r\n:5
    K: search string:No to delete after K:RHUBARB:93
    The string itself is left untouched - use an 'x:' line to strip that.

r - Add a new Fip Header Field after processing has finished.
    The SEARCH file holds the Hdr field name and the REPLACE field
    holds data. Note that any FipSeq data can be used but all
    FF, CR, DEL and NL are replaced by '_'.
    r-TI-Time is \$h:\$n:\$b
    Several 'r's may be specified. The new fields are added at the end
    of the Fip Hdr so they may be used to override existing Hdr fields
    but will in turn be overridden by any Hdr fields in the filename.
    Note the '-' used as separators as ':' is used as part of the string

y, Y - Yank or save text between from and to strings
    Lowercase y saves from the end of the FROM field to the BEGINNING
    of the TO field, while Uppercase Y saves to the END of the TO.
    y:[ptsize is:\n

u, U - UNYank or unsave text previously saved using 'y' or 'Y'
    Lowercase u adds the text at the BEGINNING of the FROM field,
    while Uppercase U adds it AFTER the FROM. The TO field is ignored.
    u:<add before this>

w, W - Allow wild character and strings
    The first character of the Lowercase w  is the wild CHARACTER ie
    a single character, while the first character of the Uppercase W
    is the wild string.
    w:? wild chr is '?'
    W:* wild string is '*'
    Note that only one wild string is allowed in the TO or SEARCH string
    of any test and it can NOT be the first chr of the SEARCH string.
    There is no restriction on the number of wild Chrs in either search
    replace strings. When used in REPLACE the saved chrs are output
    in the same order as found.
    The maximum number of chrs and string is 1024.
    ; wild CHR is '?'
    ; wild STRING is '*'
    An incoming string of 'ABCDEFGHIJ' will be stripped to 'abEfgFG'
    Note that you should make sure that the two wilds are not in
    any other SEARCH or REPLACE string anywhere else in the
    parameter file or you will get funny results :
        W .*
        x .[p*].<ptsize=*>
        x .@*&%$.abcde
    will probably NOT give the required values for the second, fixed string.
    Note that by default the wild card/string is positional - use it and lose it.
    To use the Wild string as a full string each time - as in the example :
        x .[cp*].[cp*][cl*]
        use the option  'z:ALLWILD'

    To use the contents of the wild buffer in a later xchg line,
        use the option  'z:KEEPWILD'

z - Setup Option
    z:option            z:FCC:
    Options include:
        FCC idiot upper and lower-case. Note you may force an
            Uppercase letter by using a '~' as a precedent chr.
        NUL do NOT strip NULs ie binary zero chrs
            NULS are normally deleted from the output automatically
        TOPSPC      strip all spaces, tabs, carriage return and line feeds at the top of
the file
        BOTTOMSPC   strip all spaces, tabs, carriage return and line feeds at the
bottom of the file
        EOLSPC      strip all spaces and tabs BEFORE the end of a line (all other spaces
on the line are preserved)
                An end-of-line is either CR and/or NL chr - but optionally a single
different character may be specified
                eg  z:eolspc:\222
        SPC or SPC:before   strip DUPLICATE spaces BEFORE xchgs
        SPC:after       strip DUPLICATE spaces AFTER xchgs
        LINE or LINE:before strip DUPLICATE newlines/line feeds BEFORE xchgs
        LINE:after      strip DUPLICATE newlines/line feeds AFTER xchgs
        IGNORE-DATA     Ignore the data part of the file.
                    Any fipHdr and before/after are processed but all data from the file is
        ABSTRACT-DATA:(number)  Only preserve the first 'n' characters
                    Any fipHdr and before/after are processed but all data AFTER the first 'n'
are stripped.
        FIPHDR-IS-DATA-TOO  Allow xchgs on the whole of the file
                    including the FipHdr part!
        NOFIPHDR or NOHDR   Do NOT add the FIP header to the data.
        CHGHDR  Also replace header fields with this chrset. This
            defaults to the Source Header only (SH). Other fields
            can be specified by listing after this flag. eg :
            The header chrset can be specified by the 'h:' parameter
            or will default to the same as the primary chrset as
            in the 'c:' parameter. Note that if the fields also
            exist in the filename, they are also changed.
            Note you cannot replace individual subfields of the SH -
            like XP, XF etc. Use NEWHDR to make a copy of that
            subfield first and then chghdr it.
                z:newhdr:AP \XP
        CONVERT-FIPHDR: (options - comma separated if there are more than one)
            eg: z:convert-fiphdr:utf8,map
            convert the FipHdr field(s)
            UTF8    Convert the COMPLETE FipHdr to UTF8
            MAP Convert the field(s) in z:chghdr:xx using the map-unicode file specified
                Eg if the incoming data is Russian (CP1251) variety, put the unicode map in
tables/unicode and :
                Note you will always want to specified UTF8 too for this option
                Convert to entities.
        TRIM-HDR-SPACES: yes/no
            remove all multiple spaces and any spaces at the beginning and end of all
FipHdr fields

        HDRONLY Only process the FipHdr fields - leave the data alone - see below.

        WORDMAC If the file is in the Mac version of the Microsoft Word format, the
header and trailer are stripped, leaving just the text section. This works for
Versions 5 and 6 only.
            An optional parameter may be added to STRIP the data if NOT Word (eg XLS)
        WORDPC  If the file is in the Mac version of the Microsoft Word format, the
header and trailer are stripped, leaving just the text section. This works for
Versions 6 thru 9 (ie all WordBinary formats from 48f35).
            Optional parameters may be added to STRIP the data if NOT Word (eg XLS) and
Save Meta as MA-MS: fiphdr fields
            Unless z:nohdr' is specified, a FipHDr of M9 will be the numeric part of the
code page for the data eg MZ:65001 for UTF8
            For the Meta-Fiphdrs :
                MA - just the numeric part of the code page of metadata eg MA:1252 for
                MB - Title
                MC - Subject
                MD - Author
                ME - Keywords
                MF - Comments
                MG - Template
                MH - Last Saved By
                MI - Revision Number
                MJ - Total Editing Time
                MK - Last Printed
                ML - Create Time/Date
                MM - Last saved Time/Date
                MN - Number of Pages
                MO - Number of Words
                MP - Number of Characters
                MQ - Thumbnail
                MR - Name of Creating Application
                MS - Security

                - From DocumentSummaryInfo ..
                NB - Category
                NN - Manager
                NO - Company

        ZAPESC  For escaped chrsets - such as 5 level baudot or 6 level
            TTS, delete/ignore the escape chrs.
        ALLWILD Use the whole of the wild string in a REPLACE, not just
            the remainder.
            Normally the wild card buffer is only available for
            the one Xchg line and then zapped after use.
            This allow it to be used later on.
        PARTIAL new FIP header field created from existing field
            z:partial:qc    XK,3,8,u
            ie To make QC, take K header field from the Source Header (SH)
            posn 3 for 8 chrs, UPPERCASE.
            This field is only used internally and NOT stuffed in
            the output file unless saved with NEWHDR.
            (refer to SysAdmin FipSeq doc for more information)
        FIXHDR  additional FIP header information
            z:fixhdr:QZ 1234543
            ie If QZ is specified, replace with 1234543
            This field is only used internally and NOT stuffed in
            the output file unless saved with NEWHDR.
            (refer to SysAdmin FipSeq doc for more information)
        OPTION  test a FipHdr field before outputting
            z:option:QR ZZ,3
            This field is only used internally and NOT stuffed in
            the output file unless saved with NEWHDR.
            (refer to SysAdmin FipSeq doc for more information)
        REPEAT  create a new FipHdr field from an existing field
            z:repeat:QR ZZ,-,3
            This field is only used internally and NOT stuffed in
            the output file unless saved with NEWHDR.
            (refer to SysAdmin FipSeq doc for more information)
        COMBIE  create a new FipHdr field from one of two existing fields
            or use a default
            z:combie:QR ZZ|ZR,(defaultpls)
            This field is only used internally and NOT stuffed in
            the output file unless saved with NEWHDR.
            (refer to SysAdmin FipSeq doc for more information)
        STYLE:  create a new FipHdr field from an existing field and
            pad it with spaces, zeros or justify it using
            a 'printf' type style
            z:style:QZ  XZ,%.04f
            This field is only used internally and NOT stuffed in
            the output file unless saved with NEWHDR.
            (refer to SysAdmin FipSeq doc for more information)
        REPLACE: create a new FipHdr field from an existing field and
            replace one or more strings/chrs.
            z:replace:qK    XK  "BC-"=""
            (refer to SysAdmin FipSeq doc for more information)
        NEWDATE: create a new FipHdr field containing the date offset from
            the current datee and time
            z:newdate:QD    days+3  "\ZH - \ZD"
            This field is only used internally and NOT stuffed in
            the output file unless saved with NEWHDR.
            (refer to SysAdmin FipSeq doc for more information)
            (refer to SysAdmin FipSeq doc for more information)
        NEWHDR: create and SAVE a new FipHdr field.
            Any FipSeq can be  added.
            z:newhdr:JF FormatIs.\XF
            z:newhdr:JN \XN+\XS=\SN
            This field is added to the FipHdr at the start of
            processing and can therefore be 'hdrXchged' etc.
            (refer to SysAdmin FipSeq doc for more information)
            Please note that any FipSeq (z:REPLACE: z:COMBIE: etc) MUST be
            specified first BEFORE any NEWHDR lines in the parameter file.
        TEXT2FIPHDR: Extract a line at the top of the file and make it a
            FipHdr field (and normally/optionally remove from text).
            A line is ended by any combination of CR and NL
            This is done on the RAW, incoming data,; not the output.
            At the top of text :
                This is done on the RAW, incoming data,; not the output.
            At the bottom of text :
                Specify with a negative line number : -1 for the last, -2 2nd last etc
                Lines are counted AFTER z:bottomspc and and Strip from (I:)
                If there is a trailing NL, CR or CRNL, they are ignored
            syntax  z:text2fiphdr: (line number):(fiphdr):(optional flags)
            eg  z:text2fiphdr:1:DI:no
            The optional flags can be :
                yes - (stripfromdata) - also delete that data
            or  no  - do NOT strip
            or  xml - ignore any xml tags and attributes and only get data
            or  html    - ignore up to <body> or </body>
            The default strip is YES. There can be 100 z:text2fiphdr lines
            (note you cannot strip XML)
        REDUCEBEFORE: (chr)
        REDUCEAFTER: (chr)
            Reduce all multiple occurances of a single chr to a one
            The character can be in FipSeq - \031
            reducebefore acts After the 'chrset' and Before any xchgs
            reduceafter acts after all processing.
            There can be up to 100 reducesbefores and afters.
        WRAPTEXT: (line length) : (End-of-Line string)
            Wrap text after all processing to a specified number
            of chrs per line. Add the End-of-Line string for long
                The default line length is 64.
                The default end of line is NL or \n
        IGNORE-XML-IN-WRAP: yes/no
            Ignore any tags within <...> in calculating wordwrap
            default: no for leave as part of the calculation
            Ignore any entities within &...; in calculating wordwrap
            default: no for leave as part of the calculation
            add a space after this number of characters to wrap long words
            eg  z:wrap-long-words:77
            default: no
        NUMBER: (octal/decimal/hex)
            Change the number system for FipSeq numbers to this
            The default is octal.
            Note this takes effect for ALL parameters in the file.
        FILEBEFORE: (complete pathname to a FipSeq file)
        BINARY-FILEBEFORE: (complete pathname to a FipSeq file)
            Add the contents of this file AFTER the FipHdr (if there
            is no 'z:nohdr' and before the 'a:' text string before
            and before the data.
            There can be up to 100 file befores.
            The file can be contain any text and FipSeq except the binary
            version will copy the data byte for byte with no checking or
            parsing for FipSeq.
        FILEAFTER: (complete pathname to a FipSeq file)
        BINARY-FILEAFTER: (complete pathname to a FipSeq file)
            ditto for after.
            The data from this file is add AFTER data and AFTER the
            'A' text string after.
            There can be up to 100 file afters.
            Special case of before and after files where the data
            is the binary contents of the input data file with NO
            changes. This is rarely used - as an extra attachment
            with a preceeding mimetype, for example.
        USE-SX: replace/increment/no
        USE-EXTERNAL-FILE: replace/increment/no
            if (the file is just a FipHdr file and uses 'SX:' to point at an external
data file,
                'I'ncrement-add inc number on ends and leave existing alone (default)
                'R'eplace existing
        LOCALE: (valid locale)
            Change the locale from the System Locale to this
            The locale MUST be valid !
        FLAG-IF-FIPHDR: (flag number) : (FipHdr field)
            Turn on a flag if a FipHdr exists and has a least one character in.
            There is an optional string after the FipHdr which is a further test that
the FipHDr equals this string.
            or use either != or # to signify NOT
        USA-DATE-FORMAT     \$A date format should be MM/DD  (default is DD/MM)
            usa-data-format:no will also force it DD/MM

        CONVERT-TO-ENTITIES: (dec|hex|quark|indesign|rtf|json)
            Convert all non-basic ascii chars to the &#999; notation.
            eg a pound sign (xA6 or octal 236) becomes  &#158;
            Use this for XML feeds that do not want UTF8.
            The option can be :
                dec - decimal notation      &#236;
                hex - hex notation          &#xA6;
                quark   - quark express notation    <\#236>
                indesign - adobe indesign notation  <0x00A6>
                rtf - rtf format            \u998? (assumes \uc1 has been set in the header)
                json    - json format           \u00A6
            Default is decimal
        MAX-ENTITY: (number) : (replacement chr)        default: no max, all entities
            Largest number allowed for entities. Use this to strip emojis and weirdo
entities. if specified, the anything above the number is replaced by the RepChr
(which defaults to a '*')
            eg  ; make the Euro symbol the biggest ... x20AC = 8364
                ; for hex number preceed with 'x'

        ENTITY-HASH:(fipseq chr)
            Replacement chr for the HASH '#' in an entity created by CONVERT-TO-ENTITIES
for dec, hex and quark options
            Use this if any FipHdr fields need converting too.
        OUTPUT-ENTITY-HASH:(fipseq chr)
            Replacement chr for the HASH '#' in an entity
            Use this only if you are using the -O switch to convert back to HASH !
            Convert entities in &#999; or &xAA; notation to a single chr
            Normally use this with z:convert-to-utf8 as most of the numbers will be >
255 dec.
            If the ENTITY-HASH is specified, that character will also be tested for
        READ-ENTITY-FILE: (name of file in tables/xchg)
            Many websites/xml feeds encode certain chrs as 'entities' - either named or
                ie &apos; for a '  &euro; for the euro sign
                numbers like &#163; for a PoundSign
            This uses a W3C entity file to automatically resolve Html/Xml entities
before the rest of the xchg is run
            If the entity is NOT in the file, it is left in the data and not converted.
                so &russiandoll; would be left alone
            These are exactly the same format/files as used by ipsgml and
            are 'borrowed' from w3c http://www.w3.org/TR/xhtml1/DTD/ appendix 2
            They look like
                <!ENTITY copy   "&#169;"> <!-- copyright sign, U+00A9 ISOnum -->
            Multiple files can be specified :

            Convert the incoming data to the UCS2 variant of Unicode.
            This process is done on ALL xchged data as an absolutely last stage.
            UCS2 is the format that MicroSoft Word uses by default for example.

            Convert the incoming data to the UTF8 variant of Unicode.
            This process is NOT done on any xchged data.
            If you need the reverse process, you will have to find all the funny chrs
and map them to whatever eg
                x +\u2034+(euro)

            Convert the data to or from Base64
            when converting TO base64,
                - Adjust line lengths with the optional number (default 72)
                - Only the data part is converted - which allows befores
                    and/or filebefores to add lines like:
                    Content-Transfer-Encoding: base64
                - Note that if you are encoding PDF files, you nearly
                    always need to preserve NULs - using z:nul
            See the example below.

        CONVERT-TO-UUENCODE:    (TO uuencoded - not implemented yes)
            Convert the data to or from uuencoded

        UNMATCHED-UNICODE: (FipSeq String)
            Where the chrset of the incoming data is UTF8 or UTF16 or UCS2 (as specified
by 'c:utf8' or 'c:ucs2' for example), use this to map any hi-byte unicode chr
to a replacement.

        UNICODE-MAP (file of character maps in tables/unicode)
            Convert the data from the source set to unicode using a Unicode Mapping
            Use where the source encoding such as BIG5 or KOI-8 and iyu need to map to
            Note that you need also to specify how to represent the unicode chr - which
will often be > 255.
                So select an output encoding such as
                Also the input chrset should be 'c:isoascii'
             For standard maps - look at
            or  http://tcl.apache.org/sources/tcl/tools/encoding
            In these lookup tables, '#' is a comment.
            Each chr is on a separate line:
            (raw hex chr) (spc) (unicode chr) spc [ opt comment]
        eg from KOI-8.TXT
        0x9F    0x00F7  #   DIVISION SIGN
        0xA0    0x2550  #   BOX DRAWINGS DOUBLE HORIZONTAL
        0xA1    0x2551  #   BOX DRAWINGS DOUBLE VERTICAL
        0xA2    0x2552  #   BOX DRAWINGS DOWN SINGLE AND RIGHT DBL
        0xA3    0x0451  #   CYRILLIC SMALL LETTER IO

        REVERSE-UNICODE-MAP (file of character maps in tables/unicode)
            Convert the data from unicode to the source.
            The syntax and explanation is the same as for UNICODE-MAP (above)
            Note that normally the input chrset will be 'c:utf8' or 'c:ucs2'
            If you have both a UNICODE-MAP and a REVERSE-UNICODE-MAP, the REVERSE is
always done first

            Some code maps - like BIG5 or SHIFTJIS - are 7 bit codes not 8 as the
mapping start at xA0 not xFF
            ie anything from xA0 needs to be mapped
            default is NO (or unspecified) for 8 bit
            eg  ; map from SHIFTJIS
                ; note it is a 7bit jobbie
                ; .. and remember to output in UTF8

        UNICODE-MAP-BASE: (ascii | isoascii | none)
            fill in any missing elements with the basic ASCII set
                ascii   - basic 7 bit from 0 to x7F
                isoascii- basic 8 bit from 0 to xFF
                none    - ignore it !
            default - none

        MAP-DATA: (no | yes)        default is yes
            If you want to ONLY map one of more FipHdr fields, use this to NOT map data.
            ie - convert ONLY DI and XK
            Flip a line of data from left2right to right2left - or viceversa
            Use this for mixed data - Hebrew-English or Arabic-English
                z:fliplines:ltr     text default is left-to-right, so flip Hebrew
                z:fliplines:rtl     text is right-to-left, so flip Latin9
                z:fliplines:rtl,xml text is right-to-left, so flip Latin9 and ignore any
                z:fliplines:xml,x40,x80 text is ltr, and flip any chrs between x40 and x80
            A line end is CR, NL or CRNL

        CONVERT-FIPHDR-TO-QP: (orig FipHdr field):(new):(language):(length):(WrapChr)
            Convert a single FipHdr to Quoted-Printable for use
            in a Mime header (as used by the 'Subject' line in smtp)
            If no language is specified, iso-8859-1 is the default.
            If no length is specified, the whole field is converted otherwise
            this is the maximum allowed on the line (normally 76).
            If length is specified and the language is 'utf-8',  the chrs are
            converted to and from beforehand so that complete chrs are included.
            The Optional WrapChr is a FipSeq chr which signals to ipsmtp to wrap here
            rather than trimming the Fiphdr to that size.
                DI:DJIA at 3pm is 9999.3
        CONVERT-ENCODED-WORDS: (all|hdr|data)
            This converts the FipHdr or the Data or Both from Mime Encoded Words format
            Use this to convert Email subject lines like
                  Subject: =?windows-1255?B?5uQg8uXjIPDx6eXvIOH54ensIODw4/jl?=
            An Encoded Word line (usually an email field) has the format  '=? (chrset)
'?' (type) '?' (data) '?='
            This only works within text, so if the data is in a FipHdr field, use this
             Z:FIPHDR-IS-DATA-TOO in the xchg parameter file (and make sure you have no
other xchg lines which could
            damage the FipHdr !).

            Search for common web urls and wrap an <A> tag around them
            Urls starting with 'http://', https://, ftp://', 'www.',
            'ftp.' and 'mailto:' are converted.
            eg www.fippo.org becomes
                <a href="www.fippo.org">www.fippo.org</a>
            This works for Text only and can have a negetive effect on wordwrap.

            Output filename in FipSeq
            Normally the output filename has a few fipHdr fields added.
            This is the same as the -N input switch except output-filename is used for
the xchg it is specified in while -N acts on the LAST xchg (not just -1 single)
            The input filename will be used as the output filename.
            Normally the output filename has a few fipHdr fields added.
            This is the same as the -P input switch, except only files using this xchg
will be preserved

        COPY-FILE: (full path and filename in FipSeq)
            Up to 10 copies of the original file can be made
            eg  Z:COPY-FILE:/fip/data/nefarious/ends/\$d/\SU_\EN

        SCRIPT: (script and parameters in FipSeq)
            Run a script AFTER processing.
            Care must be taken NOT to run a script on a file that normally is written to
a spooled queue as that file probably will not exist when ipxchg runs the
            So you do not run out disk space, the script could delete the input file
after use OR delete them all in the nightly maintenance - 'zapfiplog'
            Tmp FipHdr fields are created for script :
                E1 - original input file (note that if this has spaces in, pls quote)
                E2 - input file (will be different for multixchgs)
                E3 - output file
                E4 - name of the xchg
            eg  Z:SCRIPT:/fip/local/fip_sortoutname.pl input='\E2' output='\E3'

        SEQNO-NAME: (fipSeq)
            Use a dedicated seqno for this traffic      default: use common one
            eg  z:seqno-name:GNS\DU

        REPLACE-TAG: (original xml tag) (spaces) (replacement tag)
            used to replace a tag with its attributes
            Also use 'z:replace-tag-attX' (where X is 1-9) to put the attributes into a
temporary, private FipHdr area, so you can use FipSeq to reouput them in the
new tag.

            EG - Incoming data has format :
                <ticker companyLongName="Reed Elsevier NV Cert." companyShortName="REED
ELSEVIER NV" sedol="4148810" zzcode="ELS" isin="NL0000349488"
countryCodeId="NL">Reed Elsevier</ticker>
            But we need the style of :
                Manpower <symbol tickercode="MAN" country="US" cusip="56418H100"
isin="US56418H1005">MAN</symbol> shares led the gainers, rising 2.5
percent to $44.52.

            z:replace-tag:ticker    \T0 <symbol tickercode="\T6" country="\T5" cusip="\T3"
            ; Sedol is T3

        LOG-LINE: (extra information for logging in FipSeq)
            z:log-line:\SN company=\A1 indcode=\A2 isin=\A3
            The default is \SN \CX for filename and xchg(s)
            ipxchg needs the '-l' input switch for this to print
        ALWAYS-LOG: Always log ignoring the -l switch
        NEVER-LOG:  Never log ignoring the -l switch

        DONE-QUEUE: Name of a folder which the incoming file is copied to
            default - incoming file is deleted if spooled

        MAX-SINGLE-FIPHDR-SIZE: max length that a single fiphdr field can be.
            default is 8196-sep2012 (was 2024)
            The minimum is 1024 and maximum is 31000

        INTERPRET-BOM: (yes/no)
            Look for the Byte Order Mark and cater for this automatically
            This is usually the first two or three bytes of a Windows or Adobe text
            (first two bytes as BigEndian 0xFE 0xFF, LittleEndian 0xFF 0xFE or 0xEF 0xBB
0xBF for UTF8)
            Some common programs from Microsoft, such as Notepad and Visual C++ add BOMs
to UTF-8 files by default.
            Google Docs adds a BOM when a Microsoft Word document is downloaded as a
.txt file.
            default is yes (from version 48f46)

        MASK-EMAIL: (yes/no)
        MASK-NUMBERS: (yes/no/ or a number)
            Blank out any of these with 'x' if found
            For numbers, only mask out if the run of numbers (ignoring spaces +-.,) is
equal or more than this (to mask telephone numbers)
        MASK-SOCIAL-MEDIA: (yes/no)
            This blanks out the word (hopefully the address) after twitter:, snapchat:,
facebook:, skype:, instagram:

An Example of an xchg :

    ; use the MAC chrset ie /fip/tables/chrset/MAC
    ; strip all text from notes mode to the next mode change
    ; strip formats
    ;.. replace CR CAN (Octal 15 then 30) by CR
    ; .. otherwise make CAN a space (chg Octal 30 to 40)
    ; use a different delimiter for changes sake
    ; strip duplicate spaces both before and after the xchgs
    ; default is BEFORE the xchgs

Multiple Passes

From version 47b you can run multiple xchgs in the same run by specifying in
the CX Fiphdr file (usually in the USERS file)

stocks  DP:edsys1   DQ:junk-edp CX:Stocks1,Stocks2,Stocks3  SC:NO DC:YES  

This will run the same file through 3 xchgs successively - tables/xchg/STOCKS1,
STOCKS2 and STOCKS3 - before sending to the Editorial System.

Previously the OLD method was ...
    Note that only one pass of the file can be made in one go. So to run multiple
passes, add dummy addresses in the tables/sys/USERS file eg :

    stocks  DP:localhost    DQ:2go   CX:Stocks1 SC:NO DC:YES   DU:stocks1
    stocks1 DP:localhost    DQ:2go   CX:Stocks2 SC:NO DC:YES   DU:stocks2
    stocks2 DP:edsys1   DQ:junk-edp CX:Stocks3  SC:NO DC:YES   DF:stocks

How to use flags
- ON 'o' and OFF 'f' and FlipFlop 'p'  lines do not affect the data themselves.

- The flag is turned on/off AFTER the string.

- add spaces around the flag for better legibility, These are the same :
    x 77 :\n\n\n:\r\n

- NOT all lines are affected :
    x, X, I, o, f, r, k, K, d, D

- you can use flags within flags :
        o 22 +\n+32
    Only if flag 22 is ON, a newline will turn 32 ON.

- Test for a flag being OFF with a '#' before the number :
        f # 19 :abc:33
    if flag 19 is OFF, string 'abc' will turn flag 33 OFF (if 33 was ON)
        o # 19 |abc|33
    if flag 19 is OFF, string 'abc' will turn flag 33 ON (ignoring if 33 was ON or

- Flags 1 to 100 are acceptable.

- You want to set flags on and off before any xchgs which use them as the xchg
may zap the data you need to key off. ie Turn the first 'A' after a NL to a
    x 32 :A:B
    f 32 :A:32
will NEVER turn flag 32 OFF as the preceeding line will have modified it !
Better do :
    f 32 :A:32
    x 32 :A:B

From version 44j, you may specifiy multiple flags on/off by putting a
non-number between :
    o +\n+1,2,3,4

Current limitations are :
    Max size of an XCHG parameter file is 64k chrs.
    Max size of a search OR a replacement string is 255 chrs each.
    Max number of flags is 100 (1-100).

Running IPXCHG from a Script

The program can be used in a script to run against a single file using the '-1'
input switch. In this case, if the output file exists, it is OVERWRITTEN. If
the output queue is the same as the input, the input FILE is overwritten. The
'-D/-F/-o' switches are also advised in this case to be sure where the output
file will end up. Eg
    /fip/bin/ipxchg -1 formwork/SHARES.RUBY -F -o formsave -D crunchy
will process (and delete after) the file from spool/formwork using the xchg
file tables/xchg/CRUNCHY and leave the xchged file in formsave.
or  ipxchg -1 Cruddo -D mangle
will process Cruddo in the current queue using xchg file MANGLE.

Note that the output queue is ALWAYS under /fip/spool.

Special Case of Running an Xchg against the FipHdr

Using 'z:hdronly' allows you to ONLY modify data in the FipHdr and leave the
data part completely untouched.

Only a subset of commands are useable - some are obviously ignore like Word
format. Others not taken into account are :
    - 'h' for hdr field translate (use normal 'c' for chrset)
    - 'a' and 'A' for adding before and after the data part

You may find it advisable to use flags to limit the damage an xchg can do to a
particular FipHdr field.
For example - we want to xchg LAG, SOC and ORO in the 'NC' FipHdr field and and
word ManU in the 'NZ' :

; no chrset chg here

; turn ON HdrOnly

; Flag 1 is ON during the NC field only - turned off at the end of the line
; note we use pipes as delimiters as we have a colon ':' as part of the data
o |\nNC:| 1
f |\n| 1

; Only search for these if flag 1 is on
x 1 |LAG|PAR:12000000:LAG
x 1 |SOC|PAR:21000000:SOC
x 1 |ORO|PAR:21000000:ORO

; Flag 2 is ON during the NZ field only - turned off at the end of the line
o |\nNH:| 2
f |\n| 2

; Only search for these if flag 2 is on
x 2 |ManU|Liverpool

Example of using ipxchg to encode mail attachments
Use this for sending PDFs for example therough 'ipsmtp'

; main parameter file
; default chrset
; preserve NULs
; convert to base64 with a line length of 75 chrs
; read and parse the mime-header template beforehand
; At the end - finish the last part

... and MIMEHDR.BEFORE could look like:

Content-Type: multipart/mixed;\r

Content-Type: text/plain\r\n\r

The following is a PDF from the Fippo

Content-Disposition: attachment; filename="\SN"\r
Content-Type: application/pdf\r
Content-Transfer-Encoding: base64\r\n\r

File tracing thru the Fip

There are a series of parameters common to several Fip Programs which will log
what is happening to all files, a service or a single file.

    file-trace: (valid actions - yes/no, + hdr + in + out)
        Actions are 'Yes' - log, 'No' - dont; if NOT NO, then in addition to logging,
we can save a copy of the FipHdr, Input file and/or Output
        (delete is ignored in ipxchg))

    file-trace-name: (UniqueId in FipSeq if not default -  \SU-\SN-\HR)

    file-trace-log: (extra File Tracing logging in FipSeq)
        The Hdr and files are left in /fip/log/file_trace (which should be purged in
the nightly maintenance zapfiplog)

    These can be overridden by FipHdr fields in each file if a particular file -
or service needs tracing.
        (notice the '_' if using in the FipHdr but '-' if parameters)
    eg  ZFILE_TRACE: hdr
        ZFILE_TRACE_NAME: louislouis
        ZFILE_TRACE_LOG: seqno.\ZQ

Picking up data in the file and reusing as a FipHdr

Say you want to pick up the funny date/time stamp at the end of a wire file..

;   create TI field from wire file
; needs ipxchg.44d plus (13nov00)

 stuff the lot into the FipHdr 'TI'

; a '*' is a wild string

; do not zapt the wild buffer after the 'x' line

; strip from the ETX to the EST

; ** ENDS **

Input flags

Input options are (all optional) :

Either  -i : queue in spool to scan         default: xchg
Or  -1 : name of a single file to xchg      default: none
        In this case the input file is NOT deleted
        and the output is forced to the folder '-o' field (if specified)
        or the filename '-N' (if specified)
    -o : default destination queue if no CQ field is present. default: 2atex
        Normally a FIP hdr field, CQ, determines the output queue.
        If that is missing, this field overwrites the default.
        If this does NOT starts with a '/', it is assumed to be under /fip/spool.
    -O : display the resultant, xchged file and do NOT save to disk.
        Use this inside a CGI for example   default: disk
    -A : allow W4_ABSTRACT_DC           default: ignore
        this will run the xchg and copy the file to a temp area for ipw4 the original
file is untouched, except the W4_ABTRACT_FILE points to the xchged abstract
    -d : name of the default xchg           default: no default
        If there is more than one,separate by a comma
        eg -d DEFAULT1ST,DEFAULT2ND (48d3 onwards)
        Use '-L' to NOT log when this is used.
    -D : force the xchg file to this        def: use SC2DC or CX
    -e : Supplementary xchg filename        default: none
        This name is appended to the CX or SC2DC file name.
        This enables a 2nd group of xchges to be used.
    -E : name of a single entity file that should be run on ALL data.   def: none
            see Z:READ-ENTITY-FILE: above
    -f : file creep time                default: 0
        For files arriving from a network, this is the wait time
        Generally this should be 5 secs for RCPed files etc.
    -F : force all output to the output destination queue and ignore
        the CQ field.               default: use CQ
    -G : WINNT/2k - grab the first file only    default: no
    -h : Extra FipHdrs to be added to the input fiphdr  default: none
    -I : ignore any include-files in the parameter file default: allow
        ie ignore z:FILEBEFORE:abc.pl
    -l : log items through              default: no
    -L : do NOT complain/log files requesting non-existent xchgs
        - just use the default          default: complain
    -N : output filename (or path and filename) for single files (-1) only.
                            default: overwrite input file
    -P : Preserve the filename for the output file  default: if spooled, chged.
    -r : move the original file to this queue   default: delete it
    -R : move the original file to this queue   default: delete it
        and Change the name to CCYYMMDD.HHMMSS.(name)
    -S : display any message that is being logged.  default: log only.
    -t : scan time of directory         default: 2 secs
    -u : default is to USE-SX/USE-EXTERNAL-FILE,    default use-sx must be specified
in the param file
        (this only has effect if SX exists in the incoming file of course)
    -x : do NOT delete the original         default: delete it
        This only applies to spooled files; running single mode
        will NOT delete the input.
    -X : do NOT allow multiple xchgs in the CX FipHdr field .   default: allow
    -W : 10 ths of a second - MUST wait if nothing processed    default: 0
    -Y : one or more flags to set ON at the start of the run    default: none
        -Y 33,44
    -9 : do NOT run speedyFip with named pipes  default: do if available
    -v : print version number and exit      default: no

Version Control
;48g03  ; 17apr07 added convert-to-entities:indesign
    ;a-b 14may07 added Z:WRAP-LONG-WORDS:(length)
    ;c 05nov07 added xml for TEXT2FIPHDR
    ;d WINNT bugette ;d1-2 poss bug if single chr X and already-zapped data)
        ;3 allow multixchg for default ;4 text2fiphdr html added ;5 bugette with
        ;67 allow quoted filenames for -1 and winnt bugette ;89 hdronly in multis
    ;e4 01feb09 added convert-to/from-uudecode/uuencode ;2 TOPSPC after processing
    ;f1 25mar09 added -A allowW4Abs and W4_ABSTRACT_DC: ;2 bugette with
yanl/unyank and mulitple CX
        ;3-4 added z:convert-hdr:utf8,unicodemap and copy-file ;5-6
IGNORE-XML-IN-WRAP for entities too !
        ;7 added ABSTRACT-DATA ;8 added W4_ABSTRACT_CX ;9 entities: rtf ;10 7dec09
bugette in wrap-long-words
        ;11 15dec09 bugette with anchor tags being xchged (they should NOT be)
        ;12 bugette with missing multiple xchg ;13-14 bugette - dont use BOTTOMSPC
        ;15 3jun10 added z:convert-from-unicode ; 16 added ALWAYS-LOG and NEVER-LOG
        ;17-18  5aug10 added MAP-DATA:no/yes ;18 allow blank Z:COPY-FILE using an
option .. Z:COPY-FILE:\V1abc\$o
        ;19 16sep10 CHG - now looks for SC2DC before start of filename
        ;20-21 15nov10 better filename sizing / Win2k isspace bug
        ;22-24 wrinkleette in fipSpeedy and CONVERT-ENTITIES added
        ;25-27 19may11 added TRIM-HDR-SPACES
        ;28 29nov11 make sure output folder exists for -1/-N
        ;29 bugette in 'X' if flag specified but flag is never turned on
        ;30-2 9mar12 tuned TRIM-HDR-SPACES
        ;33 1may12 added z:fliplines:RTL and XML
        ;34 14jun12 added z:done-queue:(folder)
        ;35-44 14jul12 redid Word binary for all versions including Meta
        ;45 4sep12 added MAX-SINGLE-FIPHDR-SIZE
        ;46 8nov12 added INTERPRET-BOM to turn it off !
        ;47 5dec12 woops bugette with convert-fiphdr-to-qp with multiple lines
        ;48 3jan13 read-unicode-map can be FipSeq ; 49-50 bugette with WORD
        ;51-53 bugette in QP
        ;54 28oct13 added convert-to-entities:JSON
        ;55-60 chasing bugette in Word-DocumentSummaryInfo
        ;61-65 27jan14 fudge/fix latin1->utf8 !
        ;66 29apr14 file-trace added and ignore after if added (wrongly) to EOLSPC !!
        ;67 20jun14 maxCopies 10->30
        ;68 30jun14 bugette in convert-to-entities:JSON
        ;69 30jul14 unicodes 100 -> 1000 ; 70 if z:hdronly do NOT convert NL
        ;71-75 23aug14 allow \uFFFF as replace string .. (72-74 was BAD for utf8)
        ;76-77 bugette in replacement text
        ;78-80 added z:script ;81-85 cleanups ;85 bugette in flags not zapping buffer
        ;86-89 added z:output-filename and added speedy on -1
        ;90-92 11apr16 added mask-email and mask-numbers ;92 allow output-name in
spooled version
        ;93 WIN EACCES ;94 added z:SEQNO-NAME:ss\AB ;95-96 added MAX-ENTITY
        ;97 22sep16 bugette if readonly and -1 ;98 if FipSeq process on entry not
with the other lines (11413)
    ;g3 ;1 31oct16 added z:mask-social-media:yes ;2 bugette in script - need space
at end
;047z   bugette with multiple xchgs losing CQ
    ;a-b 19apr06 added z:replace-tag
    ;c 15may06 bugette if flag set on input or FLAG-IF-FIPHDR without other 'o'
and 'f'
    ;de 25may06 added z:convert-to-entities at last and -N newname for single
    ;fg 28may06 added z:ignore-xml-in-wrap
    ;h-k 22jun06 added z:convert-to-entities:quark and ENTITY-HASH
    ;l 25jul06 allow flags in i/I strips and fixed bugette in -Y startflags
    ;m-n 14aug06 wrinkle in convert264 one too many '=' at end
    ;o 21sep06 added WrapChr to CONVERT-FIPHDR-TO-QP
    ;p 23oct06 bugette with c:utf8/z:convert-to-utf8
    ;q-s 13nov06 added convert-encode-words:
    ;t 08dec06 added <@\> to quark-entities and removed TAB
    ;u-v 08jan07 allow ^ and % in parameter file names (v bugette in wild)
    ;w-y 08feb07 added z:FLIPLINES
    ;z 17mar07 added -3 extraFH3 plus Winnt -O is now binary
;046z3  mod for unicode-map (single bytes bugette)
    ;b 26jan04 added INPUT-FILEBEFORE/AFTER for original, raw file
        added CONVERT-FIPHDR-TO-QP
    ;c 30mar04 chrset can now be hex too \x4E
    ;d-e 19apr04 bugettes with unicode in hdr and include_file
    ;f-i 28apr04 added z:anchor
    ;j 29jun04 added max length to Convert-to-QP
    ;k-l 16aug04 added Multiple xchgs..
    ;m-q 07sep04 added -9 speedy and x/m speedup and -I
    ;r-s 01apr05 add unicode-map-is-7bit for CP850 and KOI7
    ;t 06may05 bugette with wildchrs (emmettlanebug)
        and the hex in CONVERT-FIPHDR-TO-QP should be UCASE
    ;u 30jun05 bugette in map-unicode - sometimes making a mapped chr NUL
    ;v 02aug05 bugette in i/I if string > 120 chrs
    ;w-z3 22sep05 added CONVERT-TO-UCS2 and read-entity-file - z3 thsxhg->STDBUF
;045z   16apr02 added UNMATCHED-UNICODE
    ;a/b 18apr02 added 'm' for multibyte chr as in m+\u20ac+EURO\s
    ;c 25jun02 added -G for WINNT/2k grab first file
    ;d 31jul02 z:NEWHDR - new fipHdr field may also have '$', '-' and/or '_'
    ;e 09aug02 z:FIPHDR-IS-DATA-TOO added
    ;f/g 15aug02 ':' is only matched/moved to 372 in FipHdr SH
    ;h 30oct02 bugette in flags where files has thousands of flags
    ;i 21dec02 woops - allow h:utf8 too where UTF8 chrs are in the fiphdr
    ;j 22dec02 allow z:text2fiphdr for lines at the END of text
    ;k 14jan03 bugette with h:utf8
    ;m 16jan03 CHANGE - d:<:> will now be ignored if the end is not found
    ;n 03mar03 bottomspc now zaps replaced copy too
    ;o 10mar03 added IGNORE-DATA
    ;p-q 27mar03 better filenames for Win2k plus bugette in NEWHDR with NEWDATE
    ;r-t 10apr03 added FLAG-IF-FIPHDR
    ;u-w 14apr03 added USA-DATE-FORMAT/minors
    ;x-y 25sep03 bottomspc now zaps beforehand too
    ;z 31oct03 timings
;044z   18aug00 allow multiple filebefores/afters and -h for fiphdr and -O for
    ;a/b 19aug00 added text2fiphdr
    ;c 10oct00 finally got the repeated chr bug
    ;d 03nov00 ..but that blew Yank/Unyank
    ;e 12nov00 allow SN length to be variable and allow wilds in 'r'
    ;f 20nov00 bug - stripped FipHdr for zero length data part
    ;g 06mar01 possible bug with flags and big files.
    ;h 27mar01 bugette - strips to/from (i/I) were parsed twice
    ;i 16apr01 bugette with flags again
    ;j/k 05jul01 allowed multiple flags on/off  o +\n+1,2,3,4
    ;m 03sep01 bugette with a/A and no data
    ;o/p/q 13sep01 added binary-filebefore/after and move_it added
    ;r 03oct01 bugette with FileBefore and no data in file.
    ;s 25oct01 bugette with text2fiphdr and stripping.
    ;t 14nov01 added FlipFlipFlags.
    ;u 19nov01 added z:bottomspc.
    ;v 26nov01 added c:utf8, c:utf16 and z:convert-to-utf8
    ;w 29nov01 added flags to A-add after
    ;x 29jan02 added convert-to/from-base64
    ;y 25feb02 bugette in d and D where 2nd string may be matched early
    ;z 15apr02 c:utf8 not working brilliantly.

(copyright) 2017 and previous years FingerPost Ltd.