IPSPOOL is used to bring files into FIP from another system. It runs against an
input queue and any file found is sent to the output queue possibly with FIP
header information added possibly running a program/script against it.

Mac resource forks may be zapped at the same time. For files arriving across
the network, a file creep time may be used to wait for the file to complete
before it is processed.

The file contents are normally left unaltered but a small change - removing the
CR of a CR NL combination may be used for incoming PC files.

A filename mask may be specified where other programs use the same queue for
temporary files. Only files matching the mask are processed.

The incoming file name is used for the output name except that certain
characters are removed and replaced by '_'. These include space, tab, control
chrs (deleted), $, *, /, |, \, < and > and single and double quotes.
This can be stopped used the -n switch and/or quoted with to -N (quotename)

Note that in order to pass parameters to a script specified in -p, put double
quotes around. For example :
    ipspool -i eyeye -o delete -p "scripto param1 \SN"
will run script called 'scripto' (which MUST be in /fip/local if no path is
specified) and pass it parameters :
    Parameter 1 : the fixed string 'param1'
    Parameter 2 : whatever is in the Fip Hdr field SN
    Parameter 3 : The full path and filename of the input file

Errors from the script will be logged - 256 or higher means that the script
cannot be found or does not have execute permissions.

Note that all files starting with a dot are ignored and, on Unix, any files
called 'core' are assumed to be core dumps and are ignored !

If the '-A' switch or 'allowsubques:' parameter is used, if a queue is dropped
into the input queue, then the contents of that queue are treated in the normal
manner but any sub-subqueues are flagged as error. If there is a done queue,
the subqueue is moved into that and the a subfolder in the output folder is
created too unless the folder is '2go', 'xchg', '2edsys' or '2brouted' (or the
no-output-subqueues: parameter is specified).
- Normally the sub folder is removed when empty, so use input switch -a or
parameter 'preserve-subqueues:' to leave alone.
- Normally the sub folder is created in the output folder (as mentioned above),
so use input switch -Q or parameter 'no-output-subqueues:'.

Using subqueues and any MAC switch means the program will ignore queues like
'.rsrc', 'Trash', Network Trash Folder' and 'DeskTop'.

When accessing/writing to queues that are physically on another system - NFS
drives for example - it is a good idea to check that drive is actually mounted
Two switches, -I and -O (and their identical parameter file keywords,
chkexists-input and chkexists-output) can be used to check whether a specified
file or queue exists before the input queue is scavenged (or output written).
As filenames starting with a dot are ignored, one way to implement this is to
touch a file in this input queue. eg :
    cd /mountedOnRemore/spool/input
    touch .DontZapMePls
While in the SYSTEM file we could have something like :
remo    local   ipspool -i /mountedOnRemore/spool/input -I .DontZapMePls -o 2go -h

Input switches are :
    Mandatory parameters :
    -i : input q (normally in ~/spool)          no default
        if the path starts with a '/', it is considered a hard path
    -o : output q (normally in ~/spool) or 'delete' to delete it.  no default
        if the path starts with a '/', it is considered a hard path

Advised parameters :
Either  -Z : name of the parameter file to use      default: none

Or  -h : hdr string to be added BEFORE the filename in the output q
        This parameter is advised but not mandatory.    no default
        Note for NT this has no effect on the FipHdr,
        so use the 'extra-FipHdr-fields' parameter to create a FipHdr
        with meaningful fields that can be used for other Fip programs.
        For Unix, please do NOT add any '/' as it will create subqueues.
    -H : hdr string to be added AFTER the filename in the output q
        This parameter is advised but not mandatory.    no default
        Note for NT this has no effect on the FipHdr,
        so use the 'extra-FipHdr-fields' parameter to create a FipHdr
        with meaningful fields that can be used for other Fip programs.
        For Unix, please do NOT add any '/' as it will create subqueues.

Optional parameters :
    -2 : (linux only) use Inotify to trigger a new file default: do not
        Inotify will ONLY work on a local filesystem - and NOT NFS.
    -8 : input is NOT speedy (valid on a speedy system only)
        ie the input folder is NOT piped but the rest of the system is.
    -9 : do NOT run speedy (valid on a speedy system only)
    -A : allow files to be placed in sub queues in the main input queue.  default:
    -a : if using subqueues, do NOT zap  empty subfolders   default: zap em
    -b : default round robin maximum    default is none
        see also parameter 'round-robin:'
    -B : balance queue for save-data - same as save-data-balance-folder default:
    -c : copy the file NOT link. This is for files residing on
        a different volume to /fip/spool        default: link
    -C : supercede: Overwrite any existing files        default: NO
    -d : done queue for a copy of the raw input file    default: none
        if the path starts with a '/', it is considered a hard path
    -e : error queue if script returns an error     default: same as output
        if the path starts with a '/', it is considered a hard path
    -E : ignore the file if the script returns an error default: move file to '-o'
    -f : file wait (ie wait x secs for file to finish before processing)
                                default: 5 secs.
    -h : FipHdr before filename (see above)
    -H : FipHdr after filename (see above)
    -i : Input folder name (see above)
    -I : Check File/Queue name for input            default: none
        see above
        If the path starts with a '/', it is considered a hard path
        otherwise it is considered relative to the input queue.
    -j : input folder is FipSeq - so can contain dates etc. default: no - fixed
        eg ipspool -j -i /data/input/BW/\$e\$y\%i\$d -Z BW.FIP
    -k : filename mask. Send only files containing this string.
                    default: send all files
    -K : filename unmask. Do NOT Send files containing this string.
                    default: send all files
    -l : do NOT log each file through           default: log.
    -m : for Mac files : zap the ResFork file pls
        Removes the resource fork           default: no
        See also -R for the resfork Path/name
    -M : for Mac files : move the ResFork file to the done que. def: no
        Note that you must ONLY move resource forks within AppleShare
        volumes as they contain unique ids which are guaranteed to
        screw up if moved between vols!
        See also -R for the resfork Path/name
    -n : do NOT look for Unix meta characters in the filename.  Normally
        $, * <> |\/ etc are changed to '_'.   default: change to '_'.
    -N : Put single quotes around the filename  default: no quotes
        when running a script. This enables filenames with spaces and other
metacharacters to be passed.
    -o : Output folder name (see above)
    -O : Check File/Queue name for output       default: none
        see above
        If the path starts with a '/', it is considered a hard path
        otherwise it is considered relative to the output queue.
    -p : program or script to execute BEFORE sending    default: none
    -P : parameters for the script specified by '-p'    default: none
    -q : quiet mode - do not message if we find directories or other
        non-files are found in the input queue - just ignore them.
                    default: log message if directory found
    -r : replace CR or CR LF with just LF
                    default: leave file contents alone.
    -R : for Mac Files Res Fork path/prefix         default: .rsrc/
        For Mac NFS where the resourcefork is a second file in the
        queue with a '%' in front : use -R%
    -s : Minimum size for file          default: no
        Files smaller than this are deleted
        (or, if the parameter min-size-action is set, moved to the done que or woops
    -S : Maximum size for file          default: no
        Files larger than this are deleted
        (or, if the parameter max-size-action is set, moved to the done que or woops
    -t : sleep between scans of the directory       default: 5 secs
    -T : maximum time allowable for a script before it is killed. def: no limit
        This is to catch scripts/programs which are looping or are
        flow controlled or for someother reason they have hung.
    -W : add a FipHdr and add normal FipHdr fields such as the
        date and time (HH, HM etc) plus internal fields
        such as EN, EP and EQ           default: no
    -X : If both Mask and Unmask are set and a file meets both criteria,
        Send it.        default: do NOT send it.
    -z : log file for the executing program or script
        if you have a program running against each file, this can
        be used to log the result.          default: none

    -v : show version no and exit

Using the -Z switch you may (optionally) specify a parameter file in
tables/spooler can be used to provide much the same info as the other input
switches (plus a bit more !!). If both switches and parameter file are used,
the parameter file takes precedent.
The parameter files are held in tables/spooler.

Parameters are the usual FIP style of :
    ; comment line
    (keyword): (subparams) ....
and include :
    hdrbefore:  same as switch -h
    hdrafter:   same as switch -H
    script:     same as switch -p
    param-script:   same as switch -P
    logscript:  same as switch -z
    timeout:    same as switch -T
    copy:       same as switch -c
    nolog:      same as switch -l
    minsize:    same as switch -s
    maxsize:    same as switch -S
    wait:       same as switch -f
    outque:     same as switch -o
    doneque:    same as switch -d
    errorque:   same as switch -e
    strip:      same as switch -r
    zapmac:     same as switch -m
    movemac:    same as switch -M
    respath:    save as switch -R
    quiet:      same as switch -q
    mask:       same as switch -k
    unmask:     same as switch -K
    meta:       same as switch -n
    quotename:  same as switch -N
    allowsubques    same as switch -A   allow-subqueues and allow-subfolders are all
    preservesubques same as switch -a   preserve-subqueues and preserve-subfolders
are all identical
    no-output-subqueues:    same as switch -Q
    add-all-fiphdr-fields:  same as switch -W
    no-fiphdr:  same as switch -X

Plus these extra ones
    wait-for-data: (FipSeq) Do not process the file until this string
        appears in the data. In this case the -f input switch is used as a
        MAXIMUM time.
    extra-fiphdr-fields:    FipSeq to add to the FipHdr.
        eg extra-fiphdr-fields:#SU:outgoing#SC:notes#SN:\EN#PR:\EQ
    extra-fiphdr-script:    program which will create more FipHdr fields
        \E3 hold the name of a TMP file to create that will be read for the list.
        \E4 hold the name of a TMP file that holds the contents of the incoming file
        eg extra-fiphdr-script:/jpeg/rdjpgcom -fiphdr \E4 > \E3
    fiphdr-input-filename: (2 letter FipHdr field)
        Add the complete Path and filename of the INPUT file as this FipHdr field
        eg if the input folder is /fip/spool/2edsys and the filename is
        Note '#' are mapped to x9D as it is normally an end of FipHdr field marker.
            gives   PN:/fip/spool/2edsys/^SN:tester^DQ:2edsys
            where '^' is used to denote a '#'
    fiphdr-input-file-hash:(FipSeq chr)
        When changing the filename (which has 'hash' signs) for a script, use this
        default is x9D
    winnt-input-drive: (letter)
    winnt-output-drive: (letter)
    winnt-done-drive: (letter) drive letter of the queue if not the default
    wait-for-remote-server: (secs) On flaky networks, the remote drives
        may not be available all the time. Use this parameter to log it
        has gone missing, pause a few seconds and retry - but do not stop the
        Default is none. If specified with (secs) missing, the default is 10
    number: octal/dec/hex   Number system to use for FipSeq data
    first:      FipSeq data to insert at the top of the file
    last:       FipSeq data to insert at the bottom of the file
    prefile:    full path and filename of a file to be included
            after any 'first' text and before file text.
    endfile:    full path and filename of a file to be included
    chkexists-input: same as -I switch
    chkexists-output: same as -O switch
    chkexists-input-script: Script (in FipSeq) to run before accessing folder
        If this script does NOT return '0', it is assumed to be an error.
    allowsubques:   same as -A switch
    supercede:  Overwrite any existing file default is NO.
        This also sets the default if you have several masks/unmasks
    save-data-path: (pathname for data)
        This puts the data of the incoming file into this folder and creates a FipHdr
        file that contains 2 FipHdrs containing the full path/filename
            SX: and FTP_EXTERNAL_FILE:
        (ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE)
            eq  save-data-path:/fip/data/jpegs/\$e\$y\$i\$d/
        Use this for big files that you do not want to copy around the Fip Spool
    save-data-filename: (FipSeq name)
        Use this to specify exactly what the the 'save-data-path' name should be
        default is (incoming filename).(time).(seqno)
        eg  save-data-filename:\HR-\SU.raw
    save-data-fiphdr: (yes/no)
        Save any incoming FipHdr with the saved file    default: yes
    save-data-balance-group: (Balance group)
        Balance all save-data files to the following group. default: do not balance
    save-data-balance-folder: (Balance folder)
        If balancing, put the token in this folder under spool. default: 2balance
    create-checksum: (2 letter FipHdr field)
        Create a 32 letter MD5 signature checksum in this FipHdr field
        Many web-type organisations which allow downloads will also give, separately,
an MD5
        signature. Use this to compare so you can verify the data has not been
tampered with.
        eg  create-checksum:CS
            Create a FipHdr field CS like 'CS:62663683bbbd3b84779d072f53178919'
    round-robin: (number)
        Send to output queues ending with a digit up to this number
        default: none
        ie if round-robin:3 and outque:/data/today
        The first file through will be put in a folder /data/today1, the 2nd in
/data/today2 etc
    round-robin-fiphdr: (2 letter FipHdr field)
        Add this Fiphdr to the output file so it can be used downstream
        default: none
    tracker-script: (fullpath to script name)
        default: none
    zap-empty-subfolders: opposite of -a/preserve-subfolders.
        if using subqueues, zap  empty subfolders   default: zap em
        The parameter can be Yes/No or a time in seconds for the folder to be empty
before it is zapped
    ignore-mime-header: (yes/no)
        A file that has been grabbed by webwire or an email program may still have
the MIME header prefixing the data.
        Use this switch to ignore/keep the mime-header. Normally, if using
'file-type', you will want to ignore it.
        default - no
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/pdf
Date: Wed, 20 Mar 2013 14:30:30 GMT
Connection: close

.. data continues

    wildmask:*  Define a character to be used as a wild-card for masks.
            default: none
    wildsingle:?    Define a character to be used as any single chr for masks.
            default: none
    exact-match:yes/no  NO (default) - the mask is a string which can be ANYWHERE
in the
                    eg mask:8   will match any file with '8' in the name
                YES - the match is an exact match
                    eg mask:1954*   will ONLY match files starting 1954....
    file-type0,...file-type9    check the first 4 bytes of the file for its 'magic'
    check-mask-first: yes/no
            Change logic ....
            Either check mask/unmask before checking file-type (YES)
            or check file-type BEFORE checking masks (NO)

    Up to 10 mask and/or unmasks can be specified.
    Any of the numbers can be used - they do not have to be sequential.
    If ANY unmask if found, the file is ignored.
    If more than one mask is found, the one with the lower number is used.
    If the optional script is required, if there is a script of the SAME number,
it is run else 'script' is run. Masks/Unmasks can be in FipSeq.
    The file which matches the mask/unmask can also be moved to a different folder
with 'outqueX' where X is the index of the (un)mask.
    Supercede2 etc can be either YES or NO.
    The base 'wait-for-data' is used as a default for any index which does NOT
have a wait-for-data.
    Note that 'fiphdr0' etc add additional fiphdr fields to files matching that
one mask. While 'extra-fiphdr-fields' is added to ALL fields irrespective of
the mask.

    Note that if 'no-fiphdr' is specified, all other options which use the fipHdr
to store metadata are ignored including (for example)

    Example of using masks
        ; we have an input folder which can receive several different
        ; types of data. Each needs a different Fip destination (and
        ; other FipHdr info) so it can be treated differently
        ; using '*' as the bits to ignore
        ; 1. stream off all xml files
        ; 1.1 stuff into /fip/spool/sgml
        ; 1.2 add the relevant fiphdr info (DY is the ipsgml param file)
        ; 2 stream off the pdfs
        ; 2.1 just copy to edsys to place in a nice folder somewhere
        ; 2.2
        ; 3. catch all for all the other files
        ; 3.1 send them via iproute to check more
        ; 3.2 route needs a SU (source) and ZI for archive

    check-primary-server: pseudo-host name that is specified in
        Use this when an ipspool on 2 systems is accessing the same remote folder -
in a redundant way.
        that is used whether the current host should be getting the files or not.
        ie in the parameter file REMOTEGET is
        and in the DEST_REDUN is
            ; psuedohost    primary secondary
            remotewire  fip1    fip2
        and in the SYSTEM file for both fip1 AND fip2 there is a line
            rem1    local   ipspool -i /nfs/data/input -o 2go -Z REMOTEGET
        Then if fip1 is up, the ipspool on fip1 will always get while on fip2 it will
just check/loop.
            if fip1 is down, the 'ipspool' on fip2 will start getting the files.
    hostname: (hostname)
        Hostname to use for this server for using with redundancy
    check_redun-every: (no of secs)
        If running redundant systems with check-primary-server, this is the delay
between checks for the secondary system.
        default is 60 secs.
        Show timings statistics     default: no
    wait-between-files: (tenths of a sec)
        small pause between each file.. in tenths of a sec
        default is 10 for 1 sec (max is 100 for 10 secs)
        eq  wait-between-files:2

    fiphdr-original-time:(2 letter FipHdr field)
        Add the UTC time when the file was created as this FipHdr field.

    blacklist-repeats: (yes/no)             default: no
    blacklist-repeats-folder: (folder name in spool)    default: spool/woops
        Use this, if the same filename is repeatedly sent because it is NOT deletable
and you wish to deduplicate/blacklist it. Perhaps the file is locked on the
remote server and you do NOT wish copies to be sent out.
        Use the folder to move the file (if possible to move it) out of the Fip copy
flow (if spool/woops is not ideal)

        There can be up to 10 copies of the same file
        Take care to speicfy the FULL path AND filename in fipseq


Use this for big files that you do not want to copy around the Fip Spool area.

Use save-data-path to squirrel a copy of the data file while the 'output' file
is just a small FipHdr file.

Normally you would also want    'add-all-fiphdr-fields'
2 extra FipHdrs fields are created containing the full path/filename
    (ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE)
        eq  save-data-path:/fip/data/jpegs/\$e\$y\$i\$d/

Any version of ipftp (from version 14k) will pick it up.

** Note it is up to you when you purge the external or save folder - in the
above example /fip/data/jpegs/(todays date)

If you have one file in -> one ftp out, then get ipftp to delete it by adding
FTP_ZAP_EXTERNAL: to the fiphdr - again use ipspool for this

Otherwise if you have masses of disk space, just add a line in the nightly
maintenance /fip/local/zapfiplog (or zapfiplog.cmd) to wipe out the files after
a day or so.
On Unix/Linux :
    # save 8 days of jpegs ..
    if [ -d /fip/data/jpegs ]; then
       /fip/bin/ipdelque -q /fip/data/jpegs -A 8 -i 0

- For AIX 4.1 : If calling a script using the '-p' switch or the 'script:'
parameter, you MUST make sure the first line of the script starts with a '#!'.
    eg for a bourne shell :     #! /bin/sh

- Always make sure you exit from a script with an 'exit 0' if it has finished

- Note that some internal FipHdr fields are available to the program
    They are NOT added to the output file automatically.
    But you can do automatically with the keyword 'add-all-fiphdr-fields'
        which will also add all the HH/HD/HM fields for date and time
    SN - output filename
    EN - input filename
    EQ - lastpart of the input folder/queue
    EP - full input folder
    see also fiphdr-input-filename: above which is how to handle filenames with

WIN 2K and UNC drives

If using UNC drives, note that the Win2k 'security model' no longer allows the
default service logon (LocalSystem) to access external drives. So you will need
to start the 'Fip Comms Manager' service under a Logon which exists on the
remote system AND has permissions to read AND WRITE to the folders you want to
read and/or write.

You will need to
either  - The easiest method is to have the same logon/password for both systems
        - if running a Domain server, use a single logon for both
        - if not create/use the same logon/password on both
or  - The remote system needs to know about the local system's logon and have
permissions to read/write

How do you know it will work
    - Logon the Fip/local server with the logon
    - Open RUN and type the full UNC path
        - try reading and/or writing a file
        - if it does not work, there is no way Fip will either ! So fix it first
    - When that works..
        - Open CMD window and change to the drive  where Fip is.
        - type in the complete ipspool parameter line from the SYSTEM file
            (remember to have NO spaces between input switch and parameter ie -i2proc
not -i 2proc)
            - if that does not work, check your syntax and ogon again
    - When that works..
        - Add the logon to the ControlPanel->Admin Tasks->Services->Fip Comms Manager
            - stop and start

- Note for Fip UNC drives, always use '/' rather than '\' as the latter is the
escape chr (so you need '\\' for every '\' !)

- For NT note that the path name for a script is using forward slashes NOT
 The '-P' or param-script keyword however is not changed to forward slashes

Version Control
;32t17  21oct05 added create-checksum:
    ;a-b 17jan06 added -b and round-robin and slightly better logging
    ;c 23jan06 added -8
    ;d 29mar06 -q also affects locked files
        and possible bugette with NOT cleaning out remnants of missing files.
    ;e-h 05aug06 for -A subqueues : added -a for preserve-subfolders and -Q
    ;i 18jan07 bugette - supercede0 was missing and added param-script0-9 and
    ;j-k 22mar07 bugette - do not use DEST_REDUN if not needed
    ;l 04oct07 added filesize at the end of the log message
    ;m-n 24oct07 added extra-fiphdr-script and save-data-filename:
    ;o 31oct07 check for primary server before each file in addition to once every
folder scan
    ;p2 08nov07 added copy-file0-9 and no-fiphdr0-9
    ;r1-6 29nov07 added rename in doneque ;r1 added fiphdr-original-time: ;r2
added E4 ;r3 bugette with -p /-P ;r4 for fipredunBalanced
    ;s1 31jul07 added save-data-balance-group/queue and -B ;3456 bugette WINNT
curdrv and speedy
        ;7-8 bugette in wait-for-data ;9 redid -r ;10 added round-robin-fiphdr ;11
added file-type: 0-9
        ;12 added tracker-script ;13 added secs to wait-for-remote-server ;14
file-type tuned ; 15 usleeps tuned
        ;16 allow dbl qtes around any input switch que - input, balance, error etc
        ;17 added chkexists-input-script
        ;18-20 5oct11 bugette with subques (both linux and win2k) and added filter
;20 added wildsingle
        ;23-26 19oct11 added check-mask-first:yes/no,subques - do NOT allow '..' ;26
and '.'
        ;27 stop spins on network drives which have disappeared
        ;28-32 1oct12 -j added parse input queue (and bugette in exec)
        ;33 10oct12 added min-size-action and max-size-action and wrong-size-wait
        ;34-36 added linux Notify   ;37 added fiphdr-input-filename and
        ;38 added ignore-mime-header
    ;t1-9 11apr13 allow up to 20 subfolders (was 1 !) ;3-4 added winnt-done-drive
;8 30apr14 file-trace ;9 outq parse better
        ;10-12 make_fip_unid added ;13-15 allow doneq NOT on same drive
        ;16 added blacklist-repeats
        ;17 31oct16 added log-line/log
;031z1  01feb01 added wait-for-data ;a bugette with masks when using -k/-K
    ;b 01may01 masks are now FipSeq
    ;c 11may01 added outque as a parameter
    ;d 14jun01 added outque1-9 according to mask/umask
    ;e 17aug01 small problem with mask/unmask and wild at the END
    ;f 13sep01 added supercede2-9
    ;g 02jan02 added mask1/unmask1/supercede1 etc
        and found the bugette where SN sometimes disappeared!
    ;h 03oct02 added fipseq functions - partial/repeat/style etc
    ;i 16oct02 bugette in FipHdr bit
    ;j 15nov02 added redundancy/check-primary-server
    ;k/l 01jan03 MACOSX
    ;m 23may03 added fiphdr0-9
    ;n 06aug03 bug in outque/outque1
    ;o 10feb04 added filename0-9
    ;p 15may04 added timing-stats
    ;q 01jul04 bugette in incoming fipHdrs.
    ;r-x 07oct04 speedy (w allow drive letter for script)
    ;y 18jul05 added wait-for-remote-server
    ;z1 11oct05 added save-data-path:/fip/data/incoming/\$d/
        doneque is now FipSeq and not just hard paths
;030    30nov00 upto 9 masks/unmasks and scripts
;029f   01sep99 added extra-fiphdr and add-all-fiphdr-fields
    ;a 13sep99 NT mods for drives
    ;b 15sep99 added -E
    ;c 28oct99 added toomanyfiles check
    ;d 14jan00 small WINNT hdr mod
    ;e/f 20jan00 supercede added

(copyright) 2017 and previous years FingerPost Ltd.