ipdelay

CHK 734 - filtim v soon
    ipdelay

Files which are embargoed and messages for clients who wish to take advantage
of cheaper telecom tariffs are saved in a delay queue until their time of
departure is exceeded.

This program scans the delay directory once a minute and checks the FIP header
field DE which contains the release time. If this time has passed, then the
file is moved to spool/2go for program IPWHEEL to distribute normally.

The format of the FipHdr DE field is sone of two formats :
1.  a long number which matches the unix time function (default)
2.  a UTC string with/without timeoffset (if the input switch -UTC is specified)

If no DE FipHdr exists, 'CE' is used instead.

FipHdr CE can have one of four formats :
1.  If the '-R' input switch is specified :
    CE:ssss     where ssss is the number of seconds to delay by from ths moment the
file arrived.

2.  If the '-r' input switch is specified :
    CE:hh:mm+d  where hh is the hour, mm is the minute and d is no of days relative
from now

3.  If the '-r' input switch is NOT specified :
    CE:hh:mm_dd_mm  where dd/mm is the day/month

4.  If the '-r' input switch is NOT specified and the USADATE environment
variable is set :
    CE:hh:mm_mm_dd  where mm/dd is the month/day if parameter usadate is set

In CE or DE, random spaces may be inserted (and are ignored) for readability.

Any punctuation can be used as separators except HASH and you are advised NOT
to used Unix metachrs like '*' etc (except for DE, UTC version where the
punctuation BEFORE any timeoffset must be '+' or '-')

If there is a parameter file, it is (optionally) called by the '-z' switch and
holds just FipSeq - for the -N switch for example plus 'early-warning' messages

While the default is to process each and every file, the -N input switch can be
used to create and use a unique key in the hold/delay folder.
So any new item with the same key will overwrite any older version.
Whether the timer continues or is restarted (default) depends on a second
switch : -x.
If specified, any new item with the same unique filename (-N) will NOT reset
the start time
With -x also set, the delay time is NOT updated if LONGER than the original.
EG -N and -x : if a file with a delay of 5 mins (CE:300) arrives and then a
second with the same key arrives at 92 seconds with a CE of 300 or more, then
the new version is updated 108 seconds after.
But if the second file has a CE:95 it will be output just 3 seconds later.
Note that -N uses a 2 letter FipHdr field as the Key and the contents of this
FipHdr will be used; it can be an existing FipHdr in all the files or a FipSeq
one - ie the result of a combie/fixed etc

The optional Early Warning messages are to pre-announce or early warn that a
file is about to be released.
This is triggered ONLY for those files that have non-empty FipHdr field as
specified by the early-warning-fiphdr field
    early-warning-fiphdr: (FipHdr)
    early-warning-extra: (FipSeq)
        Extra FipHdr info to add to any EarlyWarning file - but NOT the main trigger
    early-warning: (no of secs before release time) (FipHdr and message in FipSeq)
        There can be a number (up to 20) early-warnings
eg
    early-warning-fiphdr: QP
    combie:QP   PR|XP
    early-warning:3600  DI:File \SN will be released in one hour
    early-warning:600   DI:File \SN will be released in 10 mins
    early-warning-extra: DA:\CA\n
    eary-warning-script: (script to run)
        Same as the -S input switch
        Run the script when outputting the early-warning message

    check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
        Use this when an ipdelay on 2 systems is accessing the same remote folder -
in a redundant way.
        that is used whether the current host should be getting the files or not.
        ie in the parameter file is
            check-primary-server:webserver
        and in the DEST_REDUN is
            ; psuedohost    primary secondary
            webserver   fip1    fip2
        and in the SYSTEM file for both fip1 AND fip2 there is a line
            rem1    local   ipdelay
        Then if fip1 is up, the ipdelay on fip1 will always send while on fip2 it
will wait, check at the last moment and NOT send
            if fip1 is down, the 'ipspool' on fip2 will start getting the files.
    hostname: (hostname)
        Hostname to use for this server for using with redundancy
    balance-group: (group)
        Name of the Group in sys/BALANCE for sending to ipdelay on the secondary
system
    extra-fiphdr: (FipHdr fields in FipSeq)
        Add these FipHdr fields/setting to all output files
    replace-DU:(FipSeq)
        replace the DU of the output file to this   default: same as incoming
    script: (script to run)
        Same as the -s switch below
        Run this script when releasing the file

    log-line: (FipSeq)
        replacement log line if input switch '-L'
        default:

    output-queue: (FipSeq)
    outque: (FipSeq)
        Output folder for files - this can be in FipSeq to vary the output path; if
under /fip/spool, you do NOT need to specify the /fip/spool
        default is spool/2go
        This overrides the -o input switch
        combie:QQ   AQ,2edsys
        outque:/fip/spool/\QQ
    done-queue: (FipSeq)
    doneque: (FipSeq)
        Done folder for incoming files - this can be in FipSeq to vary the output
path
        This overrides the -d input switch
        doneque:/fip/data/raw.data/\$e\$y\$i\$d_delay_done
    hold-queue: (FipSeq)
    holdque: (FipSeq)
        Holding folder for files in delay
        default uses the name of the input folder and replaces 'delay' with 'hold'
(or adds '_hold' if there is none)
        This overrides the -h input switch
    error-queue: (FipSeq)
    errque: (FipSeq)
        When a script is run at the end of the delay period, files which ERROR are
placed in this folder

Input parameters are (all optional) :
    -9 : do NOT run speedy on a speedy system
    -d : done queue                     default: input is deleted after sending
    -D : replacement DU                 default: same as incoming
    -f : extra fiphdr info to add to each file      default: none
        this may be overwritten by the 'extra-fiphdr' parameter, if one exists in the
parameter file
    -F : when released from hold, create new FipHdr H* fields   default: use any
existing fields
    -G : resolve all times against GMT with the released time/date  default: local
system time
        (overwriting any existing fields)       default: no
    -h : holding queue for items to be sent.        default: spool/hold_delay
        this may be overwritten by the 'holdque' parameter, if one exists in the
parameter file
    -i : queue to scan                  default: spool/delay
    -l : do NOT log when the file is released.      default: log each file when
released
    -L : log incomings too                  default: log each file when released
    -M : Maximum no of days you can delay a file        default: 90 days
    -N : 2 letter FipSeq field to create a unique filename in the hold/delay
folder.
        see notes above                 default: all items are different

    -o : output queue for sending               default: spool/2go
        if this is 'delete' the file is deleted (usually after running a script).
        this may be overwritten by the 'outque' parameter, if one exists in the
parameter file
    -q : queue to scan                  default: spool/delay
        -i and -q are interchangeable
    -Q : do not complain if hold file is missing        default: complain
    -r : date in the CE FipHdr is RELATIVE to today     default: date is specified in
dd/mm format
    -R : time in the CE FipHdr is number of secs to delay   default: date is
specified in dd/mm format
    -s : scriptname                     default: no script name
        This will be triggered Before the file is released.
        Take care not to run for a long time as the file is NOT sent until the script
has finished.
    -T : release file this many seconds BEFORE time.    default: 3
    -UTC : time in the DE FipHdr field is in UTC format default: DE holds a unix
no-of-secs-since-epoch
        ie 20050928123400
        or 20071129T115959+0430
    -w : wait in seconds after a file has been released default: 1 second
    -W : file stable time for NFS or scripted files     default: 0 secs
    -x : any new item with the same unique filename (-N) will NOT reset the start
time
                                default: restart delay time
    -Y : default delay in seconds if there is NO CE or DE   default: 0 secs
    -z : name of optional parameter file            default: none
    -v : print version no and exit

The environment variable for USA style dates is
    setenv  FIP_USA_DATE    yes

The maximum number of items is 5000.

Version Control
;11x33  19apr99 added -R and and max items 200 -> 4000
    ;a 18jun99 added delete as an option and -Q for quiet
    ;b 14jan00 WINNT mods for new version
    ;c 01feb00 added -M for maximum no of days
    ;d 27may02 on restart, -R now checks file-time. (max is now 5000)
    ;e  8dec02 added -F for new H*: FipHdr fields with date/time of release
    ;f 28apr03 added -T delay offset (default is 3)
    ;g 06jun04 bugette in WINNT stuff
    ;h-j 23sep04 speedy for outque
    ;k 09mar05 added -UTC for DE is in UTC time
    ;l 17sep05 added -N (fiphdr)
    ;m-n 11aug06 added -z pram file and early-warning:
    ;o-p 03nov06 added DR and balance
    ;q 15nov06 added offset to UTC
    ;r-t 17nov06 early-warning-extra: added and -d doneque
    ;u-w 15mar07 slice time reduced..
    ;x 23jul07 added -w - wait plus bugette in copy - not using the right fiphdr!
        ;9 10mar09 bugette for win2k and DST ;10 31mar11 added -x replace
        ;11-16 poss occasional bugette in hdr and added log-line
        ;17 added file-trace ;18-22 23jan15 added -Y defaultCE and doneque strparse
        ;23 4dec16 added replace-DU
        ;24 24aug17 added chk_size to output filename in case of v long names
        ;25-26 22dec17 bugette - did NOT save the incoming filename (in hold) - so
might miss some FipHdrs
        ;27-28 buglettes
        ;29-33 1apr20 outque in FipSeq ; 31 chksize/zap X for output and doneq ;32 to
close spOut ;33 ignore folder in holdQue

(copyright) 2024 and previous years FingerPost Ltd.