ipdelay
CHK 734 - filtim v soon
ipdelay
Files which are embargoed and messages for clients who wish to take advantage
of cheaper telecom tariffs are saved in a delay queue until their time of
departure is exceeded.
This program scans the delay directory once a minute and checks the FIP header
field DE which contains the release time. If this time has passed, then the
file is moved to spool/2go for program IPWHEEL to distribute normally.
The format of the FipHdr DE field is sone of two formats :
1. a long number which matches the unix time function (default)
2. a UTC string with/without timeoffset (if the input switch -UTC is specified)
If no DE FipHdr exists, 'CE' is used instead.
FipHdr CE can have one of four formats :
1. If the '-R' input switch is specified :
CE:ssss where ssss is the number of seconds to delay by from ths moment the
file arrived.
2. If the '-r' input switch is specified :
CE:hh:mm+d where hh is the hour, mm is the minute and d is no of days relative
from now
3. If the '-r' input switch is NOT specified :
CE:hh:mm_dd_mm where dd/mm is the day/month
4. If the '-r' input switch is NOT specified and the USADATE environment
variable is set :
CE:hh:mm_mm_dd where mm/dd is the month/day if parameter usadate is set
In CE or DE, random spaces may be inserted (and are ignored) for readability.
Any punctuation can be used as separators except HASH and you are advised NOT
to used Unix metachrs like '*' etc (except for DE, UTC version where the
punctuation BEFORE any timeoffset must be '+' or '-')
If there is a parameter file, it is (optionally) called by the '-z' switch and
holds just FipSeq - for the -N switch for example plus 'early-warning' messages
While the default is to process each and every file, the -N input switch can be
used to create and use a unique key in the hold/delay folder.
So any new item with the same key will overwrite any older version.
Whether the timer continues or is restarted (default) depends on a second
switch : -x.
If specified, any new item with the same unique filename (-N) will NOT reset
the start time
With -x also set, the delay time is NOT updated if LONGER than the original.
EG -N and -x : if a file with a delay of 5 mins (CE:300) arrives and then a
second with the same key arrives at 92 seconds with a CE of 300 or more, then
the new version is updated 108 seconds after.
But if the second file has a CE:95 it will be output just 3 seconds later.
Note that -N uses a 2 letter FipHdr field as the Key and the contents of this
FipHdr will be used; it can be an existing FipHdr in all the files or a FipSeq
one - ie the result of a combie/fixed etc
The optional Early Warning messages are to pre-announce or early warn that a
file is about to be released.
This is triggered ONLY for those files that have non-empty FipHdr field as
specified by the early-warning-fiphdr field
early-warning-fiphdr: (FipHdr)
early-warning-extra: (FipSeq)
Extra FipHdr info to add to any EarlyWarning file - but NOT the main trigger
early-warning: (no of secs before release time) (FipHdr and message in FipSeq)
There can be a number (up to 20) early-warnings
eg
early-warning-fiphdr: QP
combie:QP PR|XP
early-warning:3600 DI:File \SN will be released in one hour
early-warning:600 DI:File \SN will be released in 10 mins
early-warning-extra: DA:\CA\n
eary-warning-script: (script to run)
Same as the -S input switch
Run the script when outputting the early-warning message
check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
Use this when an ipdelay on 2 systems is accessing the same remote folder -
in a redundant way.
that is used whether the current host should be getting the files or not.
ie in the parameter file is
check-primary-server:webserver
and in the DEST_REDUN is
; psuedohost primary secondary
webserver fip1 fip2
and in the SYSTEM file for both fip1 AND fip2 there is a line
rem1 local ipdelay
Then if fip1 is up, the ipdelay on fip1 will always send while on fip2 it
will wait, check at the last moment and NOT send
if fip1 is down, the 'ipspool' on fip2 will start getting the files.
hostname: (hostname)
Hostname to use for this server for using with redundancy
balance-group: (group)
Name of the Group in sys/BALANCE for sending to ipdelay on the secondary
system
extra-fiphdr: (FipHdr fields in FipSeq)
Add these FipHdr fields/setting to all output files
replace-DU:(FipSeq)
replace the DU of the output file to this default: same as incoming
script: (script to run)
Same as the -s switch below
Run this script when releasing the file
log-line: (FipSeq)
replacement log line if input switch '-L'
default:
output-queue: (FipSeq)
outque: (FipSeq)
Output folder for files - this can be in FipSeq to vary the output path; if
under /fip/spool, you do NOT need to specify the /fip/spool
default is spool/2go
This overrides the -o input switch
combie:QQ AQ,2edsys
outque:/fip/spool/\QQ
done-queue: (FipSeq)
doneque: (FipSeq)
Done folder for incoming files - this can be in FipSeq to vary the output
path
This overrides the -d input switch
doneque:/fip/data/raw.data/\$e\$y\$i\$d_delay_done
hold-queue: (FipSeq)
holdque: (FipSeq)
Holding folder for files in delay
default uses the name of the input folder and replaces 'delay' with 'hold'
(or adds '_hold' if there is none)
This overrides the -h input switch
error-queue: (FipSeq)
errque: (FipSeq)
When a script is run at the end of the delay period, files which ERROR are
placed in this folder
Input parameters are (all optional) :
-9 : do NOT run speedy on a speedy system
-d : done queue default: input is deleted after sending
-D : replacement DU default: same as incoming
-f : extra fiphdr info to add to each file default: none
this may be overwritten by the 'extra-fiphdr' parameter, if one exists in the
parameter file
-F : when released from hold, create new FipHdr H* fields default: use any
existing fields
-G : resolve all times against GMT with the released time/date default: local
system time
(overwriting any existing fields) default: no
-h : holding queue for items to be sent. default: spool/hold_delay
this may be overwritten by the 'holdque' parameter, if one exists in the
parameter file
-i : queue to scan default: spool/delay
-l : do NOT log when the file is released. default: log each file when
released
-L : log incomings too default: log each file when released
-M : Maximum no of days you can delay a file default: 90 days
-N : 2 letter FipSeq field to create a unique filename in the hold/delay
folder.
see notes above default: all items are different
-o : output queue for sending default: spool/2go
if this is 'delete' the file is deleted (usually after running a script).
this may be overwritten by the 'outque' parameter, if one exists in the
parameter file
-q : queue to scan default: spool/delay
-i and -q are interchangeable
-Q : do not complain if hold file is missing default: complain
-r : date in the CE FipHdr is RELATIVE to today default: date is specified in
dd/mm format
-R : time in the CE FipHdr is number of secs to delay default: date is
specified in dd/mm format
-s : scriptname default: no script name
This will be triggered Before the file is released.
Take care not to run for a long time as the file is NOT sent until the script
has finished.
-T : release file this many seconds BEFORE time. default: 3
-UTC : time in the DE FipHdr field is in UTC format default: DE holds a unix
no-of-secs-since-epoch
ie 20050928123400
or 20071129T115959+0430
-w : wait in seconds after a file has been released default: 1 second
-W : file stable time for NFS or scripted files default: 0 secs
-x : any new item with the same unique filename (-N) will NOT reset the start
time
default: restart delay time
-Y : default delay in seconds if there is NO CE or DE default: 0 secs
-z : name of optional parameter file default: none
-v : print version no and exit
The environment variable for USA style dates is
setenv FIP_USA_DATE yes
The maximum number of items is 5000.
Version Control
;11x33 19apr99 added -R and and max items 200 -> 4000
;a 18jun99 added delete as an option and -Q for quiet
;b 14jan00 WINNT mods for new version
;c 01feb00 added -M for maximum no of days
;d 27may02 on restart, -R now checks file-time. (max is now 5000)
;e 8dec02 added -F for new H*: FipHdr fields with date/time of release
;f 28apr03 added -T delay offset (default is 3)
;g 06jun04 bugette in WINNT stuff
;h-j 23sep04 speedy for outque
;k 09mar05 added -UTC for DE is in UTC time
;l 17sep05 added -N (fiphdr)
;m-n 11aug06 added -z pram file and early-warning:
;o-p 03nov06 added DR and balance
;q 15nov06 added offset to UTC
;r-t 17nov06 early-warning-extra: added and -d doneque
;u-w 15mar07 slice time reduced..
;x 23jul07 added -w - wait plus bugette in copy - not using the right fiphdr!
;9 10mar09 bugette for win2k and DST ;10 31mar11 added -x replace
;11-16 poss occasional bugette in hdr and added log-line
;17 added file-trace ;18-22 23jan15 added -Y defaultCE and doneque strparse
;23 4dec16 added replace-DU
;24 24aug17 added chk_size to output filename in case of v long names
;25-26 22dec17 bugette - did NOT save the incoming filename (in hold) - so
might miss some FipHdrs
;27-28 buglettes
;29-33 1apr20 outque in FipSeq ; 31 chksize/zap X for output and doneq ;32 to
close spOut ;33 ignore folder in holdQue
(copyright) 2024 and previous years FingerPost Ltd.