ipformd
ipformd
This is the daemon for data formats.
It uses a parameter file is used to route and process incoming files. This
parameter defaults to tables/form/PROCESS.
It first uses a selection table to decide what the job really is. As the list
is top down, only the first valid selection is processed.
If the incoming file has a FipHdr 'CJ' then the contents of that are used as
the name of the Job.
The 'jobname' found is usually the name of a parameter file in
tables/form/text.
IPFORMD will automatically start IPFORMAT with parameters of the input file and
the jobname/parameter file.
However, optionally IPFORMD can be used to run a sequence of 'jobs' specified
for a particular jobname.
The syntax for the PROCESS file is :
; comment
locale:(name of locale)
default: current system locale - which may be none
; the following is a selection line
(hdr field) = string [opt (tab) & (hdr) = string ...] (tab) >jobname (nl)
job: (jobname) (program to run)
trace: (jobname)
test: (jobname)
default-output-name: (Outputname in FipSeq)
This is used to determine any "$o" for output name which is NORMALLY used for
single files (-1) and should have had a -o input switch.
But it can also be used in a general catch-all where the same job is used in
the same PROCESS file for single shot and spooled jobs.
Note that the '-o' name is used if specified !
eg default-output-name:/fip/spool/2go/#SU:form#DU:\AB#SN:\SN#XX:\$u\$z
default is default-output-name:/fip/x/TMP.form.\$u\$v
If the outname does not start with a '/' it is assumed being under
/fip/spool
continue-on-error: continue/stop
if a script/program returns an error, it is logged in the Item log and
further processing stops unless the continue-on-error: parameter has been et to
'continue'
this switch should be used with caution as normally, if a program has
errored, the data for subsequent programs will be affected
default is 'stop'
max-timeout: (secs)
Max time in seconds that any job should run for before it is stopped default
1800 (for 30 mins)
Specify 0 (zero) for no max - ie infinite wait.
Use this if you have jobs which may hang - such as scripts running 'zip' !
double-carats: (yes/no)
For early Win2k systems (pre 2008), a '^' needed to be doubled in filenames
default: no
No, this has nothing to do with Gromit.
You can also use the normal FipSeq (from 21u onwards) to break apart FipHdr
fields :
partial, combie, option, repeat, style, replace, newdate, unique, valid,
swop, lookup, perl, merge, sum
To describe the Selection syntax in detail :
(hdr field) = string [opt (tab) & (hdr) = string ...] (tab) >jobname (nl)
Each selection is on a single line. If necessary, multiple conditions
can be specified with the '&' to 'and' them.
The operation equal, '=', can also be NOT equal '!='.
Source Header fields (in SH) are preceeded by X, ie XC for category.
'*' signifies a wild card string;
'?' signifies a single wild card chr.
So to search for a string/chr embedded somewhere in a field,
use a '*' before and after.
If embedded spaces are needed in the string-to-be-searched, put double
quotes arond the string. If you do NOT know how many spaces
there may be, use the '*' to specify one or many.
eg "Long Beach"
"Long*Beach"
Note that the search string is case_insensitive.
Both the selection file and the main file are scanned completely, so
that one file may be sent to none, one or several jobs
according to the same or different criteria.
To search inside the data for a string use '$$' as the Pseudo FipHdr
field. Always put double quotes around the string if there are
spaces and you MUST add the wild chr '*' before and after if
the string does not either start or finish the data.
eg SU=race & $$="*Rosecroft*" >rosecroft
For the 'job' parameter :
- '$i' refers to the input file name (Note \$i is still the FIP System
Variable 'month')
- all queues and files are assumed to be under /fip/spool
- Never assume however that the path environment has been setup, so we advise
you specify full pathnames for the programs.
- all 'job' lines MUST precede the selection - ie be above.
- FIP System variables and Header fields can be accessed.
- there can be one or many or very many job lines.
- any program can be run
- if a script/program returns an error, it is logged in the Item log and
further processing stops (unless the continue-on-error: parameter has been
specified)
If a 'job' exists for a jobname, ipformd will NOT run ipformat but will run
what is specified - which may be ipformat of course.
The 'trace' parameter is used for setup, tuning and testing a new job. All it
does is tell IPFORMD to log each line in the Item log. EG:
trace:shares
Trace MUST always be specified in the PROCESS file BEFORE the jobs (ie on a
line nearer the top of the file) and all jobs must be before the selection
lines.
The 'test' parameter does the same as 'trace' BUT NONE of the programs are
actually run. This allows you just to check syntax etc. EG:
test:racecards
Test MUST always be specified in the PROCESS file BEFORE the jobs (ie on a line
nearer the top of the file) and all jobs must be before the selection lines.
First examples - simple jobs :
; Sports copy to Tranmere Rovers FC.
XC=S* & XK=*Football* >tranmere
EP=TAR. >tarmac
RD=*Broken_Hill* >bhp
where TRANMERE, TARMAC and BHP are all various parameter files in
tables/form/text
Second example of a sequence of jobs. Here, if the file starts 'borsen',
IPFORMAT is strted twice (sequencially, serially - ie NOT at the same time)
using two parameter files in tables/form/text : WKENDSHARES and DAILYSHARES .
; shares : for both weekday and weekend
job:shares /fip/bin/ipformat -p dailyshares -i $i
job:shares /fip/bin/ipformat -p wkendshares -i $i
; Selection for job called 'shares'
SN=borsen* > shares
Third example shows how to sort, xchg and basically really screw around.
; Reformat, sort and generally destroy the horses ...
job:geges /bin/rm -f formsave/NAGS*
job:geges /fip/bin/ipformat -p geges -i $i -D -S NAGS
job:geges /fip/bin/ipxchg -1 formsave/NAGS -D geges -F -o formsave
job:geges /bin/sort +0 -3 -o formsave/NAGS.done formsave/NAGS
job:geges /bin/mv formsave/NAGS.done 2go/#SN:\SN#DU:nagsdone
; Selection for job called 'geges'
SU=RACEWIRE & SN=horse* > geges
Fourth Example is where the 'test' parameter is added to the 'geges' job in the
Third example (Remember the 'test' must be specified on a line at the top of
the PROCESS file BEFORE any 'jobs' for the same jobname) :
test:geges
Going into the MUI, ip, and doing a 'l' to list the log (or 'm' to more) gives:
Sat Mar 4 11:52:30 ipformd !i : Incoming File : geges : : geges
Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /fip/bin/ipformat -p geges -i
/fip/spool/form/geges -D -S NAGS
Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /fip/bin/ipxchg -1 formsave/NAGS
-D geges -F -o formsave
Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /bin/sort +0 -3 -o
formsave/NAGS.done formsave/NAGS
Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /bin/mv formsave/NAGS.done
2go/#SN:geges#DU:nagsdone
Example of using FipSeq :
; JobsWeek-CWK - ccibookings
; Filename in SN is '100805a.txt' ..
; .. but the input to the combine booking files is just '100805a'
repeat:J1 SN,.,1
combie:J0 J1|SN
job:cw_xcci /bin/cp -fp "$i" /d1/esad/news/low/opi/LOOT_BOOKINGS/SEPARATE/\SN
job:cw_xcci /fip/local/jwcombine_booking_files "\J0"
Using Single shot ..
ipformd can be used to run against a single file using the -1 input switch.
ipformd -1 //winsvr33/data/
In this case the file resides on another server - winsvr33.
You can also specify the job with -j and also the output filename with -o (if
using $o inside a job)
eg in the Process file
job:wigwam /perl/bin/perl /fip/local/geronimo.pl in="$i" out="$o"
ipformd -1 /fip/data/customer/custer -o /fip/spool/2go/\#DU:laststand\#XX:\\$u
-j wigwam
(If you are running ipformd inside a script, pls note you almost always have to
escape the backslash as a double \\, and often the Hash/Pound # and any dollar
$ too)
Input parameters are (all optional) :
-1 : name of a single file to run and then stop default: spooled
-b : path to binaries - TESTING new versions only default: /fip/bin
-c : or
-d : done or copy folder into which copies of all default: no copies
incoming files are made after processing.
-E : name of locale default: current system locale
-f : file creep time default: 0
-g : TestMode ipformat - ignore newfile. default: no
-G : TestMode ipformat - ignore outque and name default: no
-h : extra Fip Hdr to be added to all incoming files. default: none
-H : allow the runline to have FipSeq default: no
eg ipformat -h YC:\YC -p onetest -1 testme
-i : queue to scan default: spool/form
-I : id for logging default: none
the can be fipseq (from 22j18) eg -I \XK
eg ipformd -i form_soccer -I soccer -z PROCESS.SOCCER
-j : name of job to run (-1 single file only) default: use CJ: FipHdr or check
selection
-l : do NOT log every incoming file/job default: log
-m : stop after this stage number default: run whole job
-n : run the program at reduced priority default: nice 5
-o : output name to use for "$o" default: none
-p : processing file to use (same as -z) default: PROCESS in tables/form
-Q : quiet mode - do NOT log if an incoming file does
NOT match any criteria for running a job.
The file is ignored. default: log as an error
-s : run files serially (ie one after the other) default: parallel
-S : output file for Selection tests default: none
This switch will NOT run any job - just display which selection has been
triggered form this file
Used by form_manage.pl in DataFormatting Admin
ipformd -1 (DataFileName) -p (ProcessFileName) -S(OutputFileName - normally a
tmp fle)
-t : scan time of directory default: 3 secs
-T : Always trace jobs. This is the same as the 'trace' parameter
used for setup, tuning and testing a new job. All it does
is tell IPFORMD to log each line in the Item log. def: no
-x : debugging ON - ALL incoming files will be 'broken out' in formtest
parameter is 'o'ctal, 'd'ecimal or 'h'ex. default: off
-Y : log long jobs which take longer than default: no
-Y 10 for 10 secs
-z : process file to use (same as -p) default: PROCESS in tables/form
-Z : calm down time default: 5 secs
To attempt to let ipformat finish one job before the next
This is only used if ipformd is runnig jobs in parallel.
-v : display version number and exit.
----- Other Points worth noting (ish) -----
-- Break out - If either the input parameter -x or a header field FZBO is
present, the input file is 'broken apart' into blocks, records and fields. The
resultant file is called (jobname)_(SN) in spool/formtest where jobname is as
above
and SN is the filename.
-- If an incoming file matches none of the tests, it is deleted and no further
processing occurs. If the '-Q' input switch is NOT on, it is logged in the Item
log as an error. If '-Q' is specified, it is ignored completely.
-- In the selection file, remember to specify long names first. In the
following example, job 'sunrac2' never gets processed as all files will be
jobbed as 'sunrac'
XK:RAC* >sunrac
XK:RACING* >sunrac2
- It is worth making sure that ipformd is ALWAYS started from a logon
which has all the environment setup correctly - especially PATH. If
/fip/local/ and /fip/bin/ are not in the PATH then 'ipformd' will
try to add these paths to any program which fails. But it will
slow things down slightly.
Version Control
;22j22 24jul06 cleanups
;b-e added -E locale and -z (old -z is now -Z)
;f-g 19feb07 added -S seltest
;h-i1 17mar07 added -h for extra fiphdr (;i1 buglette with $o)
;j8 29aug08 winn bugette with new DF ;3 10aug10 allow FipSeq in runline/ turn
ON with -H
;4 added max-timeout
;5 added locale: as a parameter
;6-11 added -I id and -Y longjobs and double-carats: and -a formqte and -g
testmode for ipformat
;12-15 30apr14 added file-trace ; 11sep17 allow fiphdr FORM_PROCESS to
determine which PROCESS file to use (15 bug)
;16 4jan18 added -G more comprehensive TESTmode for ipformat (disable name and
outque)
;17-18 minor -I strparse
;19 allow jobname to also match '^^jobname' - useful for CJ:jobname
;20-21 donque tuning
;22 8nov24 redid tests in ChkCon for strparse
;021z 19apr99 chasing the elusive 'No Child' - turns out to be Environment
plus add new test - $$ for text search.
;a 21may99 buggette in do_job for $i
;b/c/d/e/f/g/h/j 20aug99 redid do_exec_format slightly
PLUS bugette in cnk condition
;k 14dec00 bugette for first file thru after reboot on Sparcs/FIP_GCC
;l 28dec00 PROCESS can now be any size.
;m 22aug01 PROCESS file cannot handle 'DC:\SC'
;n 22nov01 allow PROCESS.fip if WINNT
;o 01jan03 MACOSX
;p 22jan04 added -b for path for testing
;q-r 19sep04 speedy
;s-t 01jul05 added -1
;u-w 10aug05 added widgets and parse the selection (;w -d added)
;x-z 07jun06 added -o as "$o" for single file -1 only
Plus -j or FipHdr CJ: as single job name to run
Plus allow multiple jobs to run with a '+'
Plus added default-output-name
Plus added continue-on-error
(copyright) 2024 and previous years FingerPost Ltd.