ipformd This is the daemon for data formats. It uses a parameter file is used to route and process incoming files. This parameter defaults to tables/form/PROCESS. It first uses a selection table to decide what the job really is. As the list is top down, only the first valid selection is processed. If the incoming file has a FipHdr 'CJ' then the contents of that are used as the name of the Job. The 'jobname' found is usually the name of a parameter file in tables/form/text. IPFORMD will automatically start IPFORMAT with parameters of the input file and the jobname/parameter file. However, optionally IPFORMD can be used to run a sequence of 'jobs' specified for a particular jobname. The syntax for the PROCESS file is : ; comment locale:(name of locale) default: current system locale - which may be none ; the following is a selection line (hdr field) = string [opt (tab) & (hdr) = string ...] (tab) >jobname (nl) job: (jobname) (program to run) trace: (jobname) test: (jobname) default-output-name: (Outputname in FipSeq) This is used to determine any "$o" for output name which is NORMALLY used for single files (-1) and should have had a -o input switch. But it can also be used in a general catch-all where the same job is used in the same PROCESS file for single shot and spooled jobs. Note that the '-o' name is used if specified ! eg default-output-name:/fip/spool/2go/#SU:form#DU:\AB#SN:\SN#XX:\$u\$z default is default-output-name:/fip/x/TMP.form.\$u\$v If the outname does not start with a '/' it is assumed being under /fip/spool continue-on-error: continue/stop if a script/program returns an error, it is logged in the Item log and further processing stops unless the continue-on-error: parameter has been et to 'continue' this switch should be used with caution as normally, if a program has errored, the data for subsequent programs will be affected default is 'stop' max-timeout: (secs) Max time in seconds that any job should run for before it is stopped default 1800 (for 30 mins) Specify 0 (zero) for no max - ie infinite wait. Use this if you have jobs which may hang - such as scripts running 'zip' ! double-carats: (yes/no) For early Win2k systems (pre 2008), a '^' needed to be doubled in filenames default: no No, this has nothing to do with Gromit. You can also use the normal FipSeq (from 21u onwards) to break apart FipHdr fields : partial, combie, option, repeat, style, replace, newdate, unique, valid, swop, lookup, perl, merge, sum To describe the Selection syntax in detail : (hdr field) = string [opt (tab) & (hdr) = string ...] (tab) >jobname (nl) Each selection is on a single line. If necessary, multiple conditions can be specified with the '&' to 'and' them. The operation equal, '=', can also be NOT equal '!='. Source Header fields (in SH) are preceeded by X, ie XC for category. '*' signifies a wild card string; '?' signifies a single wild card chr. So to search for a string/chr embedded somewhere in a field, use a '*' before and after. If embedded spaces are needed in the string-to-be-searched, put double quotes arond the string. If you do NOT know how many spaces there may be, use the '*' to specify one or many. eg "Long Beach" "Long*Beach" Note that the search string is case_insensitive. Both the selection file and the main file are scanned completely, so that one file may be sent to none, one or several jobs according to the same or different criteria. To search inside the data for a string use '$$' as the Pseudo FipHdr field. Always put double quotes around the string if there are spaces and you MUST add the wild chr '*' before and after if the string does not either start or finish the data. eg SU=race & $$="*Rosecroft*" >rosecroft For the 'job' parameter : - '$i' refers to the input file name (Note \$i is still the FIP System Variable 'month') - all queues and files are assumed to be under /fip/spool - Never assume however that the path environment has been setup, so we advise you specify full pathnames for the programs. - all 'job' lines MUST precede the selection - ie be above. - FIP System variables and Header fields can be accessed. - there can be one or many or very many job lines. - any program can be run - if a script/program returns an error, it is logged in the Item log and further processing stops (unless the continue-on-error: parameter has been specified) If a 'job' exists for a jobname, ipformd will NOT run ipformat but will run what is specified - which may be ipformat of course. The 'trace' parameter is used for setup, tuning and testing a new job. All it does is tell IPFORMD to log each line in the Item log. EG: trace:shares Trace MUST always be specified in the PROCESS file BEFORE the jobs (ie on a line nearer the top of the file) and all jobs must be before the selection lines. The 'test' parameter does the same as 'trace' BUT NONE of the programs are actually run. This allows you just to check syntax etc. EG: test:racecards Test MUST always be specified in the PROCESS file BEFORE the jobs (ie on a line nearer the top of the file) and all jobs must be before the selection lines. First examples - simple jobs : ; Sports copy to Tranmere Rovers FC. XC=S* & XK=*Football* >tranmere EP=TAR. >tarmac RD=*Broken_Hill* >bhp where TRANMERE, TARMAC and BHP are all various parameter files in tables/form/text Second example of a sequence of jobs. Here, if the file starts 'borsen', IPFORMAT is strted twice (sequencially, serially - ie NOT at the same time) using two parameter files in tables/form/text : WKENDSHARES and DAILYSHARES . ; shares : for both weekday and weekend job:shares /fip/bin/ipformat -p dailyshares -i $i job:shares /fip/bin/ipformat -p wkendshares -i $i ; Selection for job called 'shares' SN=borsen* > shares Third example shows how to sort, xchg and basically really screw around. ; Reformat, sort and generally destroy the horses ... job:geges /bin/rm -f formsave/NAGS* job:geges /fip/bin/ipformat -p geges -i $i -D -S NAGS job:geges /fip/bin/ipxchg -1 formsave/NAGS -D geges -F -o formsave job:geges /bin/sort +0 -3 -o formsave/NAGS.done formsave/NAGS job:geges /bin/mv formsave/NAGS.done 2go/#SN:\SN#DU:nagsdone ; Selection for job called 'geges' SU=RACEWIRE & SN=horse* > geges Fourth Example is where the 'test' parameter is added to the 'geges' job in the Third example (Remember the 'test' must be specified on a line at the top of the PROCESS file BEFORE any 'jobs' for the same jobname) : test:geges Going into the MUI, ip, and doing a 'l' to list the log (or 'm' to more) gives: Sat Mar 4 11:52:30 ipformd !i : Incoming File : geges : : geges Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /fip/bin/ipformat -p geges -i /fip/spool/form/geges -D -S NAGS Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /fip/bin/ipxchg -1 formsave/NAGS -D geges -F -o formsave Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /bin/sort +0 -3 -o formsave/NAGS.done formsave/NAGS Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /bin/mv formsave/NAGS.done 2go/#SN:geges#DU:nagsdone Example of using FipSeq : ; JobsWeek-CWK - ccibookings ; Filename in SN is '100805a.txt' .. ; .. but the input to the combine booking files is just '100805a' repeat:J1 SN,.,1 combie:J0 J1|SN job:cw_xcci /bin/cp -fp "$i" /d1/esad/news/low/opi/LOOT_BOOKINGS/SEPARATE/\SN job:cw_xcci /fip/local/jwcombine_booking_files "\J0" Using Single shot .. ipformd can be used to run against a single file using the -1 input switch. ipformd -1 //winsvr33/data/ In this case the file resides on another server - winsvr33. You can also specify the job with -j and also the output filename with -o (if using $o inside a job) eg in the Process file job:wigwam /perl/bin/perl /fip/local/geronimo.pl in="$i" out="$o" ipformd -1 /fip/data/customer/custer -o /fip/spool/2go/\#DU:laststand\#XX:\\$u -j wigwam (If you are running ipformd inside a script, pls note you almost always have to escape the backslash as a double \\, and often the Hash/Pound # and any dollar $ too) Input parameters are (all optional) : -1 : name of a single file to run and then stop default: spooled -b : path to binaries - TESTING new versions only default: /fip/bin -c : or -d : done or copy folder into which copies of all default: no copies incoming files are made after processing. -E : name of locale default: current system locale -f : file creep time default: 0 -g : TestMode ipformat - ignore newfile. default: no -G : TestMode ipformat - ignore outque and name default: no -h : extra Fip Hdr to be added to all incoming files. default: none -H : allow the runline to have FipSeq default: no eg ipformat -h YC:\YC -p onetest -1 testme -i : queue to scan default: spool/form -I : id for logging default: none eg ipformd -i form_soccer -I soccer -z PROCESS.SOCCER -j : name of job to run (-1 single file only) default: use CJ: FipHdr or check selection -l : do NOT log every incoming file/job default: log -m : stop after this stage number default: run whole job -n : run the program at reduced priority default: nice 5 -o : output name to use for "$o" default: none -p : processing file to use (same as -z) default: PROCESS in tables/form -Q : quiet mode - do NOT log if an incoming file does NOT match any criteria for running a job. The file is ignored. default: log as an error -s : run files serially (ie one after the other) default: parallel -S : output file for Selection tests default: none This switch will NOT run any job - just display which selection has been triggered form this file Used by form_manage.pl in DataFormatting Admin ipformd -1 (DataFileName) -p (ProcessFileName) -S(OutputFileName - normally a tmp fle) -t : scan time of directory default: 3 secs -T : Always trace jobs. This is the same as the 'trace' parameter used for setup, tuning and testing a new job. All it does is tell IPFORMD to log each line in the Item log. def: no -x : debugging ON - ALL incoming files will be 'broken out' in formtest parameter is 'o'ctal, 'd'ecimal or 'h'ex. default: off -Y : log long jobs which take longer than default: no -Y 10 for 10 secs -z : process file to use (same as -p) default: PROCESS in tables/form -Z : calm down time default: 5 secs To attempt to let ipformat finish one job before the next This is only used if ipformd is runnig jobs in parallel. -v : display version number and exit. ----- Other Points worth noting (ish) ----- -- Break out - If either the input parameter -x or a header field FZBO is present, the input file is 'broken apart' into blocks, records and fields. The resultant file is called (jobname)_(SN) in spool/formtest where jobname is as above and SN is the filename. -- If an incoming file matches none of the tests, it is deleted and no further processing occurs. If the '-Q' input switch is NOT on, it is logged in the Item log as an error. If '-Q' is specified, it is ignored completely. -- In the selection file, remember to specify long names first. In the following example, job 'sunrac2' never gets processed as all files will be jobbed as 'sunrac' XK:RAC* >sunrac XK:RACING* >sunrac2 - It is worth making sure that ipformd is ALWAYS started from a logon which has all the environment setup correctly - especially PATH. If /fip/local/ and /fip/bin/ are not in the PATH then 'ipformd' will try to add these paths to any program which fails. But it will slow things down slightly. Version Control ;22j17 24jul06 cleanups ;b-e added -E locale and -z (old -z is now -Z) ;f-g 19feb07 added -S seltest ;h-i1 17mar07 added -h for extra fiphdr (;i1 buglette with $o) ;j8 29aug08 winn bugette with new DF ;3 10aug10 allow FipSeq in runline/ turn ON with -H ;4 added max-timeout ;5 added locale: as a parameter ;6-11 added -I id and -Y longjobs and double-carats: and -a formqte and -g testmode for ipformat ;12-15 30apr14 added file-trace ; 11sep17 allow fiphdr FORM_PROCESS to determine which PROCESS file to use (15 bug) ;16 4jan18 added -G more comprehensive TESTmode for ipformat (disable name and outque) ;021z 19apr99 chasing the elusive 'No Child' - turns out to be Environment plus add new test - $$ for text search. ;a 21may99 buggette in do_job for $i ;b/c/d/e/f/g/h/j 20aug99 redid do_exec_format slightly PLUS bugette in cnk condition ;k 14dec00 bugette for first file thru after reboot on Sparcs/FIP_GCC ;l 28dec00 PROCESS can now be any size. ;m 22aug01 PROCESS file cannot handle 'DC:\SC' ;n 22nov01 allow PROCESS.fip if WINNT ;o 01jan03 MACOSX ;p 22jan04 added -b for path for testing ;q-r 19sep04 speedy ;s-t 01jul05 added -1 ;u-w 10aug05 added widgets and parse the selection (;w -d added) ;x-z 07jun06 added -o as "$o" for single file -1 only Plus -j or FipHdr CJ: as single job name to run Plus allow multiple jobs to run with a '+' Plus added default-output-name Plus added continue-on-error (copyright) 2019 and previous years FingerPost Ltd.