ipspool IPSPOOL is used to bring files into FIP from another system. It runs against an input queue and any file found is sent to the output queue possibly with FIP header information added possibly running a program/script against it. Mac resource forks may be zapped at the same time. For files arriving across the network, a file creep time may be used to wait for the file to complete before it is processed. The file contents are normally left unaltered but a small change - removing the CR of a CR NL combination may be used for incoming PC files. A filename mask may be specified where other programs use the same queue for temporary files. Only files matching the mask are processed. The incoming file name is used for the output name except that certain characters are removed and replaced by '_'. These include space, tab, control chrs (deleted), $, *, /, |, , < and > and single and double quotes. This can be stopped used the -n switch and/or quoted with to -N (quotename) switch. Note that in order to pass parameters to a script specified in -p, put double quotes around. For example : ipspool -i eyeye -o delete -p "scripto param1 SN" will run script called 'scripto' (which MUST be in /fip/local if no path is specified) and pass it parameters : Parameter 1 : the fixed string 'param1' Parameter 2 : whatever is in the Fip Hdr field SN Parameter 3 : The full path and filename of the input file Errors from the script will be logged - 256 or higher means that the script cannot be found or does not have execute permissions. Note that all files starting with a dot are ignored and, on Unix, any files called 'core' are assumed to be core dumps and are ignored ! If the '-A' switch or 'allowsubques:' parameter is used, if a queue is dropped into the input queue, then the contents of that queue are treated in the normal manner but any sub-subqueues are flagged as error. If there is a done queue, the subqueue is moved into that and the a subfolder in the output folder is created too unless the folder is '2go', 'xchg', '2edsys' or '2brouted' (or the no-output-subqueues: parameter is specified). - Normally the sub folder is removed when empty, so use input switch -a or parameter 'preserve-subqueues:' to leave alone. - Normally the sub folder is created in the output folder (as mentioned above), so use input switch -Q or parameter 'no-output-subqueues:'. Using subqueues and any MAC switch means the program will ignore queues like '.rsrc', 'Trash', Network Trash Folder' and 'DeskTop'. When accessing/writing to queues that are physically on another system - NFS drives for example - it is a good idea to check that drive is actually mounted beforehand. Two switches, -I and -O (and their identical parameter file keywords, chkexists-input and chkexists-output) can be used to check whether a specified file or queue exists before the input queue is scavenged (or output written). As filenames starting with a dot are ignored, one way to implement this is to touch a file in this input queue. eg : cd /mountedOnRemore/spool/input touch .DontZapMePls While in the SYSTEM file we could have something like : remo local ipspool -i /mountedOnRemore/spool/input -I .DontZapMePls -o 2go -h "#DU:Toinfinity#SN:" Input switches are : Mandatory parameters : -i : input q (normally in ~/spool) no default if the path starts with a '/', it is considered a hard path -o : output q (normally in ~/spool) or 'delete' to delete it. no default if the path starts with a '/', it is considered a hard path Advised parameters : Either -Z : name of the parameter file to use default: none Or -h : hdr string to be added BEFORE the filename in the output q This parameter is advised but not mandatory. no default Note for NT this has no effect on the FipHdr, so use the 'extra-FipHdr-fields' parameter to create a FipHdr with meaningful fields that can be used for other Fip programs. For Unix, please do NOT add any '/' as it will create subqueues. -H : hdr string to be added AFTER the filename in the output q This parameter is advised but not mandatory. no default Note for NT this has no effect on the FipHdr, so use the 'extra-FipHdr-fields' parameter to create a FipHdr with meaningful fields that can be used for other Fip programs. For Unix, please do NOT add any '/' as it will create subqueues. Optional parameters : -2 : (linux only) use Inotify to trigger a new file default: do not Inotify will ONLY work on a local filesystem - and NOT NFS. -8 : input is NOT speedy (valid on a speedy system only) ie the input folder is NOT piped but the rest of the system is. -9 : do NOT run speedy (valid on a speedy system only) -A : allow files to be placed in sub queues in the main input queue. default: no -a : if using subqueues, do NOT zap empty subfolders default: zap em -b : default round robin maximum default is none see also parameter 'round-robin:' -B : balance queue for save-data - same as save-data-balance-folder default: none -c : copy the file NOT link. This is for files residing on a different volume to /fip/spool default: link -C : supercede: Overwrite any existing files default: NO -d : done queue for a copy of the raw input file default: none if the path starts with a '/', it is considered a hard path -e : error queue if script returns an error default: same as output if the path starts with a '/', it is considered a hard path -E : ignore the file if the script returns an error default: move file to '-o' -f : file wait (ie wait x secs for file to finish before processing) default: 5 secs. -h : FipHdr before filename (see above) -H : FipHdr after filename (see above) -i : Input folder name (see above) -I : Check File/Queue name for input default: none see above If the path starts with a '/', it is considered a hard path otherwise it is considered relative to the input queue. -j : input folder is FipSeq - so can contain dates etc. default: no - fixed eg ipspool -j -i /data/input/BW/$e$y%i$d -Z BW.FIP -k : filename mask. Send only files containing this string. default: send all files -K : filename unmask. Do NOT Send files containing this string. default: send all files -l : do NOT log each file through default: log. -m : for Mac files : zap the ResFork file pls Removes the resource fork default: no See also -R for the resfork Path/name -M : for Mac files : move the ResFork file to the done que. def: no Note that you must ONLY move resource forks within AppleShare volumes as they contain unique ids which are guaranteed to screw up if moved between vols! See also -R for the resfork Path/name -n : do NOT look for Unix meta characters in the filename. Normally $, * <> |/ etc are changed to '_'. default: change to '_'. -N : Put single quotes around the filename default: no quotes when running a script. This enables filenames with spaces and other metacharacters to be passed. -o : Output folder name (see above) -O : Check File/Queue name for output default: none see above If the path starts with a '/', it is considered a hard path otherwise it is considered relative to the output queue. -p : program or script to execute BEFORE sending default: none -P : parameters for the script specified by '-p' default: none -q : quiet mode - do not message if we find directories or other non-files are found in the input queue - just ignore them. default: log message if directory found -r : replace CR or CR LF with just LF default: leave file contents alone. -R : for Mac Files Res Fork path/prefix default: .rsrc/ For Mac NFS where the resourcefork is a second file in the queue with a '%' in front : use -R% -s : Minimum size for file default: no Files smaller than this are deleted (or, if the parameter min-size-action is set, moved to the done que or woops queue) -S : Maximum size for file default: no Files larger than this are deleted (or, if the parameter max-size-action is set, moved to the done que or woops queue) -t : sleep between scans of the directory default: 5 secs -T : maximum time allowable for a script before it is killed. def: no limit This is to catch scripts/programs which are looping or are flow controlled or for someother reason they have hung. -W : add a FipHdr and add normal FipHdr fields such as the date and time (HH, HM etc) plus internal fields such as EN, EP and EQ default: no -X : If both Mask and Unmask are set and a file meets both criteria, Send it. default: do NOT send it. -z : log file for the executing program or script if you have a program running against each file, this can be used to log the result. default: none -v : show version no and exit Using the -Z switch you may (optionally) specify a parameter file in tables/spooler can be used to provide much the same info as the other input switches (plus a bit more !!). If both switches and parameter file are used, the parameter file takes precedent. The parameter files are held in tables/spooler. Parameters are the usual FIP style of : ; comment line (keyword): (subparams) .... and include : hdrbefore: same as switch -h hdrafter: same as switch -H script: same as switch -p param-script: same as switch -P logscript: same as switch -z timeout: same as switch -T copy: same as switch -c nolog: same as switch -l minsize: same as switch -s maxsize: same as switch -S wait: same as switch -f outque: same as switch -o doneque: same as switch -d errorque: same as switch -e strip: same as switch -r zapmac: same as switch -m movemac: same as switch -M respath: save as switch -R quiet: same as switch -q mask: same as switch -k unmask: same as switch -K meta: same as switch -n quotename: same as switch -N allowsubques same as switch -A allow-subqueues and allow-subfolders are all identical preservesubques same as switch -a preserve-subqueues and preserve-subfolders are all identical no-output-subqueues: same as switch -Q add-all-fiphdr-fields: same as switch -W no-fiphdr: same as switch -X Plus these extra ones wait-for-data: (FipSeq) Do not process the file until this string appears in the data. In this case the -f input switch is used as a MAXIMUM time. extra-fiphdr-fields: FipSeq to add to the FipHdr. eg extra-fiphdr-fields:#SU:outgoing#SC:notes#SN:EN#PR:EQ extra-fiphdr-script: program which will create more FipHdr fields E3 hold the name of a TMP file to create that will be read for the list. E4 hold the name of a TMP file that holds the contents of the incoming file eg extra-fiphdr-script:/jpeg/rdjpgcom -fiphdr E4 > E3 fiphdr-input-filename: (2 letter FipHdr field) Add the complete Path and filename of the INPUT file as this FipHdr field eg if the input folder is /fip/spool/2edsys and the filename is '#SN:tester#DQ:2edsys' Note '#' are mapped to x9D as it is normally an end of FipHdr field marker. fiphdr-input-filename:PN gives PN:/fip/spool/2edsys/^SN:tester^DQ:2edsys where '^' is used to denote a '#' fiphdr-input-file-hash:(FipSeq chr) When changing the filename (which has 'hash' signs) for a script, use this chr default is x9D winnt-input-drive: (letter) winnt-output-drive: (letter) winnt-done-drive: (letter) drive letter of the queue if not the default wait-for-remote-server: (secs) On flaky networks, the remote drives may not be available all the time. Use this parameter to log it has gone missing, pause a few seconds and retry - but do not stop the program. Default is none. If specified with (secs) missing, the default is 10 number: octal/dec/hex Number system to use for FipSeq data first: FipSeq data to insert at the top of the file last: FipSeq data to insert at the bottom of the file prefile: full path and filename of a file to be included after any 'first' text and before file text. endfile: full path and filename of a file to be included chkexists-input: same as -I switch chkexists-output: same as -O switch chkexists-input-script: Script (in FipSeq) to run before accessing folder If this script does NOT return '0', it is assumed to be an error. allowsubques: same as -A switch supercede: Overwrite any existing file default is NO. This also sets the default if you have several masks/unmasks save-data-path: (pathname for data) This puts the data of the incoming file into this folder and creates a FipHdr file that contains 2 FipHdrs containing the full path/filename SX: and FTP_EXTERNAL_FILE: (ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE) eq save-data-path:/fip/data/jpegs/$e$y$i$d/ Use this for big files that you do not want to copy around the Fip Spool area. save-data-filename: (FipSeq name) Use this to specify exactly what the the 'save-data-path' name should be default is (incoming filename).(time).(seqno) eg save-data-filename:HR-SU.raw save-data-fiphdr: (yes/no) Save any incoming FipHdr with the saved file default: yes save-data-balance-group: (Balance group) Balance all save-data files to the following group. default: do not balance save-data-balance-folder: (Balance folder) If balancing, put the token in this folder under spool. default: 2balance create-checksum: (2 letter FipHdr field) Create a 32 letter MD5 signature checksum in this FipHdr field Many web-type organisations which allow downloads will also give, separately, an MD5 signature. Use this to compare so you can verify the data has not been tampered with. eg create-checksum:CS Create a FipHdr field CS like 'CS:62663683bbbd3b84779d072f53178919' round-robin: (number) Send to output queues ending with a digit up to this number default: none ie if round-robin:3 and outque:/data/today The first file through will be put in a folder /data/today1, the 2nd in /data/today2 etc round-robin-fiphdr: (2 letter FipHdr field) Add this Fiphdr to the output file so it can be used downstream default: none tracker-script: (fullpath to script name) default: none zap-empty-subfolders: opposite of -a/preserve-subfolders. if using subqueues, zap empty subfolders default: zap em The parameter can be Yes/No or a time in seconds for the folder to be empty before it is zapped ignore-mime-header: (yes/no) A file that has been grabbed by webwire or an email program may still have the MIME header prefixing the data. Use this switch to ignore/keep the mime-header. Normally, if using 'file-type', you will want to ignore it. default - no eg HTTP/1.1 200 OK Cache-Control: private Content-Type: application/pdf Date: Wed, 20 Mar 2013 14:30:30 GMT Connection: close %PDF-1.4 .. data continues MASKING ------- wildmask:* Define a character to be used as a wild-card for masks. default: none wildsingle:? Define a character to be used as any single chr for masks. default: none exact-match:yes/no NO (default) - the mask is a string which can be ANYWHERE in the eg mask:8 will match any file with '8' in the name YES - the match is an exact match eg mask:1954* will ONLY match files starting 1954.... mask0,...,mask9 unmask0,...,unmask9 script0,...,script9 param-script0,...,param-script9 logscript0,...,logscript9 outque0,...,outque9 supercede0,...,supercede9 fiphdr0,...fiphdr9 no-fiphdr0,...no-fiphdr9 filename0,...filename9 file-type0,...file-type9 check the first 4 bytes of the file for its 'magic' number wait-for-data0...wait-for-data9 check-mask-first: yes/no Change logic .... Either check mask/unmask before checking file-type (YES) or check file-type BEFORE checking masks (NO) Up to 10 mask and/or unmasks can be specified. Any of the numbers can be used - they do not have to be sequential. If ANY unmask if found, the file is ignored. If more than one mask is found, the one with the lower number is used. If the optional script is required, if there is a script of the SAME number, it is run else 'script' is run. Masks/Unmasks can be in FipSeq. The file which matches the mask/unmask can also be moved to a different folder with 'outqueX' where X is the index of the (un)mask. Supercede2 etc can be either YES or NO. The base 'wait-for-data' is used as a default for any index which does NOT have a wait-for-data. Note that 'fiphdr0' etc add additional fiphdr fields to files matching that one mask. While 'extra-fiphdr-fields' is added to ALL fields irrespective of the mask. Note that if 'no-fiphdr' is specified, all other options which use the fipHdr to store metadata are ignored including (for example) save-data-path extra-fiphdr-fields extra-fiphdr-script Example of using masks ; we have an input folder which can receive several different ; types of data. Each needs a different Fip destination (and ; other FipHdr info) so it can be treated differently ; using '*' as the bits to ignore wildmask:* ; 1. stream off all xml files mask2:*xml* ; 1.1 stuff into /fip/spool/sgml outque2:sgml ; 1.2 add the relevant fiphdr info (DY is the ipsgml param file) fiphdr2:DY:MYXML ; 2 stream off the pdfs mask4:*pdf ; 2.1 just copy to edsys to place in a nice folder somewhere outque4:2edsys ; 2.2 fiphdr2:DF:SEND.PDF ; 3. catch all for all the other files mask9:* ; 3.1 send them via iproute to check more outque9:2brouted ; 3.2 route needs a SU (source) and ZI for archive fiphdr9:SU:TXT#ZI: check-primary-server: pseudo-host name that is specified in tables/sys/DEST_REDUN Use this when an ipspool on 2 systems is accessing the same remote folder - in a redundant way. that is used whether the current host should be getting the files or not. ie in the parameter file REMOTEGET is check-primary-server:remotewire and in the DEST_REDUN is ; psuedohost primary secondary remotewire fip1 fip2 and in the SYSTEM file for both fip1 AND fip2 there is a line rem1 local ipspool -i /nfs/data/input -o 2go -Z REMOTEGET Then if fip1 is up, the ipspool on fip1 will always get while on fip2 it will just check/loop. if fip1 is down, the 'ipspool' on fip2 will start getting the files. hostname: (hostname) Hostname to use for this server for using with redundancy check_redun-every: (no of secs) If running redundant systems with check-primary-server, this is the delay between checks for the secondary system. default is 60 secs. timing-stats:yes/no Show timings statistics default: no wait-between-files: (tenths of a sec) small pause between each file.. in tenths of a sec default is 10 for 1 sec (max is 100 for 10 secs) eq wait-between-files:2 fiphdr-original-time:(2 letter FipHdr field) Add the UTC time when the file was created as this FipHdr field. copy-file0...copy-file9 There can be up to 10 copies of the same file Take care to speicfy the FULL path AND filename in fipseq copy-file:/fip/spool/2balance1/#SN:SN#SU:bal1#XX:$u.fip copy-file2:/fip/spool/2balance2/#SN:SN#SU:bal2#XX:$u.fip copy-file6:/fip/spool/2balance6/#SN:SN#SU:bal6#XX:$u.fip SAVING FILES FOR IPFTP ---------------------- Use this for big files that you do not want to copy around the Fip Spool area. Use save-data-path to squirrel a copy of the data file while the 'output' file is just a small FipHdr file. Normally you would also want 'add-all-fiphdr-fields' 2 extra FipHdrs fields are created containing the full path/filename SX: and FTP_EXTERNAL_FILE: (ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE) eq save-data-path:/fip/data/jpegs/$e$y$i$d/ Any version of ipftp (from version 14k) will pick it up. ** Note it is up to you when you purge the external or save folder - in the above example /fip/data/jpegs/(todays date) If you have one file in -> one ftp out, then get ipftp to delete it by adding FTP_ZAP_EXTERNAL: to the fiphdr - again use ipspool for this extra-fiphdr-fields:#FTP_ZAP_EXTERNAL:# Otherwise if you have masses of disk space, just add a line in the nightly maintenance /fip/local/zapfiplog (or zapfiplog.cmd) to wipe out the files after a day or so. On Unix/Linux : # save 8 days of jpegs .. if [ -d /fip/data/jpegs ]; then /fip/bin/ipdelque -q /fip/data/jpegs -A 8 -i 0 fi NOTES ----- - For AIX 4.1 : If calling a script using the '-p' switch or the 'script:' parameter, you MUST make sure the first line of the script starts with a '#!'. eg for a bourne shell : #! /bin/sh - Always make sure you exit from a script with an 'exit 0' if it has finished correctly. - Note that some internal FipHdr fields are available to the program They are NOT added to the output file automatically. But you can do automatically with the keyword 'add-all-fiphdr-fields' which will also add all the HH/HD/HM fields for date and time SN - output filename EN - input filename EQ - lastpart of the input folder/queue EP - full input folder see also fiphdr-input-filename: above which is how to handle filenames with '#' WIN 2K and UNC drives --------------------- If using UNC drives, note that the Win2k 'security model' no longer allows the default service logon (LocalSystem) to access external drives. So you will need to start the 'Fip Comms Manager' service under a Logon which exists on the remote system AND has permissions to read AND WRITE to the folders you want to read and/or write. You will need to either - The easiest method is to have the same logon/password for both systems - if running a Domain server, use a single logon for both - if not create/use the same logon/password on both or - The remote system needs to know about the local system's logon and have permissions to read/write How do you know it will work - Logon the Fip/local server with the logon - Open RUN and type the full UNC path - try reading and/or writing a file - if it does not work, there is no way Fip will either ! So fix it first - When that works.. - Open CMD window and change to the drive where Fip is. - type in the complete ipspool parameter line from the SYSTEM file (remember to have NO spaces between input switch and parameter ie -i2proc not -i 2proc) - if that does not work, check your syntax and ogon again - When that works.. - Add the logon to the ControlPanel->Admin Tasks->Services->Fip Comms Manager - stop and start - Note for Fip UNC drives, always use '/' rather than '' as the latter is the escape chr (so you need '\' for every '' !) - For NT note that the path name for a script is using forward slashes NOT backwards The '-P' or param-script keyword however is not changed to forward slashes Version Control ;032t9 21oct05 added create-checksum: ;a-b 17jan06 added -b and round-robin and slightly better logging ;c 23jan06 added -8 ;d 29mar06 -q also affects locked files and possible bugette with NOT cleaning out remnants of missing files. ;e-h 05aug06 for -A subqueues : added -a for preserve-subfolders and -Q no-output-subfolders ;i 18jan07 bugette - supercede0 was missing and added param-script0-9 and logscript0-9 ;j-k 22mar07 bugette - do not use DEST_REDUN if not needed ;l 04oct07 added filesize at the end of the log message ;m-n 24oct07 added extra-fiphdr-script and save-data-filename: ;o 31oct07 check for primary server before each file in addition to once every folder scan ;p2 08nov07 added copy-file0-9 and no-fiphdr0-9 ;r1-6 29nov07 added rename in doneque ;r1 added fiphdr-original-time: ;r2 added E4 ;r3 bugette with -p /-P ;r4 for fipredunBalanced ;s1 31jul07 added save-data-balance-group/queue and -B ;3456 bugette WINNT curdrv and speedy ;7-8 bugette in wait-for-data ;9 redid -r ;10 added round-robin-fiphdr ;11 added file-type: 0-9 ;12 added tracker-script ;13 added secs to wait-for-remote-server ;14 file-type tuned ; 15 usleeps tuned ;16 allow dbl qtes around any input switch que - input, balance, error etc ;17 added chkexists-input-script ;18-20 5oct11 bugette with subques (both linux and win2k) and added filter ;20 added wildsingle ;23-26 19oct11 added check-mask-first:yes/no,subques - do NOT allow '..' ;26 and '.' ;27 stop spins on network drives which have disappeared ;28-32 1oct12 -j added parse input queue (and bugette in exec) ;33 10oct12 added min-size-action and max-size-action and wrong-size-wait ;34-36 added linux Notify ;37 added fiphdr-input-filename and fiphdr-input-file-hash ;38 added ignore-mime-header ;t1-9 11apr13 allow up to 20 subfolders (was 1 !) ;3-4 added winnt-done-drive ;8 30apr14 file-trace ;9 outq parse better ;031z1 01feb01 added wait-for-data ;a bugette with masks when using -k/-K ;b 01may01 masks are now FipSeq ;c 11may01 added outque as a parameter ;d 14jun01 added outque1-9 according to mask/umask ;e 17aug01 small problem with mask/unmask and wild at the END ;f 13sep01 added supercede2-9 ;g 02jan02 added mask1/unmask1/supercede1 etc and found the bugette where SN sometimes disappeared! ;h 03oct02 added fipseq functions - partial/repeat/style etc ;i 16oct02 bugette in FipHdr bit ;j 15nov02 added redundancy/check-primary-server ;k/l 01jan03 MACOSX ;m 23may03 added fiphdr0-9 ;n 06aug03 bug in outque/outque1 ;o 10feb04 added filename0-9 ;p 15may04 added timing-stats ;q 01jul04 bugette in incoming fipHdrs. ;r-x 07oct04 speedy (w allow drive letter for script) ;y 18jul05 added wait-for-remote-server ;z1 11oct05 added save-data-path:/fip/data/incoming/$d/ doneque is now FipSeq and not just hard paths ;030 30nov00 upto 9 masks/unmasks and scripts ;029f 01sep99 added extra-fiphdr and add-all-fiphdr-fields ;a 13sep99 NT mods for drives ;b 15sep99 added -E ;c 28oct99 added toomanyfiles check ;d 14jan00 small WINNT hdr mod ;e/f 20jan00 supercede added (copyright) 2014 and previous years FingerPost Ltd.