ipformat IPFORMAT processes its input file according to the rules given by the parameter file. The binary version of the parameter file is checked and if it needs updating, IPFORMAT will automatically run IPFORMBL beforehand. As to where the file goes, is saved etc depends on both input flags and the contents of parameter file. By default the resultant file is sent to a destination (DU header field) which is the same as the parameter file name. Hence the file is moved to spool/2go for IPWHEEL to distribute as it needs. Remember that you can add header fields to the output file using the data from the inbound file header or text or fixed data or system generated data. Input parameters are : Mandatory : -i : raw input file default: none -p : parameter table to use default: none Optional : -a : append the output to an existing default: no file or create one if not there - use this with -S or -s -A : prepend the output to an existing default: no file or create one if not there - use this with -S or -s -b : name of the breakout file default: same as output file (if -xX is specified) -c : copy of Output file - if different to -S or -s default: none -C : revert to old slower method default: new -d : output destination default: same as param name -D : do NOT send the file anywhere default: see param table -E : set the locale default: default system locale (or none!) -f : output format default: FORM -F : Fipid key for SHM default: none -g : testMode - ignore newfile default: no -h : extra FipHdr fields default: none This allows the same data format to be used with variable data eg -h "#DD:longstats" -H : do NOT add a FipHdr to the output file This overrides any 'hdr' parameters in the parameter file Default is Add - unless 'nohdr' is in the parameter file -3 : extra FipHdr fields default: none (Note this is used primarily by the DataFormats system to pass extra information so in normal use, pls do not use) -j : jobname default: same as -p -l : do NOT log default: log -o : output file if needed default: none Normally this is not required. It can be used to override the 'name:' parameter if it start with a '/' -O : output queue. Force all output to this queue. default is either contents of -o or if -s/-S specified, spool/formsave or spool/2go -q : input queue default: spool/form -R : Real filename default: fiphdr 'SN' or filename if no SN -s : save this file in the save area default: no Name will be the contents of the 'name:' keyword or the default: #SN:SN#DU:DF#SC:ascii#DF:DF -S : save this file in the save area default: no with the following name eg : -S "#SN:XK#PP:PP" -t : (filename) for debugging, list all xml tags default: no or (for text, csv) show all blocks and records - save in this file and continue processing -T : for debugging, show all xml tags default: no or (for text, csv) show all blocks and records and then stop processing -w : do NOT overwrite default: overwrite -W : use SHM for debugging default: no use this with care - it is for the W4 test environment -x : force breakout/debugging (same as FZBO header flag) default: no parameter can be 'o'ctal, 'd'ec or 'h'ex (default is o) To put the breakout in HTML for the browser, add a 'b' too. -X : list of flags to turn ON beforehand default: none -X 1,110,99,22 -y : run time defaults - which can be overridden by the parameter file -y s - do NOT clobber saves (sets default to NO) -y S - clobber saves (sets default to YES) -z : this is being run serially so delete the input file at the end default: first move it to x -Z : delete input file after use default: do NOT delete it normally ipformd will delete it after use. -v : print version no and exit -------------------------------------------------------------------- -------------------------------------------------------------------- Notes for version - 0.53t Added 'chksum' r=* savnum=3 chksum f3 Added 'numrecs' and 'numflds' for Text and 'numtags' and 'numatts' for XML before: "Number of Records are " numrecs output: r=22 "No of Fields is " numflds NL Added 'outque' Released 'lookup' lookup:name file:subjectcodes.fip key-field:3 value-field:1 sep:| lookup:(name) file:(filename - no default,; this field is mandatory) key-field: (field number of the key field - starting at 1 - default 1 for 1st) value-field: (field number of the data or value field - starting at 1 - default 2 for 2nd) default-value: (FipSeq - default - none) separator: (Fipseq chr - default is '|') type-format: (filetype - Fixed, CSV, TAB-sep - default is blank for TEXT) comment-line: (character which signifies a comment line - default is ';') numeric-key: signifies the key is numeric. sorted: yes/no case-sensitive-key: yes/no allow-spaces: Added savefiphdr=FN f1 ie add the contents of field 1 to be accessible as FipHdr FN for the duration of the program. -------------------------------------------------------------------- Notes for version - 0.53 New 'filtyp' XML and NLCSV added 'filtyp' nlCSVs - which is the same as CSV except for badly formatted CSV where it resets at the end of the line. 'filtyp' XML : names of TAGs are like Records ie t=BODY for <BODY> names of ATTRIBUTES are like Fields 'a=city' ie t=Property "city is " a=city <Property city="PocketNook"> 2 more concepts are 1. endtags - use 'e' in place of 'r' at the start of line </body> e=body (.....) ie trigger when we have an endtag 2. continuations <p>start of para <I>italic</I> continuation of para</p> ; start para by resetting font to default t=p "<f$>" t* ; start of italic.. t=I "<f:Bodoni-Italic>" t* ; end italic e=I "<f$>" ; rest of para u=p t* The default record key is nothing (or '0' for numeric keys) rather than 'x1594' - however it does not look like we did that !! -------------------------------------------------------------------- Notes for version - 0.52 New functions added : --------abbrev abbrev: (name) : width:(size) [(optional other keywords)...] abbrev:mangleStox:20 width:30 strip-chr-order:eaious,.:; always need width optionals pointsize: use-afm-file: (name) use-generic-widths: strip-chr-order: strip-type-order: if NOT specified, that type of chr is NOT stripped R - strip multiple spaces H - strip Unprintables - high control chrs C - strip Unprintables - low control chrs S - strip Spaces V - strip Lowercase vowels T - strip punctuation J - strip multiples of the same character Y - strip Uppercase vowels L/U - strip lower/upper case letters N - strip numbers default is "RHCVTJLNSYU" -------------------------------------------------------------------- Notes for version - 0.46 New functions added : ------------------- - First proper WINNT version - 'hdr', 'name', 'before', 'after' are now repeatable - so you can specify several lines of each. eg : after: ifflag x44 "Flag 44 is still onn" after: ifnnul s3 ifcon "GREED" caps s3 "n I am a " s3 "personn" after: "That's all Folks!n" - 'hdr' is now processed at the end of the file, so Save Zones, Flags, Counters, Calcs which have been altered during processing are all available. - 'filename' added Small bugs fixed : ------------------- - In many cases the TMP file in /fip/x was NOT zapped at end.. it is now. - 'after' was zapping 'newfile' and all key information. - Two zones containing '0' and '.' were both considered numbers and were therefore equal. NOT now. -------------------------------------------------------------------- Notes for version - 0.44 New functions added : ------------------- Builtin 'strlen' returns the length of the string - which can be output or saved or tested ; test the field 2 is greater than 44 chrs (ie 44 is less than strlen of f2) r=HH ifnnul f2 iflt 44 strlen f2 "Big Field 2 here over 44 chrs long" n r=KK "Save Field for Name (s55) is " strlen s55 " chrs long" Builtin 'zapleadzero' removes leading zeros from a field ; data - field 99 is 00000330303, field 101 is 00000000.00 r=3 "This outputs 330303=" zapleadzero f99 ", while this is 0.00=" zapleadzero f101 New tests 'ifcon' and ifncon' for if the first string is (not)CONTAINED in the second ; Data for f22="Nationwide First Division" r=5 ifnnul f22 ifcon "first" lwrcase f22 "Yes this is the First Division" (note as per all ifs, always test for NUL field first and usually you want to force the data to either all capitals or all lower case beforehand to catch all mistypes. -------------------------------------------------------------------- Notes for version - 0.43 New functions added : ------------------- fractions and base - create fractions or base numbers. date - mangle dates and times. firstmatch - same as match except all matches are stopped when found style for output style - leading zeros or padding with spaces New Functions in 'output' section : continue - skip remaining output lines and continue with the next data record. stop! - stop processing NOW reckey - output the exact record key : This is normally used with wildcards. r=* "This is Record Key : " reckey n Keys : ----- Keys can now be specified as FipSeq for unprintable chrs. For keys with embedded spaces use double quotes. Keys can be case sensitive - use the 'keycasesens' keyword. Maximum keysize is now 100 chrs (from 20) Keys MUST be consequetive chrs/bytes and can NOT be split into several fields. Small bugs fixed : ------------------- - before/after/name - not parsing the first cmd properly - Save areas are managed better. - Total save area for ALL saves is 32k at any one time. - New message for when the data is too big for a buffer. currently the maximum field/record size is 8k. - Matches on Save zones do NOT zap that zone - Maximum number of decimal points in a calc is now 6 Explanations & Notes : ------------------- When using ifs : If the data has lots of leading spaces like | 0| ifne f3 "0" will fail as the spaces are valid characters for the test. However the following will test correctly : ifne zapspc f3 "0" Do NOT use set names with punctuation such as : set this_one "data string" ONLY alpha or alphanumeric names should be used. When using ifs, ALWAYS test that a field/zone is NOT NUL beforehand otherwise unexpected results could occur : ifeq "ZZ" f1 will NOT test correctly if f1 is NUL ifnnul f1 ifeq "ZZ" f1 If the first record is missing an initial separator, that record could be missed completely - use 'ipxchg' to add a marker at the begining of the file Currently you can NOT 'save' or 'savnum' the output from 'fraction', 'base' or 'date' Version Control ;54y46 29nov04 bugette in buffer ;c-f 05feb06 started added keydata and id= ;g 17mar06 added -t (name) and -T stops outputting ;h-i 18apr06 added -H no FipHdr on output added date ZK ZF ZV ;j-m 02may06 new flag x0 is now OFF before/after the input file is processed and ON during -S (name) can now be FipSeq ;n 07aug06 added -c copyname ;o-p 21sep06 validate xml (;p bugette in DATE) ;q-t 04nov06 redid shuffle saves (;t -b added) ;u-v 07feb07 allowed Saves and Fields in Calcs (as well as calcs and counters) ;w-x5 17mar07 added -3 extra fiphdr (x1 TagAtt ;x2-3 datepadding and SummerTime MAY give hour wrong) ;4 added max-fiphdr-size ;6 FORMsystem ;7 bugette in recsep > 2 chrs and buffer overflow ;y27 01feb09 made Tag structure variable to cope with files > 2million tags/allow Partial fields in Calcs ;3 bugette with lookup defaults and speedy on SendIt ;4 24jul internal chgs - check endtags ;5-7 26oct malloc for internals (5-6 are bad for CSV) ;8-14 28jun10 added subroutines plus WINNT isspace ;13 check DF version to see if we need to rebuild the parameter file ;14-15 added -Q - use outque NOT -S/-s, maxLookupFiles-> 100 ;16-17 bugette with f* and missing the last e=... ; 18 for maxLookups ;19 added inctemplate and signed for calcs plus bugette in date when using $u plus able to save/savefiphdr styles, fractions, id ;20 bugette with fractions ;21 5may11 added note/trim-fiphdr ;22 14may11 trim buffer after 'log' so it does not output data too ;23 21may11 bugette when P4 but no partial specified beforehand ;24 26may11 added abbrev type DF_P_multipleCHR 'J' ;25-28 27may11 added -E and bugette with repeat block range f1-* ; 27-8 clobber-saves and bequal2 ;29 14jul11 added ignore-xml to abbrev, redid repeats ;30-33 27jul11 added script and copy-file ;34-35 redid preserve-fiphdr as buggy ;36 date/dst bugette ;37-38 16apr12 added json ;39-40 31jul12 added balance-group and balance-queue and ispunct WINNT ;41 10dec12 added warn: as lookup option for missing lookup file !w not !x ;42 25jun13 ipformbl and quotes wrong for Win2K ;43-45 8jul13 p9, c9 and z9 to variables for repeats; added -g ;46 30apr14 added file-trace ;053z 08may02 added XML as file type plus sigmods ! ;b 10dec02 added CSV and embedded CR/NL ;c 13mar03 added fiphdr-* ;d-n 04jul03 cleanups - esp to XML ;o 25feb04 added 'outque' ;p-v 31mar04 bugette - r#99 not working after XML changes plus redid numblks/recs/tags/flds/atts and outque/fname bugette plus added chksum plus added savefiphdr=FN f1 ;w-x 16jul04 make comma the default fldsep for CSV ;y-z 02oct04 small mods for keys and wildstr and speedy ;052 22oct01 added abbreviation ;051 14sep01 added SHM/Single Step stuff ;050m 17jan00 added iffile/ifnfile plus -X for turning flags ON ;a 20apr00 bug in reckey with neg posn ;b 12may00 better handling of filtyp:csv ;c 18may00 handling BIG files - BUG ;d 14jun00 better handling of filtyp:fixed added type PRINTNOSPC and -xbo for browser ;e 06aug00 buggette in match - rejigged ;f 05sep00 allow '+' and '-' in MONEY and DECIMAL types for partials etc plus bugette using 'ifprv' in 'after' plus added -h (extraFipHdr) plus bugette for reckey-at-end-of-line and not enough data on the line ;g 21sep00 woops - ifprv/ifnprv not working after a 'f3' bug !! ;h 02oct00 woops - setctr not working for partials bug !! ;i 11jan01 CSV bugette - embedded double quotes ;j 16feb01 added -a/-A for append/prepend ;k 04apr01 bugette with calculations and leading '-' signs ;m 12may01 BUG- sometimes missing last line of data if no rec sep at end (copyright) 2014 and previous years FingerPost Ltd.