ipformat
ipformat
IPFORMAT processes its input file according to the rules given by the parameter
file. The binary version of the parameter file is checked and if it needs
updating, IPFORMAT will automatically run IPFORMBL beforehand.
As to where the file goes, is saved etc depends on both input flags and the
contents of parameter file.
By default the resultant file is sent to a destination (DU header field) which
is the same as the parameter file name. Hence the file is moved to spool/2go
for IPWHEEL to distribute as it needs.
Remember that you can add header fields to the output file using the data from
the inbound file header or text or fixed data or system generated data.
Input parameters are :
Mandatory :
-i : raw input file default: none
-p : parameter table to use default: none
Optional :
-a : append the output to an existing default: no
file or create one if not there - use this with -S or -s
-A : prepend the output to an existing default: no
file or create one if not there - use this with -S or -s
-b : name of the breakout file default: same as output file
(if -xX is specified)
-c : copy of Output file - if different to -S or -s default: none
-C : revert to old slower method default: new
-d : output destination default: same as param name
-D : do NOT send the file anywhere default: see param table
-E : set the locale default: default system locale (or none!)
-f : output format default: FORM
-F : Fipid key for SHM default: none
-g : testMode-ignore newfile default: no
-G : testMode-ignore outque,name,copy-file default: no
-h : extra FipHdr fields default: none
This allows the same data format to be used with variable data
eg -h "#DD:longstats"
-H : do NOT add a FipHdr to the output file
This overrides any 'hdr' parameters in the parameter file
Default is Add - unless 'nohdr' is in the parameter file
-3 : extra FipHdr fields default: none
(Note this is used primarily by the DataFormats system to pass extra
information
so in normal use, pls do not use)
-j : jobname default: same as -p
-l : do NOT log default: log
-o : output file if needed default: none
Normally this is not required. It can be used to
override the 'name:' parameter if it start with a '/'
-O : output queue. Force all output to this queue.
default is either contents of -o
or if -s/-S specified, spool/formsave
or spool/2go
-q : input queue default: spool/form
-R : Real filename default: fiphdr 'SN' or filename if no SN
-s : save this file in the save area default: no
Name will be the contents of the 'name:' keyword
or the default: #SN:\SN#DU:\DF#SC:ascii#DF:\DF
-S : save this file in the save area default: no
with the following name
eg : -S "#SN:\XK#PP:\PP"
-t : (filename)
for debugging, list all xml tags default: no
or (for text, csv) show all blocks and records
- save in this file and continue processing
-T : for debugging, show all xml tags default: no
or (for text, csv) show all blocks and records
and then stop processing
-w : do NOT overwrite default: overwrite
-W : use SHM for debugging default: no
use this with care - it is for the W4 test environment
-x : force breakout/debugging (same as FZBO header flag) default: no
parameter can be 'o'ctal, 'd'ec or 'h'ex (default is o)
To put the breakout in HTML for the browser, add a 'b' too.
-X : list of flags to turn ON beforehand default: none
-X 1,110,99,22
-y : run time defaults - which can be overridden by the parameter file
-y s - do NOT clobber saves (sets default to NO)
-y S - clobber saves (sets default to YES)
-z : this is being run serially so delete the input file at the end
default: first move it to x
-Z : delete input file after use default: do NOT delete it
normally ipformd will delete it after use.
-v : print version no and exit
--------------------------------------------------------------------
--------------------------------------------------------------------
Note for use-sx:
USE-SX: replace/increment/merge/no
USE-EXTERNAL-FILE: replace/increment/merge/no
If and only if the file is just a FipHdr file and uses 'SX:' to point at an
external data file,
'I'ncrement- make a new SX: file with the changes; the FipHdr file is output
(the existing SX file is untouched)
'R'eplace existing SX: file; the FipHdr file is output
'M'erge the FipHdr and the Data to make a single output file (default)
NO-ignore the date in the file whose path is in SX
--------------------------------------------------------------------
Notes for version - 0.53t
Added 'chksum'
r=* savnum=3 chksum f3
Added 'numrecs' and 'numflds' for Text
and 'numtags' and 'numatts' for XML
before: "Number of Records are " numrecs
output:
r=22 "No of Fields is " numflds NL
Added 'outque'
Released 'lookup'
lookup:name file:subjectcodes.fip key-field:3 value-field:1 sep:|
lookup:(name)
file:(filename - no default,; this field is mandatory)
key-field: (field number of the key field - starting at 1 - default 1 for
1st)
value-field: (field number of the data or value field - starting at 1 -
default 2 for 2nd)
default-value: (FipSeq - default - none)
separator: (Fipseq chr - default is '|')
type-format: (filetype - Fixed, CSV, TAB-sep - default is blank for TEXT)
comment-line: (character which signifies a comment line - default is ';')
numeric-key:
signifies the key is numeric.
sorted: yes/no
case-sensitive-key: yes/no
allow-spaces:
Added savefiphdr=FN f1
ie add the contents of field 1 to be accessible as FipHdr FN for the duration
of the program.
--------------------------------------------------------------------
Notes for version - 0.53
New 'filtyp' XML and NLCSV added
'filtyp' nlCSVs - which is the same as CSV except for badly formatted CSV where
it resets at the end of the line.
'filtyp' XML :
names of TAGs are like Records
ie t=BODY for <BODY>
names of ATTRIBUTES are like Fields 'a=city'
ie t=Property "city is " a=city
<Property city="PocketNook">
2 more concepts are
1. endtags - use 'e' in place of 'r' at the start of line
</body>
e=body (.....)
ie trigger when we have an endtag
2. continuations
<p>start of para <I>italic</I> continuation of para</p>
; start para by resetting font to default
t=p "<f$>" t*
; start of italic..
t=I "<f:Bodoni-Italic>" t*
; end italic
e=I "<f$>"
; rest of para
u=p t*
The default record key is nothing (or '0' for numeric keys) rather than 'x1594'
- however it does not look like we did that !!
--------------------------------------------------------------------
Notes for version - 0.52
New functions added :
--------abbrev
abbrev: (name) : width:(size) [(optional other keywords)...]
abbrev:mangleStox:20 width:30 strip-chr-order:eaiou\s,.:;
always need width
optionals pointsize:
use-afm-file: (name)
use-generic-widths:
strip-chr-order:
strip-type-order:
if NOT specified, that type of chr is NOT stripped
R - strip multiple spaces
H - strip Unprintables - high control chrs
C - strip Unprintables - low control chrs
S - strip Spaces
V - strip Lowercase vowels
T - strip punctuation
J - strip multiples of the same character
Y - strip Uppercase vowels
L/U - strip lower/upper case letters
N - strip numbers
default is "RHCVTJLNSYU"
--------------------------------------------------------------------
Notes for version - 0.46
New functions added :
-------------------
- First proper WINNT version
- 'hdr', 'name', 'before', 'after' are now repeatable - so you can specify
several lines of each.
eg :
after: ifflag x44 "Flag 44 is still on\n"
after: ifnnul s3 ifcon "GREED" caps s3 "\n I am a " s3 "person\n"
after: "That's all Folks!\n"
- 'hdr' is now processed at the end of the file, so Save Zones, Flags,
Counters, Calcs which have
been altered during processing are all available.
- 'filename' added
Small bugs fixed :
-------------------
- In many cases the TMP file in /fip/x was NOT zapped at end.. it is now.
- 'after' was zapping 'newfile' and all key information.
- Two zones containing '0' and '.' were both considered numbers and were
therefore equal. NOT now.
--------------------------------------------------------------------
Notes for version - 0.44
New functions added :
-------------------
Builtin 'strlen' returns the length of the string - which can be output or
saved or tested
; test the field 2 is greater than 44 chrs (ie 44 is less than strlen of f2)
r=HH ifnnul f2 iflt 44 strlen f2 "Big Field 2 here over 44 chrs long" \n
r=KK "Save Field for Name (s55) is " strlen s55 " chrs long"
Builtin 'zapleadzero' removes leading zeros from a field
; data - field 99 is 00000330303, field 101 is 00000000.00
r=3 "This outputs 330303=" zapleadzero f99 ", while this is 0.00="
zapleadzero f101
New tests 'ifcon' and ifncon' for if the first string is (not)CONTAINED in the
second
; Data for f22="Nationwide First Division"
r=5 ifnnul f22 ifcon "first" lwrcase f22 "Yes this is the First Division"
(note as per all ifs, always test for NUL field first and usually you want to
force the data to either all capitals or all lower case beforehand to catch all
mistypes.
--------------------------------------------------------------------
Notes for version - 0.43
New functions added :
-------------------
fractions and base - create fractions or base numbers.
date - mangle dates and times.
firstmatch - same as match except all matches are stopped when found
style for output style - leading zeros or padding with spaces
New Functions in 'output' section :
continue - skip remaining output lines and continue
with the next data record.
stop! - stop processing NOW
reckey - output the exact record key :
This is normally used with wildcards.
r=* "This is Record Key : " reckey \n
Keys :
-----
Keys can now be specified as FipSeq for unprintable chrs.
For keys with embedded spaces use double quotes.
Keys can be case sensitive - use the 'keycasesens' keyword.
Maximum keysize is now 100 chrs (from 20)
Keys MUST be consequetive chrs/bytes and can NOT be split into several fields.
Small bugs fixed :
-------------------
- before/after/name - not parsing the first cmd properly
- Save areas are managed better.
- Total save area for ALL saves is 32k at any one time.
- New message for when the data is too big for a buffer.
currently the maximum field/record size is 8k.
- Matches on Save zones do NOT zap that zone
- Maximum number of decimal points in a calc is now 6
Explanations & Notes :
-------------------
When using ifs :
If the data has lots of leading spaces like | 0|
ifne f3 "0"
will fail as the spaces are valid characters for the test.
However the following will test correctly :
ifne zapspc f3 "0"
Do NOT use set names with punctuation such as :
set this_one "data string"
ONLY alpha or alphanumeric names should be used.
When using ifs, ALWAYS test that a field/zone is NOT NUL beforehand
otherwise unexpected results could occur :
ifeq "ZZ" f1
will NOT test correctly if f1 is NUL
ifnnul f1 ifeq "ZZ" f1
If the first record is missing an initial separator, that record could be
missed completely
- use 'ipxchg' to add a marker at the begining of the file
Currently you can NOT 'save' or 'savnum' the output from 'fraction', 'base' or
'date'
Version Control
;54y64 29nov04 bugette in buffer
;c-f 05feb06 started added keydata and id=
;g 17mar06 added -t (name) and -T stops outputting
;h-i 18apr06 added -H no FipHdr on output added date ZK ZF ZV
;j-m 02may06 new flag x0 is now OFF before/after the input file is processed
and ON during
-S (name) can now be FipSeq
;n 07aug06 added -c copyname
;o-p 21sep06 validate xml (;p bugette in DATE)
;q-t 04nov06 redid shuffle saves (;t -b added)
;u-v 07feb07 allowed Saves and Fields in Calcs (as well as calcs and counters)
;w-x5 17mar07 added -3 extra fiphdr (x1 TagAtt ;x2-3 datepadding and
SummerTime MAY give hour wrong)
;4 added max-fiphdr-size ;6 FORMsystem ;7 bugette in recsep > 2 chrs and
buffer overflow
;y27 01feb09 made Tag structure variable to cope with files > 2million
tags/allow Partial fields in Calcs
;3 bugette with lookup defaults and speedy on SendIt ;4 24jul internal chgs -
check endtags
;5-7 26oct malloc for internals (5-6 are bad for CSV)
;8-14 28jun10 added subroutines plus WINNT isspace
;13 check DF version to see if we need to rebuild the parameter file
;14-15 added -Q - use outque NOT -S/-s, maxLookupFiles-> 100
;16-17 bugette with f* and missing the last e=... ; 18 for maxLookups
;19 added inctemplate and signed for calcs plus bugette in date when using
\$u
plus able to save/savefiphdr styles, fractions, id
;20 bugette with fractions
;21 5may11 added note/trim-fiphdr
;22 14may11 trim buffer after 'log' so it does not output data too
;23 21may11 bugette when P4 but no partial specified beforehand
;24 26may11 added abbrev type DF_P_multipleCHR 'J'
;25-28 27may11 added -E and bugette with repeat block range f1-* ; 27-8
clobber-saves and bequal2
;29 14jul11 added ignore-xml to abbrev, redid repeats
;30-33 27jul11 added script and copy-file ;34-35 redid preserve-fiphdr as
buggy ;36 date/dst bugette
;37-38 16apr12 added json
;39-40 31jul12 added balance-group and balance-queue and ispunct WINNT
;41 10dec12 added warn: as lookup option for missing lookup file !w not !x
;42 25jun13 ipformbl and quotes wrong for Win2K
;43-45 8jul13 p9, c9 and z9 to variables for repeats; added -g
;46 30apr14 added file-trace ;47 added json-level and json-array-level and
bugette with ifexact t=* "Total"
;48 6feb15 bugette when save was mapped back to itself but was now
zero-length
;49 22aug15 better balance ;50 10apr16 chkque before copyfile ;51 redid y48
;52-54 link not copy
;55 5may17 added use-sx ;56 bugette in copyfile ;57 11dec17 added locale as
a parameter
;58 4jan18 added -G to really wipe things for TESTMODE
;59 8jan18 bugette for v large files which need to output as one string !
;60-61 3feb18 added finish: block and 'delete!!' to stop and zap the output
file
;62-63 16may18 added fipseqs
;64 29mar23 bugette where the input data file is smaller than any Include
file
;053z 08may02 added XML as file type plus sigmods !
;b 10dec02 added CSV and embedded CR/NL
;c 13mar03 added fiphdr-*
;d-n 04jul03 cleanups - esp to XML
;o 25feb04 added 'outque'
;p-v 31mar04 bugette - r#99 not working after XML changes
plus redid numblks/recs/tags/flds/atts and outque/fname bugette
plus added chksum
plus added savefiphdr=FN f1
;w-x 16jul04 make comma the default fldsep for CSV
;y-z 02oct04 small mods for keys and wildstr and speedy
;052 22oct01 added abbreviation
;051 14sep01 added SHM/Single Step stuff
;050m 17jan00 added iffile/ifnfile plus -X for turning flags ON
;a 20apr00 bug in reckey with neg posn
;b 12may00 better handling of filtyp:csv
;c 18may00 handling BIG files - BUG
;d 14jun00 better handling of filtyp:fixed
added type PRINTNOSPC and -xbo for browser
;e 06aug00 buggette in match - rejigged
;f 05sep00 allow '+' and '-' in MONEY and DECIMAL types for partials etc
plus bugette using 'ifprv' in 'after'
plus added -h (extraFipHdr)
plus bugette for reckey-at-end-of-line and not enough data on the line
;g 21sep00 woops - ifprv/ifnprv not working after a 'f3' bug !!
;h 02oct00 woops - setctr not working for partials bug !!
;i 11jan01 CSV bugette - embedded double quotes
;j 16feb01 added -a/-A for append/prepend
;k 04apr01 bugette with calculations and leading '-' signs
;m 12may01 BUG- sometimes missing last line of data if no rec sep at end
(copyright) 2024 and previous years FingerPost Ltd.