ipformat (Sat Oct 25 2014 01:31:01)


IPFORMAT processes its input file according to the rules given by the parameter
file. The binary version of the parameter file is checked and if it needs
updating, IPFORMAT will automatically run IPFORMBL beforehand.

As to where the file goes, is saved etc depends on both input flags and the
contents of parameter file.

By default the resultant file is sent to a destination (DU header field) which
is the same as the parameter file name. Hence the file is moved to spool/2go
for IPWHEEL to distribute as it needs.

Remember that you can add header fields to the output file using the data from
the inbound file header or text or fixed data or system generated data.

Input parameters are :
	Mandatory :
	-i : raw input file			default: none
	-p : parameter table to use		default: none
	Optional :
	-a : append the output to an existing	default: no
		file or create one if not there - use this with -S or -s
	-A : prepend the output to an existing	default: no
		file or create one if not there - use this with -S or -s
	-b : name of the breakout file		default: same as output file
		(if -xX is specified)
	-c : copy of Output file - if different to -S or -s default: none
	-C : revert to old slower method	default: new
	-d : output destination 		default: same as param name
	-D : do NOT send the file anywhere	default: see param table
	-E : set the locale			default: default system locale (or none!)
	-f : output format			default: FORM
	-F : Fipid key for SHM			default: none
	-g : testMode -  ignore newfile		default: no
	-h : extra FipHdr fields		default: none
		This allows the same data format to be used with variable data
		eg -h "#DD:longstats"
	-H : do NOT add a FipHdr to the output file
		This overrides any 'hdr' parameters in the parameter file
		Default is Add - unless 'nohdr' is in the parameter file
	-3 : extra FipHdr fields		default: none
		(Note this is used primarily by the DataFormats system to pass extra
		so in normal use, pls do not use)
	-j : jobname				default: same as -p
	-l : do NOT log				default: log
	-o : output file if needed		default: none
		Normally this is not required. It can be used to
		override the 'name:' parameter if it start with a '/'
	-O : output queue. Force all output to this queue.
		default is either contents of -o
			or if -s/-S specified, spool/formsave
			or spool/2go
	-q : input queue			default: spool/form
	-R : Real filename			default: fiphdr 'SN' or filename if no SN
	-s : save this file in the save area	default: no
		Name will be the contents of the 'name:' keyword
		or the default: #SN:SN#DU:DF#SC:ascii#DF:DF
	-S : save this file in the save area	default: no
		with the following name
		eg : -S "#SN:XK#PP:PP"
	-t : (filename)
		for debugging, list all xml tags	default: no
		or (for text, csv) show all blocks and records
		- save in this file and continue processing
	-T : for debugging, show all xml tags	default: no
		or (for text, csv) show all blocks and records
		and then stop processing
	-w : do NOT overwrite			default: overwrite
	-W : use SHM for debugging		default: no
		use this with care - it is for the W4 test environment
	-x : force breakout/debugging (same as FZBO header flag) default: no
		parameter can be 'o'ctal, 'd'ec or 'h'ex (default is o)
		To put the breakout in HTML for the browser, add a 'b' too.
	-X : list of flags to turn ON beforehand	default: none
		-X 1,110,99,22
	-y : run time defaults - which can be overridden by the parameter file
		-y s - do NOT clobber saves	(sets default to NO)
		-y S - clobber saves		(sets default to YES)
	-z : this is being run serially so delete the input file at the end
						default: first move it to x
	-Z : delete input file after use	default: do NOT delete it
		normally ipformd will delete it after use.
	-v : print version no and exit

Notes for version - 0.53t

Added	'chksum'
	r=* savnum=3 chksum f3

Added	'numrecs' and 'numflds' for Text
and	'numtags' and 'numatts' for XML
	before:	"Number of Records are " numrecs


	r=22	"No of Fields is " numflds NL

Added	'outque'

Released 'lookup'
	lookup:name	file:subjectcodes.fip key-field:3 value-field:1 sep:|

		file:(filename - no default,; this field is mandatory)
		key-field: (field number of the key field - starting at 1 - default 1 for
		value-field: (field number of the data or value field - starting at 1 -
default 2 for 2nd)
		default-value: (FipSeq - default - none)
		separator: (Fipseq chr - default is '|')
		type-format: (filetype - Fixed, CSV, TAB-sep - default is blank for TEXT)
		comment-line: (character which signifies a comment line - default is ';')
			signifies the key is numeric.
		sorted: yes/no
		case-sensitive-key: yes/no

Added savefiphdr=FN f1
	ie add the contents of field 1 to be accessible as FipHdr FN for the duration
of the program.

Notes for version - 0.53

New 'filtyp' XML and NLCSV added

'filtyp' nlCSVs - which is the same as CSV except for badly formatted CSV where
it resets at the end of the line.

'filtyp' XML :
	names of TAGs are like Records
		ie	t=BODY for <BODY>
	names of ATTRIBUTES are like Fields 'a=city'
		ie	t=Property "city is " a=city
		<Property city="PocketNook">

	2 more concepts are
		1. endtags - use 'e' in place of 'r' at the start of line
			e=body (.....)
			ie trigger when we have an endtag

		2. continuations
			<p>start of para <I>italic</I> continuation of para</p>
			; start para by resetting font to default
			t=p	"<f$>" t*
			; start of italic..
			t=I	"<f:Bodoni-Italic>" t*
			; end italic
			e=I	"<f$>"
			; rest of para
			u=p	t*

The default record key is nothing (or '0' for numeric keys) rather than 'x1594'
- however it does not look like we did that !!

Notes for version - 0.52

New functions added :

	abbrev: (name) : width:(size) [(optional other keywords)...]
	abbrev:mangleStox:20	width:30	strip-chr-order:eaious,.:;
always need	width
optionals	pointsize:
		use-afm-file: (name)
			if NOT specified, that type of chr is NOT stripped
		R - strip multiple spaces
		H - strip Unprintables - high control chrs
		C - strip Unprintables - low control chrs
		S - strip Spaces
		V - strip Lowercase vowels
		T - strip punctuation
		J - strip multiples of the same character
		Y - strip Uppercase vowels
		L/U - strip lower/upper case letters
		N - strip numbers
		default is "RHCVTJLNSYU"

Notes for version - 0.46

New functions added :
	- First proper WINNT version

	- 'hdr', 'name', 'before', 'after' are now repeatable - so you can specify
several lines of each.
	  eg :
		after:	ifflag x44	"Flag 44 is still onn"
		after:	ifnnul s3	ifcon "GREED" caps s3		"n  I am a " s3 "personn"
		after:	"That's all Folks!n"

	- 'hdr' is now processed at the end of the file, so Save Zones, Flags,
Counters, Calcs which have
		been altered during processing are all available.

	- 'filename' added

Small bugs fixed :
	- In many cases the TMP file in /fip/x was NOT zapped at end.. it is now.
	- 'after' was zapping 'newfile' and all key information.
	- Two zones containing '0' and '.' were both considered numbers and were
therefore equal. NOT now.

Notes for version - 0.44

New functions added :
	Builtin 'strlen' returns the length of the string - which can be output or
saved or tested
		; test the field 2 is greater than 44 chrs (ie 44 is less than strlen of f2)
		r=HH	ifnnul f2	iflt 44	strlen f2	"Big Field 2 here over 44 chrs long" n
		r=KK	"Save Field for Name (s55) is " strlen s55 " chrs long"
	Builtin 'zapleadzero' removes leading zeros from a field
		; data - field 99 is 00000330303, field 101 is 00000000.00
		r=3	"This outputs 330303=" zapleadzero f99 ", while this is 0.00="
zapleadzero f101
	New tests	'ifcon' and ifncon' for if the first string is (not)CONTAINED in the
		; Data for f22="Nationwide First Division"
		r=5	ifnnul f22 ifcon "first" lwrcase f22	"Yes this is the First Division"
		(note as per all ifs, always test for NUL field first and usually you want to
force the data to either all capitals or all lower case beforehand to catch all

Notes for version - 0.43

New functions added :
	fractions and base - create fractions or base numbers.
	date - mangle dates and times.
	firstmatch - same as match except all matches are stopped when found
	style for output style - leading zeros or padding with spaces
New Functions in 'output' section :
	continue	- skip remaining output lines and continue
			with the next data record.
	stop!		- stop processing NOW
	reckey		- output the exact record key :
			This is normally used with wildcards.
		r=*	"This is Record Key : " reckey n
Keys :
	Keys can now be specified as FipSeq for unprintable chrs.
		For keys with embedded spaces use double quotes.
	Keys can be case sensitive - use the 'keycasesens' keyword.
	Maximum keysize is now 100 chrs (from 20)
	Keys MUST be consequetive chrs/bytes and can NOT be split into several fields.
Small bugs fixed :
	- before/after/name - not parsing the first cmd properly
	- Save areas are managed better.
	- Total save area for ALL saves is 32k at any one time.
	- New message for when the data is too big for a buffer.
		currently the maximum field/record size is 8k.
	- Matches on Save zones do NOT zap that zone
	- Maximum number of decimal points in a calc is now 6
Explanations & Notes :
	When using ifs :
		If the data has lots of leading spaces like |	0|
			ifne f3 "0"
		will fail as the spaces are valid characters for the test.
		However the following will test correctly  :
			ifne zapspc f3 "0"

	Do NOT use set names with punctuation such as :
		set	this_one	"data string"
		ONLY alpha or alphanumeric names should be used.

	When using ifs, ALWAYS test that a field/zone is NOT NUL beforehand
		otherwise unexpected results could occur :
			ifeq "ZZ" f1
		will NOT test correctly if f1 is NUL
			ifnnul f1 ifeq "ZZ" f1

	If the first record is missing an initial separator, that record could be
missed completely
		- use 'ipxchg' to add a marker at the begining of the file

	Currently you can NOT 'save' or 'savnum' the output from 'fraction', 'base' or

Version Control
;54y46	29nov04 bugette in buffer
	;c-f 05feb06 started added keydata and id=
	;g 17mar06 added -t (name) and -T stops outputting
	;h-i 18apr06 added -H no FipHdr on output added date ZK ZF ZV
	;j-m 02may06 new flag x0 is now OFF before/after the input file is processed
and ON during
		-S (name) can now be FipSeq
	;n 07aug06 added -c copyname
	;o-p 21sep06 validate xml (;p bugette in DATE)
	;q-t 04nov06 redid shuffle saves (;t -b added)
	;u-v 07feb07 allowed Saves and Fields in Calcs (as well as calcs and counters)
	;w-x5 17mar07 added -3 extra fiphdr (x1 TagAtt ;x2-3 datepadding and
SummerTime MAY give hour wrong)
		;4 added max-fiphdr-size ;6 FORMsystem ;7 bugette in recsep > 2 chrs and
buffer overflow
	;y27 01feb09 made Tag structure variable to cope with files > 2million
tags/allow Partial fields in Calcs
		;3 bugette with lookup defaults and speedy on SendIt ;4 24jul internal chgs -
check endtags
		;5-7 26oct malloc for internals (5-6 are bad for CSV)
		;8-14 28jun10 added subroutines plus WINNT isspace
		;13 check DF version to see if we need to rebuild the parameter file
		;14-15 added -Q - use outque NOT -S/-s, maxLookupFiles-> 100
		;16-17 bugette with f* and missing the last e=... ; 18 for maxLookups
		;19 added inctemplate and signed for calcs plus bugette in date when using
			plus able to save/savefiphdr styles, fractions, id
		;20 bugette with fractions
		;21  5may11 added note/trim-fiphdr
		;22 14may11 trim buffer after 'log' so it does not output data too
		;23 21may11 bugette when P4 but no partial specified beforehand
		;24 26may11 added abbrev type DF_P_multipleCHR 'J'
		;25-28 27may11 added -E and bugette with repeat block range f1-* ; 27-8
clobber-saves and bequal2
		;29 14jul11 added ignore-xml to abbrev, redid repeats
		;30-33 27jul11 added script and copy-file ;34-35 redid preserve-fiphdr as
buggy ;36 date/dst bugette
		;37-38 16apr12 added json
		;39-40 31jul12 added balance-group and balance-queue and ispunct WINNT
		;41 10dec12 added warn: as lookup option for missing lookup file !w not !x
		;42 25jun13 ipformbl and quotes wrong for Win2K
		;43-45  8jul13 p9, c9 and z9 to variables for repeats; added -g
		;46 30apr14 added file-trace
;053z	08may02 added XML as file type plus sigmods !
	;b 10dec02 added CSV and embedded CR/NL
	;c 13mar03 added fiphdr-*
	;d-n 04jul03 cleanups - esp to XML
	;o 25feb04 added 'outque'
	;p-v 31mar04 bugette - r#99 not working after XML changes
		plus redid numblks/recs/tags/flds/atts and outque/fname bugette
		plus added chksum
		plus added savefiphdr=FN f1
	;w-x 16jul04 make comma the default fldsep for CSV
	;y-z 02oct04 small mods for keys and wildstr and speedy
;052	22oct01 added abbreviation
;051	14sep01 added SHM/Single Step stuff
;050m	17jan00	added iffile/ifnfile plus -X for turning flags ON
	;a 20apr00 bug in reckey with neg posn
	;b 12may00 better handling of filtyp:csv
	;c 18may00 handling BIG files - BUG
	;d 14jun00 better handling of filtyp:fixed
		added type PRINTNOSPC and -xbo for browser
	;e 06aug00 buggette in match - rejigged
	;f 05sep00 allow '+' and '-' in MONEY and DECIMAL types for partials etc
			plus bugette using 'ifprv' in 'after'
			plus added -h (extraFipHdr)
			plus bugette for reckey-at-end-of-line and not enough data on the line
	;g 21sep00 woops - ifprv/ifnprv not working after a 'f3' bug !!
	;h 02oct00 woops - setctr not working for partials bug !!
	;i 11jan01 CSV bugette - embedded double quotes
	;j 16feb01 added -a/-A for append/prepend
	;k 04apr01 bugette with calculations and leading '-' signs
	;m 12may01 BUG- sometimes missing last line of data if no rec sep at end

(copyright) 2014 and previous years FingerPost Ltd.