The Guide to the FIP Data Formatting Module

This guide covers the following topics :

1. How to go about setting up formatting a new feed.

2. Using ‘form’, the df User Interface, to test and tune.

3. Copy Flow.

4. Reference Guide to the Data Format Text parameter file.

5. FipSeq.

6. PostScript Driver – ipsetter.

7. Program Switches and other details.

7.1 form

7.2 ipformd

7.3 ipformat

7.4 ipformbl

7.5 ipformch

7.6 ipfsel

7.7 ipfprep

7.8 ipfchk

1. HOW TO FORMAT !

Time for Some Golden Rules :
1: Study the INPUT file in detail.

Try to get several input files as you need to know EXACTLY what variations there can be in the input.

Try to get a specification for the input if it exists, as that can save hours trying to guess.

2: Study the OUTPUT file in detail.

Again a spec is useful.

3: Talk to people about why they did it that way or why they want
it that way.

If you are trying to copy an existing format, check with the Users what they REALLY need. Often things were done in a strange way because of system shortcomings or user’s personalities. Get a different person in charge and they may have different emphases. You may find that there is a subsquent system written on the system you are delivering to that is making a goodmany changes that you could include in your file processing. Similarly you may find that a lot of manual work is done on the file which you could circumvent.

4: Remember to eat Lunch.

No system is worth missing food and drink!

Now the Steps :

1. Sort out Copy Flow for both testing and live running.

Do you need an ‘xchg’ before formatting ?
It is always a good idea to rationalise the input file as far as possible
– running a preliminary xchg may help.

Do you need an ‘xchg’ after formatting ? Again there may be some tidying up needed once the file is processed that is better suited to ipxchg than to ipformat.

What about special processing like needing to ‘sort’ output ? Do you need to sort the copy inbound ? Do you need to re-sort it outbound ? Do you need to select specific records ?

Does it need mutiple datafomats – occasionally you get a file that can be sorted one way, then formatted to say pick up items between consecutive records, then sorted another way and reformatted for output.

2. Map out what you want to format.

It is usually easier to make notes BEFORE starting to test.
Just a few notes is all that are required.

3. Setup and start testing the main Format. 

Often is easier to copy an existing Text file from form/text, use it as a template and rework it to meet the new requirements.

There are two ways of testing copy ‘offline’. At the Unix command line you can use the ‘form’ program to set yourself up a testing environment. Alternatively you can use the form module in W4.

4. Work out the other parameters in the Copy Flow.

See the relevant section in this guide for an example of Copy Flow.

5. Release

Dual running over a number of days is always advised.

6. Check
It is always worth going back to the users and seeing how things can be improved for them. There may be things you have missed, or just things that they now realise you can do, that they have never thought to ask for before.

Pointers

– If you need to sort the output file : use the ‘job’ parameter in the text/PROCESS file (see sections Copy Flow and ‘ipformd’ in the Program Section).

– if you need to select only certain records from the input file matched against a standing list of keys, use ‘ipfsel’ (see description in the Program Section).

– if you need to track several input files have arrived over a period of time, flag if they are missing and process if they are there (for instance, a number of postscript files for an OPI), use ‘ipformch’, (see description in the Program Section).

– if the input file(s) are very dirty and inconsistent, clean them up beforehand with ‘ipfprep’ (see description in the Program Section).

2a. USING ‘FORM’ TO TEST AND TUNE

There are two interfaces to test & tune your dataformats … form is a fast command and basic line driven interface that runs under Unix. There is also a web based interface available through W4.

FORM, the user interface for the Data Formatting module, can be used to test and tune a process ‘internally’ – without actually sending the output file anywhere nor deleting the input.

It can also be used to run an XCHG before and/or after the format.

Several tests can be run and their settings are held in a series of Settings files in tables/form/test.

Note at the moment ‘form’ will NOT scan the PROCESS file and run any ‘jobs’ (qv) as it is used purely for getting the parameter file in form/text 100% correct.

Firstly you choose an existing Settings file – if there is one.

Then you need to get the input file into the queue in spool you are using for testing.

If it is the first time you will need to ‘mkdir’ your test area. Generally one has been setup called ‘spool/test’.

If the input file is on another system, just copy it over. If the file came from a wire service or dialup and is in an FIP Archive log, you can add a destination in the sys/USERS file and resend to it. Generally, if FIippies have been onsite, there is a default one already setup called ‘test’ which goes through ‘ipedsys’ to cleanup the filename BUT does not go through any ‘xchg’ :

	sys/USERS : test=   DP:com2  DQ:2edsys  EQ:test SC:no DC:no DF:testing

This uses edsys/TESTING to place the file in spool/test.

To get into the test bit of ‘form’, type ‘x’ at the form prompt. This will reveal firstly which Settings files are available which you can choose from.

The current settings are displayed before the list of options available.

All options are a single character (case-insensitive) followed by ‘enter’.

When changing settings, a ‘*’ can be used to list all the files in the relevant directory. Case is only sensitive for filenames.

root @ com_server2/ > form
form-com_server2:x
** Test FORM - Looking for Settings files
-- List of Files in the Queue : /fip/tables/form/test
total 2
-rw-rw-rw-  1 root	   93 Mar 17 10:53 RESRAC
-rw-rw-rw-  1 root	   94 Mar 17 19:52 RESTEST
** Hit Return to continue .. 

** Test FORM - Choose a Settings files (or return to ignore) :restest
** Test FORM - Existing settings are :
	Working Queue		: /fip/spool/test
	Input file		: rem737
	TEXT Param file		: RESTEST
	XCHG before		: PA2FORM
	XCHG after	 	: PA2ATEX
	DU for Live Tests	: sunres

** Options are :
	Change Settings		: C
	List the Working Queue	: L
	Look at the Input File	: I
	 ..  .. the Output File	: O
	 ..  .. the BreakOut	: T
	Edit TEXT file		: E
	 ..  Xchg before file	: B
	 ..  Xchg after file	: A
	 ..  PROCESS for jobs	: P
	Help			: H
	Run a Test		: R
	Send Live Test		: S
	Quit			: Q :

In (slightly) more detail, the options are :

Change Settings – change any of the 4 inputs

Listing the working queue – ‘ls -l’ of the working queue

Look at the Input file – More, Dump, Edit, Tail the input which MUST BE in the working queue

Look at the Output file – More, Dump, Edit, Tail the output which is hidden in the temp queue.

Look at the BreakOut file – More, Dump, Edit, Tail the breakout file which is that created by IPFORMAT showing how the input is split into fields and records

Edit the TEXT parameter file – IPVI the parameter file in form/text

Edit the Xchg before file – IPVI the (optional) xchg file to be used BEFORE the format.

Edit the Xchg after file – IPVI the (optional) xchg file to be used AFTER the format.

Help – More this file !

Run a Test – run IPFORMAT and the look at the output file.

Send Live Test – copy the test file and send to the DU (destination) specified.

Quit – Obvious really !!

In more more detail …..

-- Change Settings :
** Change Settings - Existing settings are :
	Working Queue   	: /fip/spool/test
	Input file      	: rem727
	TEXT Param file 	: RESTEST
	XCHG before     	: PA2FORM
	XCHG after      	: PA2ATEX
	DU for Live Tests       : sunres

  (At each prompt, Use '*' for list of files)
	Change Working Queue    : W
	Change Input file       : I
	Change TEXT Param file  : T
	Change XCHG before      : B
	Change XCHG after       : A
	Change DU for Live Tests: D
	List the Working Queue  : L
	Quit			: Q or enter :

Note that a ‘*’ ‘enter’ at any of the Change file lines will list the relevant queue before reprompting.

Type ‘-none-‘ (or in fact just ‘-‘) to set an optional field to ‘-none-‘.

Note that any reference to ‘jobs’ should be ignored in this version – I do.

Looking at any of the files :

** Look at file : rem727 : Options are :
	More    : M or enter	- This does a 'cat -v' before to show ControlChr
	Dump    : D		- Essentially an 'od -ab'
	Edit    : E		- Using 'vi'
	Tail    : T		- Last 20 lines of a 'cat -v'
	Quit    : Q :

Running a Test :

– A simple format with no xchgs –

** Running please wait ... 
	/fip/bin/ipformat -i /fip/spool/hoswrk/rem727 -p ATEXHORSES -o /fip/xFORM.XX.DEFAULT.XX -s -D -l -xo 

** Ok Done    Hit Return to continue ..

This will now go directly into the Looking at Output file menu.

– A more complicated run with an xchg before and after –

** Running please wait ... 
	Saving the input in /fip/x/FORM_rem727
	ipxchg -1 /fip/x/FORM_rem727 -D PA2FORM -o /fip/x -F 
	** Ok : Xchg Before finished 
	/fip/bin/ipformat -i /fip/x/FORM_rem727 -p RESTEST -o /fip/x/FORM.XX.DEFAULT.XX -s -D -l -xo 
	** Ok : Format finished 
	ipxchg -1 /fip/x/FORM.XX.DEFAULT.XX -D PA2ATEX -o /fip/x -F 
	** Ok : Xchg After finished 
  Hit Return to continue ..

Again this will now go directly into the Looking at Output file menu.

Note that ‘form’ allows you to save the setting you have chosen in a Settings file, so that the next time you go into form it should display the settings from the last time.

Sending a Live Test :

This will copy the input test file and send it to the Destination (DU) called the ‘Live DU’ above.

It then tails the Item Log of that system underthe assumption that ‘ipformat’ will give a message when the file is through. – Cntrl C to Stop and return to the main Form Test prompt.

Prerequisties for sending a Live test (if that is possible) :

– In form, you need to specify :

• – an input file
• – a working queue
• – a destination (DU) which MUST be in the USERS file.
• – Also remember to check your SC/DCs for your xchg’s.- Also remember to add the selection lines in tables/form/PROCESS.

2a. USING THE WEB BASED VERSION OF ‘FORM’ TO TEST AND TUNE

The web interface also allows you to run offline tests of your dataformat. It is similar to form, but gives you slightly nicer views of the file (for instance hidden characters, high characters and line endings are all high-lit
in red.)

i) You will need to have the form option enabled in your W4 logon

This is normally done by adding the line

 ;;;;; Data Formatting
options:Data Formatting:/fip-pages/form/dftest.html:_blank

to either your W4 logon, or, more usually, to a template called by your W4 logon.

ii) This should be a fairly intuitive interface (feedback welcomed though)

You first select or create a test ….. a test is a definition of how you are planning to test a file, so it will define the input file, any xchgs you plan to run against it, and the format you plan to run.

The top section of the left hand pane will describe the parameters you choose, and the links in the bottom selection can be used to select a sequence of xchgs, sorts and formats.Having selected the parameter files to run<

3. COPY FLOW

How do you route copy into, and out of, the data formatting module ?

– Using the normal FIP routings !

An simple example is the Horse Racing Cards which arrive via dialup modem from the course administrators :

; Selection criteria for Horses DU=horses >atexhorsesie if the DU field is exactly equal to ‘horses’
do the ‘atexhorses’ job.

1       z="*HORSE RACING*"      +horses

horses= DP:localhost DQ:form SC:NO DC:NO

atexhorses= DP:atex1 DQ:junk-wir SC:HORSES DC:ATEX DF:DATAFORM

EXAMPLE 2 : STOCKS : More complicated examples are for the various Stocks.

The large Hong Kong tables follow the same path as ‘horses’ except that the names of the parameter files are obviously different.

Selection in PROCESS is on Filename onlyFormat text file

form/text/HKSTOCKS

output xchg xchg/HKSTOCKS

For the Regional Stocks, there are two different input formats (plus Manilla which is different but simpler) which need to be used to create almost the same output.

The Input variations are :

First type :RICDISPLAY NAMELASTTODAY’S HIGHTODAY’S LOW

HISTORIC CLOSE

AAH.ASABN-AMRO HLDGS59.96059.3

59.5

ACHN.ASACF HOLDING35.735.735.5

35.5

AEGN.ASAEGON NV112.411312.2112.8
Second Type :*FASTCLOSERIC9503011900

KLS

SECURITYDATEHIGHLOWLAST TRADE

PREVIOUS CLOSE

AYER HITAM TIN STK9503014.1003.9604.100

3.960

AYER HITAM PLANT STK95030114.30014.30014.300

13.500

ACIDCHEM STK9503016.3506.1006.3506.000
Manilla Stocks :

NAMECLOSEHIGHLOW

PRECLOSE

A Soriano3.23.23.15

3.2

A Soriano B3.23.23.23.15

There are two outputs which are identical EXCEPT London and New York prices are quoted in fractions NOT decimals and so they require a different Character Xchg.

The processing is more complicated than for ‘atexhorses’ as some of the names of the Stocks are changed AND the output is sorted alphabetically. For example ‘INTL BUS MACHINE’ needs to be ‘IBM’, but as ‘IBM’ it will start the ‘I’s and NOT appear after ‘Inland Steel’ as in the input feed.

So we use the ‘job:’ keyword in the PROCESS file to get IPFORMD to run a series of jobs rather than just start IPFORMAT as in the example above.

Look at the processing for London and New York :

While the selection remains similar to ‘atexhorses’ :

SN=LON.TXT	>fracstocks	; London

we select on the filename this time.

But At the top of the PROCESS file there are a series of parameters for the job ‘fracstocks’ :

;
; Job sequence for London and New York - Fractions
job:fracstocks  /bin/rm -f formsave/FRACSTOCKS*
job:fracstocks  /fip/bin/ipformat -p fracstocks -i $i -D -S FRACSTOCKS
job:fracstocks  /fip/bin/ipxchg -1 formsave/FRACSTOCKS -D fracstocks -F -o formsave
job:fracstocks  /bin/sort +0 -3 -o formsave/FRACSTOCKS.s formsave/FRACSTOCKS
job:fracstocks  /bin/mv formsave/FRACSTOCKS.s 2go/#SN:SN#DU:atexstocks

So the copy flow for London Stocks is that the FORMAT stage is replaced by ALL these job lines in sequential order :

– Remove any files starting FRACSTOCKS in formsave

– Run IPFORMAT with the input file and parameter file FRACSTOCKS leaving the output in formsave/FRACSTOCKS- Run IPXCHG once on formsave/FRACSTOCKS overwriting the input with the xchged file. This is to get the correct StockNames eg:

x/Hong Kong Telecm/HK Telecom

– Sort formssave/FRACSTOCKS on the first three words, creating an output file called formsave/FRACSTOCKS.s- Move formsave/FRACSTOCKS.s to 2go with destination (DU) ‘atexstocks’ and preserving the original filename (SN)

Please refer to the documentation on IPFORMD in the programs section for more information on jobs.

One further not on the Stocks is that all the output files pass through the STOCKS2ATEX xchg.

This is used to add column headers before certain Stock names. eg :

x/Northern Elec/n{M1StockrCloserHighrLowrPrevrn{M0Northern Elec

4. PARAMETER FILE REFERENCE GUIDE

This is the reference and hints section describing the main Parameter file used for processing.

<!–

The file is in two parts :

–>

Overview

• Record Processing Lines
• Reserved Names
• Structure of the Parameter File
• Comments and the binary version of the Parameter File
• Processing Loop

Part 1. Define what the input file looks like plus a general section covering fixed information.

define the type of file.

define the record and field (if any) separators or sizes.

define the record and field keytypes (if any).define all ‘sets’ – short forms.

define all partial field structures – ‘partial’.

define all localised searches – ‘match’.

define strings placed before and/or after the data.define alternate names and headers.

Keywords:

Part 2. Output Section

• Record Processing Lines
• Spaces, End of Lines, Double Quotes
• Builtins
• Tests
• Using Flags
• Save Areas
• Using Counters
• Using Calculations

Overview

Record Processing Lines

This is flagged as beginning with the ‘output:’ keyword. It describes what processing should be done for each input record.output

 r=X

lines to describe the output.

Record lines can have system variables, input fields, tests, builtin formatting etc.

Each one of the keywords is described below.

Each line is a self-contained item ending with a NewLine (Unix) or CR LF (PC) or CR (Mac). The text parameter file can be edited by any word-processor on any (normal) platform AS LONG AS the end result is a pure, raw ascii file with no Presentation or fancy graphics embedded.

Comments are the usual semi-colon in front.

	; comment

Reserved Names

The list of keywords is the list provided above plus a series of tests and builtins described below.

Note that it is possible – but not advised – to override some of these keywords. So these names should be considered reserved. In addition a few other names have been reserved for future use :

blksep:

blkkey:

blksiz:

The Structure of the Parameter File

The text file is split into 2 main parts as described above. The OUTPUT section must the second and is marked by the keyword ‘output:’ on a line on its own.

By common consent, the first part starts with the definitions of the file, records and fields but this is NOT strictly necessary. The advice is – do whatever is easiest for you.

Comments and the binary version of the Parameter file

Comments are lines STARTING with a semi-colon. You can have millions of comment lines and, except for the first run, they will have no effect on run-time speeds.

This is because ‘ipformat’ uses a compact, binary version of the Text Parameter file which is built automatically by ‘ipformbl’ every time you modify the Text version.

The only time you need touch the binary versions (in tables/form/bin) is that they should be deleted during every software upgrade of the DF module.

Comments are encouraged !

The Processing Loop is…

The actual processing cycle is :

For each input file hitting queue spool/form :

– ‘ipformd’ will select the correct Parameter file using form/PROCESS

Normally this will mean starting ‘ipformat’ with the chosen file

Once started by ‘ipformd’, ‘ipformat’ will go through the following steps :

– Preprocessing

– Create a Fip style header unless not required with ‘nohdr’

This can be the standard one or can have extra fields added using ‘hdr’

– Add Data at the beginning of the output file if the ‘before’ keyword has been specified.

– Processing input file

– Split the input file into records and for each record :

– Start at the ‘output’ section of the Parameter file

– Check each record specification line. If it is for that record type, process it

– Loop around for the next record- Postprocessing

– Add Data at the end of the output file after the processed data if the ‘after’ keyword has been specified.

– Create a Fip filename

This can be the standard one or can be replaced if the ‘name’ keyword is specified.

– Send the file spool/2go for ‘ipwheel’ to distribute usually via ‘ipxchg’.

That’s it !

Syntax of Each Keyword

• Syntax for ‘filtyp’

Syntax:

	filtyp: (type).

Where type is

If the filtyp:f or v, you will need to specify the size (or for variable maximum size) of each record.

For most applications, filtyp:text means you have to also define the record separator, ‘recsep’ too.

• Syntax for ‘recsep’ and ‘fldsep’

Syntax:

	recsep: (FipSeq string)
	fldsep: (FipSeq string)
eg:		recsep:36

Normally the separator will be n or rn for NewLine or Carriage NewLine.

Note that if you just put ‘n’, ‘ipformat’ will automatically take any combination of CR and NL.

• Syntax for ‘reckey’ and ‘fldkey’ – Define the record or field key

This defines the type and size of either the record or the field key.

Normally keys are positioned at the beginning of the record/field but optionally these can be at the end or at an offset from the beginning or end.

Syntax:

	reckey: (length) : (type) : (posn) : (EndChr) : (delY/N)
	fldkey: (length) : (type) : (posn) : (EndChr) : (delY/N)

where

length or size of key – This can be 0 for any length

type (optional) can be

a-alphabetic

u-uppercase

l-lowercase

n-number

p-printable

s-space(or tab, CR, NL or FF)

b-binary (ie anything)

x-alphanumericc-control (ie = 0177),z-anpa hdr field (ie alnum plus non-quad/format punct.t-punctuationposn (optional) is the offset on the key from the start of the zone If negative count from the end of the zoneendchr (optional) is a single chr which terminates the key

The separator can be any punctuation chr as long as the same chr is used for each field. eg the following two are equal :

	fldkey  3,n,,|
	fldkey  3|n||174

Ie the end Chr is a pipe but as that is used as a separator, use the octal value

What is a key ? The key is used really as a TYPE of PROCESSING flag for the output section.

It can be a unique record key – such as a stock code – but if you have several thousand, it is going to be unwieldy specifying all of them.

So generally we are trying to classify records into general types. For example of a text file containing schools results like :

	School	Pinky High
	Head	James Pinky
	Pupil	Ramsay		Macdonald	31.6
	Pupil	Thatcher	Margret		77.3
	Pupil	U-Dones		Helen		99.3

We can use the first field, which is alpha and variable length followed by a space.

	reckey:0:a

This can be signalled in the output section as :

	r="school"	(do the school bit)
	r="head"	(do the head bit)
	r="pupil"	(do the pupil bit)

• Syntax for ‘recsiz’, ‘fldsiz’ – Define the length of fixed size records & fields

Syntax :

	recsiz: (length)
	fldsiz: (length)

where length is the size of key

• Syntax for ‘keycasesens’ – Field and Record keys can be upper AND lower

Normally all keys – record or field – are considered to be case insensitive. So a key r = ‘aaa’ will pick up both AAA and aaa.

Use this command to force the difference.

Syntax :

	keycasesens:yes

• Syntax for ‘number’ – Change the number system for specifying non-printable chrs

When a non-printable chr is specified in the form ’12’ the ‘number’ system can be changed to decimal or hex.

The default number system is octal.

Syntax:

		number:dec
	or	number:hex
	or	number:oct

The change takes effect for all lines in the parameter file lower down until changed again by another ‘number’ keyword (Why you would specify different number systems for different parts, I have no idea).

So a New Line chr will be

		12		octal
		10		decimal
		a		hex

• Syntax for ‘wild’ – Allow wild card strings using a particular chr

Syntax :

	wild: (Chr to use to signify a wild string)
	wild:*

This allows a wild string to be used when specifying record keys.

Note there is NO automatic wild string chr – you always have to specify it.

For example:

	wild:$

Allows us to specify in the output section :

	r=$	"This is done for all records"

• Syntax for ‘wchr’ – Allow a wild card character using a particular chr

Syntax :

	wchr: (Chr to use to signify a wild character)
	wchr:?

This allows a wild character to be used when specifying record keys.

Note there is NO automatic wild chr – you always have to specify it.

For example :

	wchr:?

Allows us to specify in the output section :

	r="abc?e"	"This is done for all records abc(something)e"

• Syntax for ‘stripeol’ – Do NOT strip multiple blank records

Syntax :

	stripeol:no

Where the ‘recsep’ is some combination of CR and NL, normally blank lines or multiple occurances of CR and NL are stripped.

This command is used to turn that option OFF and to treat all lines as valid records, even ones with no data.

• Syntax for ‘startkey’ – Force the record Key or type of the BEFORE-FIRST record

Syntax :

	startkey:yes

This forces the key BEFORE the first record to be ‘x1594’ which can be used in the ‘ifprv’ test – if previous key.

• Syntax for ‘set’ – Short forms or format names

Syntax :

	set	(name)	(any fixed text) 
	set	pagehdr	Stock&ltt&gtClose&lttr&gtHigh&lttr&gtLow&lttr&gtPrev&ltqr&gt#n

Set lets us specify easy-to-remember names to reference strings

Sets can NOT be split over several lines.

All leading and trailing spaces are stripped. So use wither double quotes to embed or the ‘s’ escape string. To specify a double quote, use the octal string.

FipSeq strings are useable but note that ‘set’ are parsed when the parameter file is chnaged/created so that any Variable data will be of that time and any FIP hdr data meaningless.

	eg	set	timedate	$d-$m-$y $h:$n

will produce the date and time of when the parameter file was last changed.

To get run time date/time, specify the same string in the output section record line NOT the ‘set’

The name of the set – timedate in our example – is case-INsensitive. So it may be called as TIMEDATE or TimeDate or any other variation known to man.

• Syntax for ‘include’ – Include another file of instructions

Syntax :

	include	(filename)
	include	quark.tags

This will include another text file in tables/form/text with more Data Formatting commands.

The filename is force UPPERCASE as normal so in the above case the file will be :

	/fip/tables/form/text/QUARK.TAGS

Generally include files will have commands such as ‘set’, ‘calc’ which are common to a series of Data Formats – like a quark styles for example.

Note that if you update the include file, ipformat will NOT rebuild its binary so you need to ‘touch’ all the other text files in form/text to get the new version. Normally of course ipformat realises a change has been made to the main text file and rebuilds its binary automatically. :

	ipt form/text
	touch *

• Syntax for ‘calc’ – Define a Calculation or Choose output style for a number

Syntax :

	calc:(name)	(The calculation) 
	calc:percent	100*(c1/c2)

Define a calculation where cX are used as variables. The ‘name’ is that used in the ‘output’ section.

Calculations can NOT be split over several lines.

Variables are loaded at run-time using savnum, savbyte, savint, savswint, savlong or savswlong.

The default precision for a calculation is 2 decimal places. This can be overridden using the syntax :

	calc:percent:0	100*(c1/c2)

where the ‘0’ after the name is the number of decimal places in the range 0-6.

Calc can also be used to change the output format of a number by specifying the precision. If the raw data is in 4 dec places and you only want 2:

	calc:dec2:2	c22

and in the output section, load c22 from data – in this case field 3 :

	savnum=22 f3

Operators can be :

+ plus

– minus

* multiply

/ divide

Take care when dividing by zero !

Use round brackets to denote how the calculation should be worked out. Ie the deepest level is calculated first, and then the calculation is worked from left to right.

For example :

	(c1/c2)*(c3/(c5+c4))/100
will add run through the following order :
	Step 1	(c5+c4)
	Step 2	(c1/c2)
	Step 3	(c3/result of Step 1)
	Step 4	(result of Step 2 * result of Step 3)
	Step 5	(result of Step 4 / 100)

• Syntax for ‘fraction’ – Define a Fraction

Syntax :

	fraction:(name):(precision) (Output style WITH & WITHOUT fraction)

where

name is the function to be used in the output

precision is smallest denomiator – 2, 4, 8, 16, 32, 64, 128 or 256 – default is eighths

+ is used as separator – which can be any punctuation

first style is parse string used if THERE IS A FRACTION and integer

+ is used as separator

second style is parse string used if THERE IS ONLY AN INTEGER and NO FRACTION

+ is used as separator

		fraction:star:16	+ZI ZD/ZN+ZI.0+
		fraction:stox:32	|ZI (ZD-ZN)|ZI|

Fraction takes a field, partial field, saved field or calculation and split into 3 portions which can then be used as the normal FipSeq.

ZS is the sign + or –

ZI is the integer

ZD is the fraction amount

ZN is the fraction denominator

Specify two styles – the first for if there is a non-zero fraction amount and the second if there is none.

• Syntax for ‘base’ – Define a Base number

Syntax :

	base:(name):(precision) (Output style WITH & WITHOUT fraction)

Base is exactly the same as fraction except base will NOT attempt to ‘reduce’ the fraction.

ie 2/4 is left as ZD=2, ZN=4 while fraction will force ZD=1 ZN=2

• Syntax for ‘date’ – Generate a date and/or time from a number

Syntax :

	date:(name):(data order)	(Output style)
	date:mono:dm    +Last \ZW was \ZD-\ZN-\ZZ+

where mono is the name which is used in the output section

The Order of the raw data is important. So we divide it up into a series of 0 or more 2-digit numbers (or space digit) whose order is specified using :

d – day — 1-31

m – month — 1-12

y – year — (last 2 digits only, must be after 1970)

c – century — 19 or 20 only

h – hour — 0-23

t – minute — 0-59

x – padding — ignore 1 or 2 numbers at this position. Only 1 padding is allowed.

So for the 21st March 2010, the incoming data must be :

Obviously only one format of data can be handled by a single ‘date’ but you can have as many ‘date’s as needed.

Note that spaces, letters and punctuation are stripped and/or used as field delimitors. So the following are all equivalent :

200896

20th 8-August, 96

This day 20th August (the 8th month) in the Year 96

The Output format is defined in normal FipSeq between two deliminators (‘+’ in our example above, but can be any punctuation except semicolon ‘;’).

The new data is added to a series of extra FipHdr fields :

ZD – 2 digit day of month

ZM – 2 digit month

ZY – 2 digit year e.g. 92

ZZ – 4 digit year e.g. 1992

ZW – Day of week e.g. Monday, Tuesday etc

ZS – 3 character day of week e.g. Mon, Tue etc

ZN – Full name of month e.g. January, February

ZL – 3 character month e.g. Jan, Feb, Mar etc

ZJ – Julian day of year

ZH – Hour 00-23

ZI – Hour 00-12

ZT – Minute 00-59

ZP – am/pm

ZA – Week Number 00-53

ZB – Week Number 01-53

ZC – Day of the week 0-6

(see also manual page for ‘strftime’ for slightly more information)

Note that actual Day and Month names depend on the LOCALE of your shell/computer.

The Default output format, if none is specfied, is :

	+ZW, ZD ZN ZZ+

Which for (order dmcy), (data) 16111997 English LOCALE gives

	Sunday, 16 November 1997

Note that if any information is NOT supplied, the run time date/time is used.

• Syntax for ‘partial’ – Further subdivide a record or field

Syntax :

	partial:(name)	(type):(length):(startchr):(endchr)

where

type can be

a : alphabetic

u : upper case alpha

l : lower case alpha

n : numeric

t : punctuation

p : printable (generally Spc to~ })s : space ( tab, space, ff, cr, nl, lf)z : anpa type string (alphanumeric plus hyphen)b : binarylength can be zero for unlimited. startchr and endchr are optional.

There can be up to 100 partial fields as required.

The contents of a Partial field are accessed by specifying pX where X is the sequential no from the start of the partial – ie first is p1, then p2 etc

Syntax to partial a field in the output section record line is

	(partial name) (field name)

and then you can use any partial fields. For example:

To pull apart a data in the form : 26th March 1997

		; comment	   26  th      March   1997 
		partial:pdate	s:0 n:2 a:0 s:0 a:0 s:0 n:4

In the output section, assuming a field 3 of record type 99 contains the date, we can out put just the year by :

		r=99	pdate f3	p7

will produce “1997”

• Syntax for ‘match’ – search and replace on a single field

Syntax :

	match:(matchname)	/(search string)/(replacement)
	match:(matchname):c	/(search string)/(replacement)

where

matchname is a unique alphabetic name

c (optional) is for case-SENSITIVE searches

the delimiters – ‘/’ in the above example – can be any punctuation as long as they are are the same for that match line

This is a localized search and replace or search and zap function which is applied ONLY to a record, field, partial field or save area.

Normally the search is case-INSENSITIVE but can be forced so with the ‘:c’ after the matchname.

No wild chr or wild strings are permissable at present.

To specify in the output section :

		(matchname) (zone)

where

matchname is the SAME as specified in the ‘match’

zone is the record, field, partial or save area.

Up to thirty matches can be specified for a single zone.

For example :

	match:jan	/1/January/ 
	match:feb	/2/February/ 
	match:mar	 /3/March/ 
	etc .. etc 
	match:dec	/12/December/

In the output section – let’s pretend field 4 of record type 23 contains a number we want to translate to a month :

	r=23	dec nov oct sep aug jul jun may apr mar feb jan f4

Note that dec nov and oct are done first else if ‘f4’ was “11” then the jan match will replace “11” with “JanJan” !

‘match’ complements the character xchg in IPXCHG. However they are slightly different in that ‘match’ is localised in that you apply it ONLY to a single field whereas IPXCHG works on a whole file (or using flags, to a selected paragraph, line or section of the file).

• Syntax for ‘style’ – Reformat a data zone

Syntax :

 	style: (name)   (a single printf conversion syntax)
        style:twonum    %.02d

– Uses printf which is nasty but a standard (of sorts). Do a man printf for fuller information if you need.

– Always starts ‘%’

– If the expression does not end with an ‘s’ (‘d’ for integer for example), then the string in the header field is first converted to that type.

– Specify One and ONLY one expression (can not have %s%d%f) – as it takes the first only

– do NOT use for fixed data, ONLY the conversion string.

– Types are :

string : s

char : c

long : d,i,o,p,u,x,Xfloat : f,e,E,g,G% : print a % !type n is ignored ??

Examples

• Syntax for ‘name’ – Overwrite the default name of the output file

Syntax :

 	name: (Fip Hdr strings)

Remember that the ‘name’ is a list of Fip Hdr fields which will probably include the SN field which is the original name.

The default name on output is :

	#SN:(original filename)#DU:(name of the paramfile)s#SC:FORM#DF:FORM

Where DU is forced lowercase.

The ‘hdr’ and ‘name’ keywords have the same syntax and roughly the same use so further information is in under ‘hdr’.

Remember ‘name’ is done AFTER all processing of the data – it is the last thing done before the file is sent on for further distribution. So information gleaned from the input, perhaps left in Save Areas, can be used in the name. ‘hdr’ however is done FIRST, BEFORE any data is touched.

• Syntax for ‘hdr’ – Add extra fields to the FIP header of the output file

Syntax :

 	hdr: (Fip Hdr strings)

Remember that although the specification MUST be kept on a single line, HASH may be used as a delimitor for Header fields.

Note also that as the output file is a completly NEW file and has no physical connection to the input file, NO Fip Hdr fields are transfered from input to output UNLESS specified in the ‘hdr’ or ‘name’.

Generally use ‘hdr’ to preserve fields you need as for ‘name’ there is a limit to the number of characters – and the type of characters : no meta characters or slashes ‘/’etc – such as the Source Header for example :

	hdr:#SH:SH#SN:SN#DF:roger

will transfer the SH and SN fields and force DF to roger.

As ‘hdr’ is processed BEFORE the data, no information generated by IPFORMAT during processing is available. However the ‘name’ keyword is processed AFTER so such data may be added then.

The default Fip Hdr put on an output file has the following fields :

Other useful fields can be :

DF – output format for ipedsys, ipgtwy, ipout, ipprint etc. This must be in the ‘name’ parameter as by default it contains ‘DF:form’ eg :

	DF:albert

will pickup tables/print/ALBERT if ipprint is the program sending to the final destination.

 

CX – force the xchg name to be used to the contexts of this field. eg

	CX:STOCKS

will override the SC2DC fields normally used for ‘ipxchg’.

• Syntax for ‘nohdr’ – do NOT add a Fip header to the output file

Syntax :

	nohdr:

This is used when the output file is to be used immediately by a Unix program which does not understand the Fip Hdr – sort for example.

• Syntax for ‘chrset’ – force the SC or Source Character set

Syntax :

	chrset: (name)

This fills in the SC: Fip Hdr field. The default is FORM.

• Syntax for ‘before’ – Insert data BEFORE any output from the input file.

Syntax :

	before: (Fip Seq strings and Record processing commands)

• Syntax for ‘after’ – Insert data AFTER ALL output.

Syntax :

	after: (Fip Seq strings and Record processing commands)

Both ‘before’ and ‘after’ can have tests, builtins, contents of save areas etc although obviously for ‘before’, most of these may have nothing in.

In the Output section – Record processing lines

The actual data is processed and output using the Record processing lines.

The Syntax of each line is that the first bit specifies which record type or key the rest of the line applies to :

		r=(key)		(output)
For example :
	r=abc	"Fried fish starts " s1 spc f4 " and the rest ..."

There can be multiple lines for the same record type. The following two lines will give the same result as the one above :

		r=abc	"Fried fish starts " s1 
		r=abc	spc f4 " and the rest ..."

For lines where you want to process for all EXCEPT a particular record type/key, use teh syntax ‘r#’ :

		r#35	"Nobody wants record 35s !"

How to specify you want to use, format and/or output zones ?

A ‘*’ can be used in certain cases to signify ‘ALL’ zones ie

Use double quotes for alphabetic keys and those with embedded spaces.

You should try not to use ‘sets’, ‘partial’s or ‘match’s with names in the form ‘z999’ where z is one of the single letters above and 999 is a number in the rangle 1-999.

Note that blocks are ‘super records’ but should be ignored for now.

Note that case is IGNORED in keys in the current version .

When ‘ipformat’ finds a name in the record processing line, it does the following sequence :

– check to see if it is an already specified constant ‘set’ name.

– if not, is it a zone – record, field, partial or block (eg p44, f3)

– if not, is it a builtin command (see below)

– if not, is it a save zone, flag or counter (eg s1, x33, c7)

– otherwise it is considered some FipSeq string and saved as such.

Spaces, End-Of-Lines and Double Quotes in the output

One common failing when putting together a new parameter file is to completely forget about spaces (or other separators) and end-of-lines (CR or NL or CR NL or whatever) in the output file.

The point is – you have to specify them as NOTHING is implicit in the output file. There is no hidden magic which suddenly realises that you want an end-of-line when you need it. You have to state where and when you want them.

Generally this will be done by either putting them as constant/’set’s or specifying them in the record processing line. The following are exactly the same :
Either

	set	spc	s
	set	ql	n
	output:
	r="BIG"		f5 spc f3 spc f99 spc f5 ql
Or
	output:
	r="BIG"		f5 s f3 " " p99 s f5 n

As you can specify a space as either ‘s’ or in double quotes, to output a double quote character, you need to specify it as an number : 00.

Builtins

There are a number of builtin conversion routines for formatting zones – records, fields, save areas etc.

These are called by placing the name of the conversion BEFORE the name of the zone eg :

		zapspcextra p5

which means :

‘zap all leading, trailing and multiple spaces from partial field 5’

A single zone can be subject to several builtins :

zappunc zapspc caps f=Z

which means :

take field “Z” and zap all punctuation and zap all spaces and force uppercase before outputting.

	r=99	incfile /home/standing/ s4

	r=abc	newfile

	r=abd	newfile	#DF:newform#QQ:$Z

Tests

There is a further selection of tests which can be made one zones inside the date.

These enable you to select even finer some processing depending on actual data. If and ONLY if the test is true is the rest of the line continued with.

Syntax for Tests

	(ifxxx) (first string) (second string if required)

where strings can be fields, partials, saves or fixed text

Note that sequence is important for comparing two fields that may be different lengths as ifeq will be true if the first field is complete ie :

		1st=AAA		2nd=AAABC	will be true 
		1st=AAABC	2nd=AAA		will be false

Example 1 :

	r=24	ifprv r=35	"Last record was type 35 and this is 24"

Only if the previous record type was “35” will the string be output

Example 2 :

	r=24	 f3 ifnul f3 " _ " x99

For record type 24, output field 3 and if there was nothing in it, output a (spc) (dash) (spc). Flag 99 will also be set if there was nothing there.

Example 3 : When using numeric data, please ensure that all extraneous characters are stripped from the zone before the test. In particular strip commas, plus signs, currency symbols etc.
For example, if field 7 has data like p9300.0007 and save field 9 has 10,000 compare the two by :

	match:mnop	/p//
	match:mnocomma	/,//
	output:
	r=99	ifgt mnop mnocomma f7 mnop mnocomma s9      "Field 7 &gt  Save 9"

Using Flags

Flags are a really useful means for deciding type of processing to do – or NOT to do.

For example, let’s use flag 3 to test if record type ‘abc’ has Richard, Helen or George in the first field. Print out ‘New name is (name) (newline)’ if it does :

	r=abc	clrflag=3 
	r=abc	ifeq "Richard" f1	x3 
	r=abc	ifeq "Helen" f1		x3 
	r=abc	ifeq "George" f1	x3 
	r=abc	ifflag x3		"New name is " f1 nl

Save areas

Save areas may be used to store strings – either in their original state or after conversion/formatting by other built-ins. The maximum save number is 299.

	eg	save=1	f3 
		save=5	caps f7

	eg	save=77 "ABC"

		savcat=77 "DEF"

Save areas may be used in the normal ‘if’ tests, eg :

	ifeq "AAA" s1	x88

if the contents of save area 1 starts “AAA., set flag 88 ON

Using Counters

Counters are integers (ie proper numbers with no decimals or fractions in the range -32000 to +32000.

They are signalled by ‘zX’ where X is a number.

They can be used to count the number of occurences of a record or field or even types of data and act accordingly.

All counters are set to zero when the program starts and by using the builtins :

incctr

decctr

setctr

clrctrin the Record processing lines, you can manipulate them.

For example, to add some random markup every 10th line of a record type AB using counter 26 :

	r="AB"	incctr=26	ifeq 10 z26	clrctr=26	"[pt9][font99]"

ie : For all records type AB, add 1 to counter 26, then test if ctr 26 is equal to 10; if so reset ctr 26 back to zero and output string ‘[pt9][font99]’.

The syntax for ‘setctr’ is

setctr=99 345

– set ctr 99 to a fixed number 345

setctr=297 p3- set ctr 297 to the contents of partial field 3.

In the second example, if the p3 is NOT a number, ctr 297 is set to zero. Also if p3 is a decimal number like ‘123.456’, only the main number is saved.

Using Calculations

Calculations are defined in the first part of the parameter file and used in the record processing part :

For example :

	calc:mktcap	c1*c2
	output:
	r=BC	savnum=1 f5	savnum=2 f7	mktcap

In this example we define ‘mktcap’ to be variables 1 and 2 multiplied together. Then in the output section, for record type BC. field 5 is saved in variable 1 and field 7 in variable 2 before we do the calculation and output the result.

A quick word about BINARY numbers.

Normally fields will hold printable data – such as in the example above – and we use the builtin ‘savnum’ to take that number for use in the calculation(s).

However some data is already in a binary form. Use builtins ‘savbyte, savint, savswint, savlong and savswlong’ to load these numbers. Often these will be derived from a partial field using the ‘b’ for binary field type. eg:

	partial:bindata	b:2 b:4 b:2 b:4

What is a swapped integer or long ? Some computers – like the PDP-11 and most Intel 16+ bit chips – hold the data in reverse byte mode.

– So if the data has been generated on a SPARC OR rs6000 or a Mac the data is ‘normal’ – use savint or savlong.

– While data from PDP-11s or Intel based PCs could well need to be swapped.

Loading Variables :

Save a printable zone as a number variable – use ‘savnum’

savnum=5 p4

– save the contents of p4 as a number. So if p4 held the string ‘789’, c5 would be the number ‘789.

Save a fixed number in a number variable – use ‘savnum’ again

savnum=7 1234

– loads the number ‘1234’ into c7.

Save the contents of a single byte – use ‘savbyte’

savbyte=33	p7

Note that the contents of the variables, c1, c2 etc are not amended by the calculation UNLESS you specifically save it, eg :

	r=BC	savnum=1 f5	savnum=2 f7	savnum=3 mktcap

will load c3 with the result of the ‘mktcap’ calculation.

Examples of Builtins :

STRLEN

	; test the field 2 is greater than 44 chrs (ie 44 is less than strlen of f2)
	r=HH    ifnnul f2       iflt 44 strlen f2       "Big Field 2 here over 44 chrs long" n
	r=KK    "Save Field for Name (s55) is " strlen s55 " chrs long"

ZAPLEADZERO

	; data - field 99 is 00000330303, field 101 is 00000000.00
	r=3     "This outputs 330303=" zapleadzero f99 ", while this is
0.00=" zapleadzero f101

Putting it all together – some examples

EXAMPLE ONE

; file is variable text type
filtyp:t
; each record is separated by CR NL 2 letter type
recsep  rn
; There are NO fldsep - we will use partials
; There are NO reckey or fldkey - we will test strings for the type of processing
; allow wild cards
wild:*
; 
set     qc      04n
set     topbit  n{M2Processing Date :
set	dash	" _ "

;Partial a Class line which contains the Class/Name/Length of race
; eg : Class 2 - ATV Anniversary Hcp. - 1000 M
partial:pclass  p:0::s s:0 n:0  s:0 t:1 p:0::- t:1 p:0

; localised matchs - search and replace
match:mhcp	/(Hcp.)//
match:mhcp2	/Hcp.//
; replace M with meters
match:mmeters	?M?meters?
;
;******************** output section ***********************8
output:
; Start by clearing flags 99 and 1 for each input record...
r=*	clrflag=99 clrflag=1

; Now test for ONLY those lines which match our needs...
;| all  |if field1 start|partial field1 |if partial    |set    |set flag 99 on
;| recs |with Class     |according using|field 3 is not|flag 1 |too
;|      |	       |pclass	 |empty	 |one    |
r=*     ifeq "Class" f1 pclass f1       ifnnul p3       x1      x99

; Print out only the names of a new race - only process if flag 1 is ON
; Use flag x101 to output [rf3] for the FIRST race only - which is the 1st class
r=*     ifflag x1       ifnflag x101    [rf3]   x101
; partial f1 again using pclass, if partial field 6 is NOT empty, remove extra
;	spaces, Do the two search and Replaces and output followed by a 004 NL
r=*     ifflag x1       pclass f1       ifnnul p6 zapspcextra mhcp mhcp2 p6 qc
; remove extra spaces from partial 1 and output it, output partial 2 and 3, then
;	if partial field 8 is NOT empty, add (spc) (dash) (spc) etc
r=*     ifflag x1       zapspcextra p1 p2 p3 ifnnul p8 dash mmeters zapspc p8
r=*     ifflag x1       qc

<!-- 
FipSeq
------

Many keywords in the DF module can have variables as well as fixed text for parameters.

These ar generically called FipSeq strings and can be :

      - Normal Ascii printable text : remember that leading and trailing spaces
		are always trimmed so use double quotes to embed :
		"   Some leading spaces and some trailing    "
		Also in the record specification ALL spaces between fields are
		stripped; again use double quotes to embed or Unix escape chr s
      - Unix style escape chrs : backslash then lowercase chr :
		Carriage return	CR  : r
		New Line	NL  : n
		Space		SPC : s
		Backspace	BS  : b
		Tab		TAB : t
		Backslash	    : \
		Form feed or Vertical Tab  FF or VT : f
		Wild chr (if specified) : w
		Hexadecimal number  :x99
		CR NL		    : l
      - Octal numbers : backslash and 3 digits zero padded : 01, 377
		These can be decimal or hex by using the 'number:' keyword.
      - Internal FIP header fields : backslash and 2 uppercase chrs :SN, DQ
		to extract fields from the Source Header ( Fip field SH) use X?
		ie XP for Priority.
      - System variables :
		$D : day of month in 99 format
		$M : month in xxx format
		$I : month in 99 format
		$Y : year in 99 format
		$H : hour (99)
		$N : min (99)
		$B : sec (99)
		$J : julian date (3digits, Jan1 is 001)
		$S : 3 digit ascending sequence number
		$Z : 4 digit ascending sequence number
		$A : atex orig field (SOURCE;06/06,14:35)
		$C : number of chrs in file
		$W : number of words in file (IP_WORD_LEN)
		$R : Random letter
		$O : end optional text
		$X : strip trailing spaces of buffer so far

Fip Header fields can be further manipulated using pseudo-fields :
	fixed: QZ       1234543
	partial:QT      ST,3,2,U,
	combie:QZ       ep|na,(000000)a
	option:QT       ep,11,7,s

For fixed fields : 
	fixed: QZ       1234543 
	ie If QZ is specified, replace with 1234543 
	Syntax  fixed: [newfield]       [tab/space]     [fixed text] 

For partial fields. An example : 
	partial:QT      ST,3,2,U, 
	ie If QT, take ST header field posn 3 for 2 chrs, UPPERCASE. 
	Syntax  partial: [newfield]     [tab/space] 
		[existing field] [comma] [startposn] 
		[opt comma length] 
		[opt comma processing] 
		[opt comma start chr] 
		[opt comma end chr] 
	where : Start and Length start from 1 not 0. 
	 Length can be zero or not defined for all characters in the field 
	 Processing is U-uppercase, L-lower, N allow only numbers, P-printables 
	 The Start Chr can be used to start the string. If there is also a 
	 length then this length is FROM the Start Chr. 
	 The End Chr can be used to end the string when it is undefinite length.

For combinations : 
	combie:QZ       ep|na,(0000000)a 
	ie Use EP header field, if not there use NA field, if not use the 
		fixed text '(0000000)a'. 
	Syntax  combie: [newfield]      [tab/space] 
		[existing field1] [|] [existing field2] 
		[opt comma] [opt default fixed text] 

For optional fields (used in conjuction with the $O flag): 
	option:QT       ep,11,7,s 
	ie If EP header field exists and has a space in the 7th position, 
		send this text else strip text until the $O flag. 
	Syntax  option: [newfield]      [tab/space] 
		[newfield] [?] [existing field] [comma] [size] 
		[opt comma] [opt posn of test chr] 
		[opt comma] [opt posn to send remainder of fld] 
	where size is minimum size of field. 
	The send parameter will send contents of the field from that position 
	onwards. If not present, the field is used ONLY as a test and NOT 
	to send chrs.  Note that both size and test are start from 1 not 0. 
	A single chr can be tested to be non-space as in the example above. 
	If either the size or the test is FALSE, all text and sebsequent data 
	whether fixed or variable (including more Optionals) is ignored until 
	the EndOpt flag is met - '$O' (see below). 

Watch out using FipSeq strings in 'set's
----------------------------------------

Note that 'set' are parsed when the parameter file is chnaged/created so that any Variable data will be of that time and any FIP hdr data meaningless. eg: 
	set	timedate	$d-$m-$y $h:$n
will produce the date and time of when the parameter file was last changed.

However FipSeq variables specified in record output lines will return run-time data eg :
	r=*	"And now the date and time : $d-$m-$y $h:$n"
will produce date and time when that record was processed.

Importance of your LOCALE
-------------------------

Unix allows you to play around with character sets - called Locales - and this can have repercussions for the data formatting module.

These are defined as part of the ENVIRONMENT.
	- look at the man pages for 'setlocale'
	- check you own settings with 'env | more'
		for LANG, LOCPATH etc.

For any non-English environment, it is important to define :
	What is a Alphabetic chr ?	- normally a-z, A-Z
		- remember all the accented characters
	What is a control character ?	- normally octal 0-037
		- sometimes these can also be octal 200-237.
	What is punctuation ?		- normally ",.!@#$%^&*()-_=+[]{};':"
		- if you want to use 'zappunc', make sure.

If you get it wrong, you may find that an accented chr you consider to be alphabetic is processed by 'ipformat' as a binary chr. So take care.

Current Version
---------------

Current version	limits are 
	flags		- 300 allowed in the range 1-300
	counters	- 300 allowed in the range 1-300
	saves		- 300 allowed in the range 1-300
	calculations	- 300 allowed in the range 1-300
	partials	- 100 allowed in the range 1-100
	matches		- 30 allowed for any one field
	record length	- 64k maximum
	fields in a record	- max 100
	there can be up to 1000 'set's and other constants
There is also an internal buffer size for the size of the binary of the parameter file which is 16k - however most binaries are under 1k and the biggest seem so far is about 5k.
	keysize must be less than < 20 chrs
	ipformat misses 1st record if the initial sep is missing
	split keys are not allowed
	all keys are case INSENSITIVE

Save areas must be less than 16K each. In versions from 040, the program should handle many changes, additions etc. However if you do use buffers which are TOO big, an error message to the fact is logged in the Item Log and data MAY be ignored.

Until modified, note that a clrsave=* will reset everything.

------------------------------------------------------------------------------

		6. POSTSCRIPT DRIVER - ipsetter
		-------------------------------

This program processes files in a funny SGML-like feed into a native setter code - actually it is very simple markup language.

It loops around a queue, when a file arrives it processes it into the codes and works out width and depth.

It passes the file onto the next stage which can either be dirrectly to the setter or a holding queue for later inclusion into a page or block.

The device file, tables/form/SETTER, is used to define fonts and character widths.
The current version supports PostScript only.

FIP Header fields in the incoming file are searched for :
	FN:	File name of the output file if different from the incoming
		file. This name is Overwritten by the Sort Key if there is one.
	FQ:	Output queue for PostScript file. This defaults to spool/setit.
	FU:	Destination for output file. Generally if a destination is
		to be added, the file also need to be sent to 'spool/2go' for
		IPWHEEL to redistribute - use FQ for this ie :
			#FN:newname#FU:nextstep#FQ:2go#etc
		where 'nextstep' is a valid destination in the USERS file.
	FCR:	Should CR, NL (LF) be thought of as quads ? Normally yes !
		Add 'FCR:' to the Fip Hdr of the incoming file and all CRs and
		NLs are ignored and a quad is ONLY the Quad command 
		(where ? is l, c or r).
	FK:	Sort key - used generally by the User Interface 'form' for
		collecting and collating many files that have been processed
		individually previously - like election results for example.
	FP:	Page number to start with - for books and documents only.
	FH:	name of PS HEADER file in tables/form/postscript if NOT the -h
		input option or, failing that, the default (PSHDR). To specify
		NO header file use a null parameter eg #SH:bigpage#FH:#FP:99#
	FM:	USER file to include after PSHDR header file and before main
		body of the text. This overrides the -u input option whether it
		has been specified or not. The file should be pure PostScript
		and be in tables/form/postscript. FM is case-sensitive.
	FT:	name of PS TRAILER file in tables/form/postscript if NOT the -t
		input parameter or, if no -t, the default (PSTLR). To specify
		NO header file use a null parameter eg #SH:lilpage#FH:#FM:#FT:#
	FF:	Use the normal font on the setter or use a reencoded one which
		will be supplied in the PS HEADER file. Default is use the
		normal font with the name supplied. Options are :
			FF:yes		Use normal font
			FF:no		Use Encoded font as defined by CMP array
					in the PSHEADER.
			FF:small	Use the Small changes to the font as
					defined in the FipSmallVec and called
					by FipReEncodeSmall in the PS HEADER.
	FZ:	Name of SETTER file to use if not the default in tables/form
	FA:	Name of SETPAGE file to use if not the default in tables/form

Output fields are :
	FX, FY	x = width, y = depth in points*scale
	FL	no of lines
	FS	scale
	HT	is either passed thru or added (time in LONG format)
	FP	this pageno

Where the file will be merged with several others by the MUI, form, you may optionally NOT include font calls, header and trailers.

Formatting commands use the syntax :

	Only the first letter of the command is required.
	All measurements are in points.

Commands are :
	f	Change Font			
			Font name MUST be in the SETTER table
	p	Change Point and Set size

s Change Set size This defaults to the point size – and changes on change of point size too. c Change Column width l Change Leading The leading value is that moved in the Y or down direction after a Quad. q Quad (default is Quad Left) Quad may be Left, Right, Centre or Middle (ie stretch to fill). All quads except Zero will then Lead down the Leading value. Quad:zero formats text on the line and then returns to the left hand margin at the same Y position. Note that if FipHdr Field FCR: is NOT set, any combination of CR and/or NL will be treated as a Quad:Left. t Tab , Tab may be Left, Right, Centre, Set, Clear, Save or Restore. Tab Left, Right and Centre do just that. Remember to set tabs beforehand using one of the three Tab Set commands. If no tabs have been set, a Tab LRorC becomes a Quad. There are 3 different ways of specifying tabs : – X position. ie distance from lefthand margin. – Width of each field – Divide the column width proportionally Tab Set : Each tab is the X distance from the left hand margin in points. The righthand margin need not be specified. Tab Set-Width : Each tab is the width of that tab in points. Tab Set-Width can also have a REMAINDER flag set in 1 of the fields. In this case, if the total width of all the tabs is less that the column width, the extra is added to this tab. eg Tab Set-Proportional : Each tab is a proportion of the whole width : Tab Clear is This clears and ignores all tabs. Tab Save notes the current settings and clears. Tab Restore restores to last settings. This only works after a TabSave (x) NOT a TabClear (z) If you change Column width, respecify (ie S, P or W) the tabs. w White and Fixed Space , If specified with no parameter this is a White Space command which will force any data left and/or right within the quad or tab irrespective of whether the Quad/tab is L,R or C. eg X123456X X123 456X x Xtra Lead command This drops immediately the amount specified. z Pass thru command Use this to access PostScript routines you have written in you PSHDR or USER file. The whole string assumed to be in FipSeq and is parsed before output. This does assume you know a little bit about PostScript, so use with care. Please note that if you MUST preserve (or save/restore) the currentpoint, use the option z on or off. Normally the currentpoint is assumed to have been zapped. n Notes or Comment This will appear in the output file as a PostScript comment complete with preceeding ‘%’ : becomes % FipComment – Next Section here Please r Rules Specify width (default is column width) then height. Specify up and back using a ‘-‘ Optionally specify a tint percentage is the third parameter Note that rules go to the RIGHT (x direction) and UP (y dir) To force LEFT or DOWN, use negitive values. d Leader dots , The parameter is the leader character and defaults to ‘.’. m Set mark. Up to 100 marks are permitted. Note that both X and Y values of the current position are saved. j Jump to Mark The Mark should have been set beforehand. i Indent , , Specify whether left (default) or right. To clear indent, use WITHOUT parameters . Indent can be used with an optional delay and/or maximum number of lines. Syntax: where width is is points, starting at line is where the line may not be the first; forLines is the indent stops after x lines g Page heading The 1st parameter is the End of the last Page and the second is the top of the next Page. Note the punctuation /, to separate the FROM from the TO – this should be unique in the string. This is ONLY triggered if paginated. Square brackets are used to signify commands. The End of last page is NOT counted in the depth – pls make allowance manually in SETPAGE. However the Continued to IS taken into account. h Column headings. This is ONLY triggered if paginated. Square brackets are used to signify commands. Do NOT embed NL or CR as these, plus the end command ‘>’ are used to end the command. These can be replaced by a heading by specifying ‘rephead:all’ in the SETPAGE file. A heading in this case meaning the last block (ie between and ). An alternative Col heading can be specified for the replacement by using : Do NOT mix the separator, ‘:’ or ‘=’, in the same command. The flag must be ‘1’ for normal and ‘2’ for alternate hdrs. Use ‘althead:all’ to flag you want the alternate option. Note althead IGNORES the next block – ie replaces both the header and the normal column header. v Dis/Allow Vertical Justification. , or Note VJ point and set max VJ addition, , VJ is ON automatically for pagination. b,e Block , Flag the start or end of a Block. This block will NOT be split by VJ unless forced to by the force: parameter in the file. signals the beginning of the block and the end. a Column turn point. For column turn at this point This can also be used as an optional col turn point where the number is a priority turn (1 being highest). If found when checking for Widows/Orphans, this is used instead of number of lines. Note that or is a ‘MUST turn here’ while , etc are ‘if you have to turn, turn here’. For ‘book:’ work the syntax is slightly different. An extra parameter is added which is the depth that MUST remain in the page/column. If less, the page/column is turned and any text between and or in the page/col hdr is stripped. The last option is which means do-not-turn-the-column- before-this-line pls. You can still turn after though. This option is only triggered properly if orphan is >= 1. y Yank or Save all text from until a u Unsave or restore the previously saved (ie yanked) text o Options : Allow reduced pnt size to fit Disable — ditto — Squeeze Word/Letter spacing % – to 5 % Disable Word/Letter spacing Enable zero and negative leading Normally and are ignored so that quads always incremnt down the page. This affects ONLY Leading NOT xtra lead Reset Page Number for book/document work Output Page Number for book/document work Make default quad Left, Centre, Right The normal default is Left. For PassThru, assume the currentpoint is preserved or zapped : or and <o:z:zap or Zap is the default. Zap will force a MOVETO before the next text while Preserve will place any text following. Create Fraction , with bar or without (use space as separator) This will create a fraction chr by shrinking font and moving The advantage is that setter can determine the width. The Fraction Bar can be any valid chr except space and numbers. If it is not-printable, use XXX format to specify an octal number Load or Display Variables , Please see the section below on variables. k Include file – for PostScript this is an EPSF The pathname is /fip/spool/epsf unless changed on startup. Note that the currentpoint is NOT set explictly so that you may insert after text. However normally you will want to just before. eg for a 32 pt blob : GOOD : BAD : 32pt xlead will be ignored Note that if you want to set a – which are normally cmd chrs – use the alernate chrs which are usually 0376 for . The main parameter file, normally SETTER, has the following syntax : ; comment name: postscript name of the output device defscale:1000 default scale if none specified band:5 band letterspace:3 letterspace unprint:o number system used for non-printable chrs o – octal (default), d – decimal, h – hex. ; first font = this will be font 0 font:helvroman:Helvetica font: IpsetterName : OutputDeviceName scale:960 scale if different from the default 222=’`ijl widths – all letters in this font with this width etc Widths is repeated for each different width for all the chrs of a font. Font and widths may be repeated as many times as is needed. If the character is not printable, it can be specified in the normal 00 fashion. NOTE that backslash should be specified as either \ or 134. Note also that comments start with a semi-colon – ‘;’. So to Specify a semicolon, use 73 (if octal) or 59 (decimal). The Pagination parameter file, SETPAGE, is optional and is used for ONLY those files requiring some sort of pagination. The syntax is : Note that there can be several lines for the same page – all starting with the ‘page:’ keyword. In fact for any parameters that may be specified several times, such as ‘hole:’, these MUST be on separate lines otherwise only the last on the line will be used. ; comment page: (filename) col: (col1),(col2).. name: (newname) page: (filename) hole:(col):(depth):(StartPosn) page: (filename) gutter: (width) vjinc:12 page: (filename) widow: (NoOfLines) orphan: (NoOfLines) page: (filename) pagebreak: (col), (col)… page: (filename) rules:(indent),(width),(top),(bottom) page: (filename) rephead: (col), (col)… or rephead:all page: (filename) althead: (col), (col)… or althead:all page: (filename) boxx: (replacement bounding box width) page: (filename) boxy: (replacement bounding box depth) where filename is case-sensitive and is the name specified in the hdr field FK or FN or SN. – NEWNAME is an new name for the output file. This is forced to a maximum of 31 chrs, Mac style. – GUTTER is the width in points of the gutter AND column. – WIDOW is the no of lines required at the bottom of a column. – ORPHAN is the no of lines required at the top of a column. – COL is the no of points per column. There can be multiple ‘col’. – REPHEAD will allow column header replacement if a Block (which is normally a header) is at the top of the column. This can be the word ALL or a series of columns. Note we assume the first column ALWAYS has a Block. – ALTHEAD uses the alternate header. This also replaces the next header block. The alt header is assumed to be the same depth as the normal header. It needs rephead to be flagged previously on the columns is question. – VJINC is the optimum extra lead to inset at each vj point. – PAGEBREAK will start newpages/output files at the top of this/these columns – RULES allows you to specify column rules in the gutters indent is the 1st rule from the gutter measurement. width is the width of the rule. top is the drop from the top of the page. bottom is the bong from the bottom of the page. – HOLE allows you to leave a hole in a column for a specified depth either from the top of the column (default) or from a start depth. Where a block covers several columns, specify a ‘hole’ for each column. There is no restriction on the number of holes in a column. – BOOK: Variable no of pages using the same col depth – no VJ. No Widow and orphan control – use to check before – DOCUMENT: Varable no of pages which are VJed and/or widow/orphan. Max is about 200 pages of 200 lines – BOXX: Replacement bounding box width. – BOXY: Replacement bounding box depth. Normally the bounding box is calculated internally. This allows a different BB to be specified when you may have used PassThru (z command) to add extra text and rules. is calculated internally. This allows a different BB to be specified when you may have used PassThru (z command) to add extra text and rules. – NOHOLECOLHDR: if using Holes – Do NOT add a col hdr if the Hole is in the middle of text – but still leave ones at the top of the Columns. Input parameters are (all optional) : -0 : do NOT flag zero width chrs in the item log default: yes -c : do NOT output comments default: leave comments in -d : default output destination if not overwritten by FU FK or FN hdr fields default: none -e : Path for EPSF includes default: spool/epsf -f : exclude font calls default: include -r : use an encoding table for fonts default: use the raw font This is usually CMP in the PSHDR file. -H : no ps header or trailer pls default: add both -i : input queue if not default default: spool/2set -o : output queue if not default default: spool/setit -l : do NOT log items through default: log -L : log items AND log each col details default: log only -p : output devices parameter file default: tables/form/SETTER -b : add bounding box if postscript -h : header file name default: tables/form/postscript/PSHDR -t : trailer file name default: tables/form/postscript/PSTLR -u : User header file name in tables/form/postscript default: none -x : make x, y independent (ie transform) -w : do NOT overwrite this file default: replace -v : version and exit Variables ——— There are a number of save zones available to store and reuse settings. For the following types, up to 100 zones can be saved : – Font Numbers – PointSize – SetSize – Leading – Column Width – Xlead values (vertical movement) – White Space (horizontal movement) – These can be specified for any of these

uses the stored value 22 for pointsize. – Of the 100 zones, note that zone 1 (one) is used by the system to hold the current value and zone 0 (zero) is used to hold the previous – ie current one before the last change. – There are several notations used : The normal one is % (optional Type Code) (Zone Number or Current flag or Previous flag) The % (percent) is the precedent char for a variable. The Type Code can be : f – Font Numbers p – PointSize s – SetSize l – Leading c – Column Width x – Xlead values (vertical movement) w – White Space (horizontal movement) The Type Code can be left out if it is the same as that specifed. ie These are the same :

Current value can be specified (QuarkLike) as a ‘$’ (dollar) : make setsize the current pointsize : or Previous valid value can be specified as ‘#’ (hash/numbersign : reload the pointsize used before the last change :

or

The six main types are : %p22 Use contents of PointSize zone 22. %22 Use contents of zone 22 of whatever you are specifying. %p$ Use the current pointsize %p# Use the previous value pointsize $ Use the current value of whatever you are specfying # Use the previous value of whatever you are specfying Obviously Current and Previous values (zones 0 and 1) are loaded automatically by the program. To Load values in other zones, use the Option command : load 22.2 into colwidth zone 23 save contents of pointsize zone 88 in setsize zone 88 save the current pointsize as leading zone 66 These values may be DISPLAYED as a PostScript Comment in the output file. Note that the values are held as DECIpoints which appear to be 10 times bigger that they need to be. This makes it easier for the program to process. To Display, use Option with NO load values : display the contents of colwidth zone 77 This produces : %FipSet Variables:ColumnWidth : 77 = 2000 or %FipSet Variables:ColumnWidth : 77 = NotInUse To Display ALL zones with data stored, use a ‘*’ (star) : display all Pointsize zones used. To Display ALL Types and ALL zones with data stored, use a type code of ‘a’ and ‘*’ (star) : display all zones used. %FipSet Variables:Font : Previous = f17 (0), Current = helvroman (90) %FipSet Variables:PointSize : Previous = 240, Current = 260, 22 = 2000 Tips —- — To Spread data within a tab : ie push data either side of the centre, use (a command with no parameter) 12345789 will be presented as (tab)12345 789(tab) Notes on this version ——————— – Indents and tabs – dont mix them in this version. wot you can do is strip tabs, redo col width, add indent TEXT .. say no turn here (a:n), quad and reverse, reset col, reset tabs and tab goddit ? — RULE is placed at gutter minus diff from left of column ie if gutter:132 and rules:6,.5 = rule .5 pt wide drawn at 126 pts — COL is depth ignoring any holes — HOLES – pls specify only one per line only. If you have more than one hole, use more than one line! — GUTTER – currently only possible to set a regular gutter – use PassThru to alter/Translate if you need different — Notes on PassThru : – Note that the scale factor is 1000 so all sizes, Moves etc need to multiplied by 1000. – ALWAYS top and tail gsave and grestore so that the address space is NOT touched. – preserve currentpoint …. use gsave/grestore and opt Z – Ignore DOCUMENT in this one. – WIDOWS and ORPHANS can be up to 100 —————————————————————————— 7. PROGRAM DETAILS —————— This section covers the following programs : 7.1 form 7.2 ipformd 7.3 ipformat 7.4 ipformbl 7.5 ipformch 7.6 ipfsel 7.7 ipfprep 7.8 ipfchk ———————————– 7.1 form Manual interface to the data formatting package Allowable commands are : x go into test mode l look at log m look at log – for ‘form’ items c check crontab – for items about to go – c all for ALL of root’s crontab t look at the individual Parameter file in tables/text/text and show the contents p look at main form files in tables/form – PROCESS, SETTER, SETPAGE etc g go auto h help v version q quit ———————————– 7.2 ipformd This is the daemon for data formats. It uses a parameter file is used to route and process incoming files. This parameter defaults to tables/form/PROCESS. It first uses a selection table to decide what the job really is. As the list is top down, only the first valid selection is processed. The ‘jobname’ found is usually the name of a parameter file in tables/form/text. IPFORMD will automatically start IPFORMAT with parameters of the input file and the jobname/parameter file. However, optionally IPFORMD can be used to run a sequence of ‘jobs’ specified for a particular jobname. The syntax for the PROCESS file is : ; comment ; the following is a selection line (hdr field) = string [opt (tab) & (hdr) = string …] (tab) >jobname (nl) job: (jobname) (program to run) trace: (jobname) test: (jobname) To describe the Selection syntax in detail : (hdr field) = string [opt (tab) & (hdr) = string …] (tab) >jobname (nl) Each selection is on a single line. If necessary, multiple conditions can be specified with the ‘&’ to ‘and’ them. The operation equal, ‘=’, can also be NOT equal ‘!=’. Source Header fields (in SH) are preceeded by X, ie XC for category. A ‘*’ is used a wild card string; a ‘?’ is single wild card chr. To search for a string/chr embedded somewhere in a field, uses a ‘*’ before and after. If embedded spaces are needed in the string-to-be-searched, use an ‘*’. Note that the search string is case_insensitive. Both the selection file and the main file are scanned completely, so that one file may be sent to none, one or several destinations according to the same or different criteriae. For the ‘job’ parameter : – ‘$i’ refers to the input file name (Note $i is still the FIP System Variable ‘month’) – all queues and files are assumed to be under /fip/spool – Never assume however that the path environment has been setup, so we advise you specify full pathnames for the programs. – all ‘job’ lines MUST precede the selection – ie be above. – FIP System variables and Header fields can be accessed. – there can be one or many or very many job lines. – any program can be run – if a script/program returns an error, it is logged in the Item log and further processing stops. If a ‘job’ exists for a jobname, ipformd will NOT run ipformat but will run what is specified – which may be ipformat of course. The ‘trace’ parameter is used for setup, tuning and testing a new job. All it does is tell IPFORMD to log each line in the Item log. EG: trace:shares Trace MUST always be specified in the PROCESS file BEFORE the jobs (ie on a line nearer the top of the file) and all jobs must be before the selection lines. The ‘test’ parameter does the same as ‘trace’ BUT NONE of the programs are actually run. This allows you just to check syntax etc. EG: test:racecards Test MUST always be specified in the PROCESS file BEFORE the jobs (ie on a line nearer the top of the file) and all jobs must be before the selection lines. First examples – simple jobs : ; Sports copy to Tranmere Rovers FC. XC=S* & XK=*Football* >tranmere EP=TAR. >tarmac RD=*Broken_Hill* >bhp where TRANMERE, TARMAC and BHP are all various parameter files in tables/form/text Second example of a sequence of jobs. Here, if the file starts ‘borsen’, IPFORMAT is strted twice (sequencially, serially – ie NOT at the same time) using two parameter files in tables/form/text : WKENDSHARES and DAILYSHARES . ; shares : for both weekday and weekend job:shares /fip/bin/ipformat -p dailyshares -i $i job:shares /fip/bin/ipformat -p wkendshares -i $i ; Selection for job called ‘shares’ SN=borsen* > shares Third example shows how to sort, xchg and basically really screw around. ; Reformat, sort and generally destroy the horses … job:geges /bin/rm -f formsave/NAGS* job:geges /fip/bin/ipformat -p geges -i $i -D -S NAGS job:geges /fip/bin/ipxchg -1 formsave/NAGS -D geges -F -o formsave job:geges /bin/sort +0 -3 -o formsave/NAGS.done formsave/NAGS job:geges /bin/mv formsave/NAGS.done 2go/#SN:SN#DU:nagsdone ; Selection for job called ‘geges’ SU=RACEWIRE & SN=horse* > geges Fourth Example is where the ‘test’ parameter is added to the ‘geges’ job in the Third example (Remember the ‘test’ must be specified on a line at the top of the PROCESS file BEFORE any ‘jobs’ for the same jobname) : test:geges Going into the MUI, ip, and doing a ‘l’ to list the log (or ‘m’ to more) gives: Sat Mar 4 11:52:30 ipformd !i : Incoming File : geges : : geges Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /fip/bin/ipformat -p geges -i /fip/spool/form/geges -D -S NAGS Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /fip/bin/ipxchg -1 formsave/NAGS -D geges -F -o formsave Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /bin/sort +0 -3 -o formsave/NAGS.done formsave/NAGS Sat Mar 4 11:52:30 ipformd !f : Test/NotRun : /bin/mv formsave/NAGS.done 2go/#SN:geges#DU:nagsdone Other Points worth noting (ish) .. Break out – If either the input parameter -x or a header field FZBO is present, the input file is ‘broken apart’ into blocks, records and fields. The resultant file is called (dest)_(SN) in spool/formtest where dest is as above and SN is the filename. If an incoming file matches none of the tests, it is deleted and an error logged. In the selection file, remember to specify long names first. In the following example, job ‘sunrac2’ never gets processed as all files will be jobbed as ‘sunrac’ XK:RAC* >sunrac XK:RACING* >sunrac2 Input parameters are (all optional) : -c : name of a queue into which copies of all incoming files are made. default: no copies -f : file creep time default: 0 -i : queue to scan default: spool/format -l : do NOT log every incoming file/destination default: log -n : run the program at reduced priority default: nice 5 -p : processing file to use default: tables/form/PROCESS -s : run files serially (ie one after the other) default: parallel -t : scan time of directory default: 3 secs -T : Always trace jobs. This is the same as the ‘trace’ parameter used for setup, tuning and testing a new job. All it does is tell IPFORMD to log each line in the Item log. def: no -x : debugging ON – ALL incoming files will be ‘broken out’ in formtest parameter is ‘o’ctal, ‘d’ecimal or ‘h’ex. default: off -z : calm down time default: 5 secs To attempt to let ipformat finish one job before the next -v : display version number and exit. ———————————– 7.3 ipformat IPFORMAT processes its input file according to the rules given by the parameter file. The binary version of the parameter file is checked and if it needs updating, IPFORMAT will automatically run IPFORMBL beforehand. As to where the file goes, is saved etc depends on both input flags and the contents of parameter file. By default the resultant file is sent to a destination (DU header field) which is the same as the parameter file name. Hence the file is moved to spool/2go for IPWHEEL to distribute as it needs. Remember that you can add header fields to the output file using the data from the inbound file header or text or fixed data or system generated data. Input parameters are : Mandatory : -i : raw input file default: none -p : parameter table to use default: none Optional : -z : this is being run serially so delete the input file at the end default: first move it to x -Z : delete input file after use default: do NOT delete it normally ipformd will delete it after use. -o : output file if needed default: none Normally this is not required. It can be used to override the ‘name:’ parameter if it start with a ‘/’ -d : output destination default: same as param name -D : do NOT send the file anywhere default: see param table -f : output format for ipout default: FORM -c : source chrset if not ascii default: ascii -l : do NOT log default: log -q : input queue default: spool/form -s : save this file in the save area default: no -S : save this file in the save area default: no with the following name -w : do NOT overwrite default: overwrite -x : force breakout/debugging (same as FZBO header flag) default: no parameter can be ‘o’ctal, ‘d’ec or ‘h’ex (default is o) -v : print version no and exit ———————————– 7.4 ipformbl IPFORMBL takes the text parameter file and builds a binary so that IPFORMAT can run faster. Input parameters are : Mandatory : -i : parameter table to build default: none Optional : -o : name of output binary file default: text file name -l : do NOT log default: log -w : do NOT overwrite existing default: replace -v : print version number and exit ———————————– 7.5 ipformch This is the checker for data formats. It is started usually by crontab. It reads a parameter file which is used to describe the file name to look for and the process to run (if required), what logging is needed and how long to look. The parameter file is in tables/form/check and has the syntax : ; comment file: (name) nolog: found:(scriptname) nofile: (name) notfound:(scriptname) found:(scriptname) stop: wait: (seconds) totalwait:(seconds) filechk:(seconds) display:(seconds) errmsg: (error message) okmsg: (ok message) waitmsg:(still waiting message) log: (log this message) maxtime:(secs) where file: Path and name of file to check. If this starts with a ‘/’ it is considered to be the full pathname else it is under fip/spool. Trailing wild cards, ‘*’, are permitted. eg file:corrit/G1SI* nofile: Path and name of file to check : true if file is NOT FOUND. The following options are for both file and nofile : If the filename for file/nofile is ‘$f’ then the filename will be that passed on input for the -f switch. There are also 10 other variable parameters you can pass under -0 to -9 switch. These are defined as $0 in the path/filename. nolog: do NOT log if found/not found is TRUE default is to log if running script found: optional process to run if found. This is usually a script in /fip/local. The filename is passed as the 1st parameter. There is an example script in local called egformchk. notfound: as per found for non-existence. stop: optional – stop processing if this is NOT TRUE. Do not continue down the list. Normally if there is a wait it will wait nostop: optional – do NOT stop processing if NOT TRUE totwait: Total time to wait before we timeout in seconds. default is NONE or if parameter is NO – run once only. if parameter is 0 – wait indefinitely msgafter: optional interval before the first waiting messages. Note this will be after the first check after the msgafter time. default: 300 secs wait: optional interval between waiting messages default: 60 secs filechk: optional interval between checking for the file if waiting. default: 5 secs. min: 1 sec. max: 60 secs errmsg: error message string default: std error msg okmsg: message string if ok default: std ok msg waitmsg: message string if we are waiting default: std wait msg error: 1st param – output waiting message every x secs 2nd param – only start printing waiting message after x secs eg error:60:300 print wait msg every minute after 5 mins defaults: 1st param=60 2nd param= 60 maxtime: to wait for a single process before we message a blowout Note : this does need to be set for very long processes defaults: 60 secs/1 min There can be several files in the parameter file which are checked in sequence. There can be multiple wait/chk/errmess/okmess parameters which take effect for the next file/nofile. Input parameters are : -f : Path and name for the $f string which may be used for either ‘file’ or ‘nofile’ to check. If this starts with a ‘/’ it is considered to be the full pathname else it is under fip/spool. Note that the wildcard chr, ‘*’ can be specified but, depending on the shell used, may need to be quoted. default: none -0 .. -9 : As for -f/$f but allowing up to 10 more variables to be passed on input default: none -p : parameter file to run default: none -n : run the program at reduced priority default: nice 5 -c : do NOT run process – just check default: run if specified -v : display version number and exit. ———————————– 7.6 ipfsel This program is used to Select and sort lines from an incoming data files using a selection file. A parameter file is used to determine where the files should be sent and other parameters. This file is in tables/from/select and default to SELECTION. This may be overridden by the contents of the Fip Hdr field ‘DF’. Keywords for SELECTION parameter file are : ; comment line dest: One or more FIP destinations for the output file as per the USERS file. There is NO default and the parameter is advised ! before: String of FipSeq to stuff at the start of the file def:none after: String of FipSeq to stuff at the end of the file def:none script: Script to run on the new file before sending to the output que The first parameter will be the outputfile containing the correct data using a TMP filename. The second is the name requested. default: none nohdr: Flag to ask for no Fip Header on the output file. def:allow hdr nodata: Flag to ask for NO output file at all. If a script is flagged, a file is created for the script and is then zapped. default: allow data chrset: Name of the Source Character Set to use for IPXCHG. def:ascii extra: Extra Fip Header information to be added. default: none archive: Archive the data in IPWHEEL using this Logname. default: no filename: FipSeq for a replacement filename. default: as input doneq: Done queue for original input file default: file is deleted Keywords describing the data file : ——————————— You must define the key of the data line. This can be at a fixed position on the line (using fixedkey:) or the nth field (using fieldkey:). EITHER fieldkey: Field number of the sortkey or index field. OR fixedkey: Character position on the line of the key starting at 1. datasep: Field separator. default: any combination of spc or tab This is a single chr and maybe specified as a FipSeq. Eg: fieldsep:177 for an octal 177 keylen: Maximum length of key. sortdata: Sort parameters for incoming file as per the unix ‘sort’. default: -d +0 -1 for descending order, first field Note that if the field separator is NOT spc/tab you will also have to specify this. nosortdata: Do NOT sort the incoming data. befline: String of FipSeq to stuff at the start of the file def:none aftline: String of FipSeq to stuff at the end of the file def:none eoln: Line separator default: any combination of CR NL Keywords describing the selection file : ————————————– selfile: Full Path and Filename of the selection file – NO DEFAULT. sortsel: Sort parameters for selection file as per the unix ‘sort’. default: -d +0 -1 for descending order, first field Note that if the field separator is NOT spc/tab you will also have to specify this. nosortsel: Do NOT sort the selection file before use. selsep: Field separator. default: any combination of spc or tab This is a single chr and maybe specified as a FipSeq. Eg: fieldsep:177 for an octal 177 The actual selection file itself (as specified in ‘selfile’ parameter), obeys the following Syntax: ; comment line with preceeding semicolon (eoln) (key) (spc/tabs) (field2) (spc/tabs) (field3) .. etc (eoln) Keywords describing how to match/merge ————————————– Which file is sorted in the required output sequence. major:data major:sel – this is the default Combination of match and merge defines what is output : 1. Only data items that match the selection file match: 2. Data items that do NOT match the selection nomatch: 3. Merge complete data and selection files merge: default is selection record first or merge:selfirst or merge:datafirst or merge:onmatch 4. All Selection file plus matched items of the data file match: merge: The default, if neither match or merge is stated is option 1 – match only. Where the records in the Selection file is in a different format from that of the Data file, you can reformat the output of the Selection data with : seldata: (Fip Seq data) where Hdr Fields DK is the key and DD is the data part of the Selection. eg: seldata:XX,DK\zDD will give selection record 123456 Sample Data for Selection outgoing data XX,123456zSample Data for Selection NL A NL, or whatever ‘eoln’ is set to, is added automatically to the line. Plus option, combie and fixed Pls never allow any of your scripts to loop as ipfsel waits for completion before continung ! Input Parameters are : EITHER -i : input queue default: spool/formselect If this does NOT start with a ‘/’, it is assumed under spool. OR -1 : name of a single file (complete path pls) default: spooled queue This will run the program once for that file only -o : output or done queue default: file is deleted If this does NOT start with a ‘/’, it is assumed under spool. -z : default parameter file in tables/form/select. def: SELECTION -w : file wait for files arriving across a network. def: 0 secs -u : owner if not that of the logon default: logon at start -l : do NOT log files in default: log -v : print version number and exit ———————————– 7.7 ipfprep This program prepares incoming data to tweak it before IPFORMAT is run against it. Notes – the NAME of a ‘fmt’ must be unique but there can be several ‘key’s for that fmt which may be testing different things – and creating newrecords. Each FMT is checked sequentially – ie top down in the parameter file. If a single key matches, then the parameter file is checked for more key lines with the same name. Processing stops on the next ‘fmt’ record. This means all the keys for a fmt should follow immediately after. KEY – Syntax : key:(name) chk:(fldx) tst:(fldx)=(string) tst:(fldn)#(string) addkey:(posn):(len) delete: newrec:(syntax for newrec) key:(name) same name as FMT previously specified chk:3 chk field 3 is of the format is the right length or if field 3 is varibale length, has 1 or more. tst:7=ABC Does field 7 equal ABC tst:9#” ” Does field 9 NOT equal 4 spaces Note that tst is case sensitive. addkey:1:3 Add a key of the name to the record at 1 delete: Delete this record newrec:PAG0003 Add a record before with data PAG0003 (recsep) DO NOT ADD THE END OF LINE it is done automatically FMT – Syntax : and a single line , field marked [] are optional fmt:(name) (typ1):(len1)[:(startpos1):(startstring1):(endstr1)] … (typn):(lenn) where name is the name of the record format (case insensitive) tabs or spaces are separators between fields colons are separators between parms for the same field length or size of field is 0 for variable size type is a-alphabetic, u-uppercase, l-lowercase n-number, p-printable, s-space(or tab, CR, NL or FF) b-binary (ie anything), x-alphanumeric, c-control (ie = 0177), z-anpa hdr field (ie alnum plus non-quad/format punct. t-punctuation startposn is the starting posn in the record from posn1 startstring is the starting string for the field endstring is the end string for the field RECKEY Check the type and size of the key Syntax : reckey: : (length) : (type) where length or size of key is 0 for no size check type is a-alphabetic, u-uppercase, l-lowercase n-number, p-printable, s-space(or tab, CR, NL or FF) b-binary (ie anything), x-alphanumeric, c-control (ie = 0177), z-anpa hdr field (ie alnum plus non-quad/format punct. t-punctuation Input parameters are : Optional : -i : input queue default: spool/fprep -o : output queue default: spool/form -f : file creep time for files default: 0 arriving across the network – normally make it 5 secs -t : scan time of directory default: 5 secs -l : do NOT log every incoming file default: log -z : use the basename of the file for the parameter file if there is .. .. no FP FIP header field default: use FP only -v : display version number and exit. Example : ; ; type of file is text filtyp:t ; each record is separated by a CR CR NL recsep:rrn ; strip all cr nl combinations including blank lines ; this boils the ‘recsep’ into a single CR stripeol:y ; There are no reclen and reckey commands – comment them out ;;reclen:121 ;;reckey:3:u ; change the wild card chr to a dollar (normally it is a star ‘*’) wild:$ ; blank lines – to be stripped – put first to catch all of them fmt:BLANK s:0 b:0 key:BLANK tst:2=”” delete ; text 8 col lines fmt:TX8 s:1 p:38 p:1 key:TX8 tst:3=”-” addkey:1:TX8 ; format of the trailer records ; (spc) AJ (4 num) (2 spc) (10 prinatable repeated 4 times) fmt:TRAILER s:1 u:2 n:4 s:2 p:28 p:28 p:28 p:28 ; a trailer record needs to be deleted – and subrecords created from it key:TRAILER chk:1 tst:2=AJ chk:3 chk:4 delete key:TRAILER chk:1 tst:2=AJ chk:3 chk:4 tst:5#” ” newrec:E{5} key:TRAILER chk:1 tst:2=AJ chk:3 chk:4 tst:6#” ” newrec:E{6} key:TRAILER chk:1 tst:2=AJ chk:3 chk:4 tst:7#” ” newrec:E{7} key:TRAILER chk:1 tst:2=AJ chk:3 chk:4 tst:8#” ” newrec:E{8} ; Main Heading – check the syntax of fields 1 and 3 and make sure f2 = ‘1’ fmt:HEAD S:8 N:1 S:20 P:14 P:12:44 p:4:56 key:HEAD chk:1 tst:2=1 chk:3 addkey:1:HD1 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”CITROEN” newrec:PAG0003 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”PEUGEOT” newrec:PAG0024 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”RENAULT” newrec:PAG0104 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”TALBOT” newrec:PAG0105 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”FORD” newrec:PAG0106 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”MAZDA” newrec:PAG0107 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”LADA” newrec:PAG0108 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”ROVER” newrec:PAG0109 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”NISSAN” newrec:PAG0141 key:HEAD chk:1 tst:2=1 chk:3 tst:4=”BMW” newrec:PAG0144 ; 1st Col headers fmt:COL n:0 s:39 p:10 p:10 p:10 p:10 p:10 p:10 p:10 p:10 key:COL tst:1#0 tst:3#” ” addkey:1:COL ; 2nd Col headers fmt:CL2 s:40 p:10 p:10 p:10 p:10 p:10 p:10 p:10 p:10 key:CL2 chk:1 tst:2#” ” addkey:1:CL2 ; Section Head fmt:SECTION s:1 u:1 t:1 p:0 key:SECTION chk:1 chk:2 tst:3=”-” addkey:1:SCT ; SubSection Head fmt:SUBSECT s:1 u:1 n:0 t:1 p:0 key:SUBSECT chk:1 chk:2 chk:3 tst:4=”-” addkey:1:SUB ; setup the default keyname for text files defkey:TXT addkey:1:TX8 ———————————– 7.8 ipfchk This program scans a directory and checks the incoming files for CRC errors. If found the lines are stuffed into an error file and sent to an errdst while the resultant file is flagged with the HE field as ERROR. The HC header field contains the total number of records in the incoming file. Input parameters are : Mandatory : -i : input queue default: none -o : output queue default: none Optional : -d : replacement destinations default: current DU This replaces the DU field. -t : scan time of directory default: 5 secs -l : log every incoming file default: do not log -P : preserve incoming file default: strip CRC -v : display version number and exit. —————————————————————————— –> © FingerPost Ltd. 1996