The Guide to the FIP Data Formatting Module

This guide covers the following topics :
1. How to go about setting up formatting a new feed.
2. Using 'form', the df User Interface, to test and tune.
3. Copy Flow.
4. Reference Guide to the Data Format Text parameter file.
5. FipSeq.
6. PostScript Driver - ipsetter.
7. Program Switches and other details.
7.1 form
7.2 ipformd
7.3 ipformat
7.4 ipformbl
7.5 ipformch
7.6 ipfsel
7.7 ipfprep
7.8 ipfchk

1. HOW TO FORMAT !

Time for Some Golden Rules :

Now the Steps :

1. Sort out Copy Flow for both testing and live running.

    Do you need an 'xchg' before formatting ? It is always a good idea to rationalise the input file as far as possible - running a preliminary xchg may help.

    Do you need an 'xchg' after formatting ? Again there may be some tiying up needed once the file is processed that is better suited to ipxchg than to ipformat.

    What about special processing like needing to 'sort' output ? Do you need to sort the copy inbound ? Do you need to resort it outbound ? Do you need to select specific records ?

    Does it need mutiple datafomats - occasionally you get a file that can be sorted one way, then formatted to say pick up items between consecutive records, then sorted another way and reformatted for output.

2. Map out what you want to format.

    It is usually easier to make notes BEFORE starting to test.

    Just a few notes is all that are required.

3. Setup and start testing the main Format.

    Often is easier to copy an existing Text file from form/text, use it as a template and rework it to meet the new requirements.

    There are two ways of testing copy 'offline'. At the Unix command line you can use the 'form' program to set yourself up a testing environment. ALternatively you can use the form module in W4.

4. Work out the other parameters in the Copy Flow.

    See the relevant section in this guide for an example of Copy Flow.

5. Release

    Dual running over a number of days is always advised.

6. Check
    It is always worth going back to the users and seeing how things can be improved for them. There may be thigs you have missed, or just things that they now realise you can do, that they have never thought to ask for before.
 
 
 
Pointers

- If you need to sort the output file : use the 'job' parameter in the text/PROCESS file (see sections Copy Flow and 'ipformd' in the Program Section).

- if you need to select only certain records from the input file matched against a standing list of keys, use 'ipfsel' (see description in the Program Section).

- if you need to track several input files have arrived over a period of time, flag if they are missing and process if they are there (for instance, a number of postscript files for an OPI), use 'ipformch', (see description in the Program Section).

- if the input file(s) are very dirty and inconsistent, clean them up beforehand with 'ipfprep' (see description in the Program Section).

2a. USING 'FORM' TO TEST AND TUNE

There are two interfaces to test & tune your dataformats ... form is a fast command and basic line driven interface that runs under Unix. There is also a web based interface available through W4.

FORM, the user interface for the Data Formatting module, can be used to test and tune a process 'internally' - without actually sending the output file anywhere nor deleting the input.

It can also be used to run an XCHG before and/or after the format.

Several tests can be run and their settings are held in a series of Settings files in tables/form/test.

Note at the moment 'form' will NOT scan the PROCESS file and run any 'jobs' (qv) as it is used purely for getting the parameter file in form/text 100% correct.

Firstly you choose an existing Settings file - if there is one.

Then you need to get the input file into the queue in spool you are using for testing.

If it is the first time you will need to 'mkdir' your test area. Generally one has been setup called 'spool/test'.

If the input file is on diskette or on another system, just copy it over. If the file came from a wire service or dialup and is in an FIP Archive log, you can add a destination in the sys/USERS file and resend to it. Generally, if FIippies have been onsite, there is a default one already setup called 'test' which goes through 'ipedsys' to cleanup the filename BUT does not go through any 'xchg' :

	sys/USERS : test=   DP:com2  DQ:2edsys  EQ:test SC:no DC:no DF:testing

This uses edsys/TESTING to place the file in spool/test.

To get into the test bit of 'form', type 'x' at the form prompt. This will reveal firstly which Settings files are available which you can choose from.

The current settings are displayed before the list of options available.

All options are a single character (case-insensitive) followed by 'enter'.

When changing settings, a '*' can be used to list all the files in the relevant directory. Case is only sensitive for filenames.

root @ com_server2/ > form
form-com_server2:x
** Test FORM - Looking for Settings files
-- List of Files in the Queue : /fip/tables/form/test
total 2
-rw-rw-rw-  1 root	   93 Mar 17 10:53 RESRAC
-rw-rw-rw-  1 root	   94 Mar 17 19:52 RESTEST
** Hit Return to continue .. 

** Test FORM - Choose a Settings files (or return to ignore) :restest
** Test FORM - Existing settings are :
	Working Queue		: /fip/spool/test
	Input file		: rem737
	TEXT Param file		: RESTEST
	XCHG before		: PA2FORM
	XCHG after	 	: PA2ATEX
	DU for Live Tests	: sunres


** Options are :
	Change Settings		: C
	List the Working Queue	: L
	Look at the Input File	: I
	 ..  .. the Output File	: O
	 ..  .. the BreakOut	: T
	Edit TEXT file		: E
	 ..  Xchg before file	: B
	 ..  Xchg after file	: A
	 ..  PROCESS for jobs	: P
	Help			: H
	Run a Test		: R
	Send Live Test		: S
	Quit			: Q :

In (slightly) more detail, the options are :
Change Settings- change any of the 4 inputs
Listing the working queue- 'ls -l' of the working queue
Look at the Input file- More, Dump, Edit, Tail the input which MUST BE in the working queue
Look at the Output file- More, Dump, Edit, Tail the output which is hidden in the temp queue.
Look at the BreakOut file- More, Dump, Edit, Tail the breakout file which is that created by IPFORMAT showing how the input is split into fields and records
Edit the TEXT parameter file- IPVI the parameter file in form/text
Edit the Xchg before file- IPVI the (optional) xchg file to be used BEFORE the format.
Edit the Xchg after file- IPVI the (optional) xchg file to be used AFTER the format.
Help - More this file !
Run a Test- run IPFORMAT and the look at the output file.
Send Live Test- copy the test file and send to the DU (destination) specified.
Quit- No documentation available on this

In more more detail .....

-- Change Settings :
** Change Settings - Existing settings are :
	Working Queue   	: /fip/spool/test
	Input file      	: rem727
	TEXT Param file 	: RESTEST
	XCHG before     	: PA2FORM
	XCHG after      	: PA2ATEX
	DU for Live Tests       : sunres

  (At each prompt, Use '*' for list of files)
	Change Working Queue    : W
	Change Input file       : I
	Change TEXT Param file  : T
	Change XCHG before      : B
	Change XCHG after       : A
	Change DU for Live Tests: D
	List the Working Queue  : L
	Quit			: Q or enter :

Note that a '*' 'enter' at any of the Change file lines will list the relevant queue before reprompting.

Type '-none-' (or in fact just '-') to set an optional field to '-none-'.

Note that any reference to 'jobs' should be ignored in this version - I do.

Looking at any of the files :

** Look at file : rem727 : Options are :
	More    : M or enter	- This does a 'cat -v' before to show ControlChr
	Dump    : D		- Essentially an 'od -ab'
	Edit    : E		- Using 'vi'
	Tail    : T		- Last 20 lines of a 'cat -v'
	Quit    : Q :

Running a Test :

- A simple format with no xchgs -

** Running please wait ... 
	/fip/bin/ipformat -i /fip/spool/hoswrk/rem727 -p ATEXHORSES -o /fip/xFORM.XX.DEFAULT.XX -s -D -l -xo 

** Ok Done    Hit Return to continue ..

This will now go directly into the Looking at Output file menu.

- A more complicated run with an xchg before and after -

** Running please wait ... 
	Saving the input in /fip/x/FORM_rem727
	ipxchg -1 /fip/x/FORM_rem727 -D PA2FORM -o /fip/x -F 
	** Ok : Xchg Before finished 
	/fip/bin/ipformat -i /fip/x/FORM_rem727 -p RESTEST -o /fip/x/FORM.XX.DEFAULT.XX -s -D -l -xo 
	** Ok : Format finished 
	ipxchg -1 /fip/x/FORM.XX.DEFAULT.XX -D PA2ATEX -o /fip/x -F 
	** Ok : Xchg After finished 
  Hit Return to continue ..

Again this will now go directly into the Looking at Output file menu.

Note that 'form' allows you to save the setting you have chosen in a Settings file, so that the next time you go into form it should display the settings from the last time.

Sending a Live Test :

This will copy the input test file and send it to the Destination (DU) called the 'Live DU' above.

It then tails the Item Log of that system underthe assumption that 'ipformat' will give a message when the file is through. - Cntrl C to Stop and return to the main Form Test prompt.

Prerequisties for sending a Live test (if that is possible) :

- In form, you need to specify :
- an input file
- a working queue
- a destination (DU) which MUST be in the USERS file.
- Also remember to check your SC/DCs for your xchg's.
- Also remember to add the selection lines in tables/form/PROCESS.

2a. USING THE WEB BASED VERSION OF 'FORM' TO TEST AND TUNE

The web interface also allows you to run offline tests of your dataformat. It is similar to form, but gives you slightly nicer views of the file (for instance hidden characters, high characters and line endings are all high-lit in red.)

i) You will need to have the form option enabled in your W4 logon

This is normally done by adding the line

;;;;; Data Formatting
options:Data Formatting:/fip-pages/form/dftest.html:_blank

to either your W4 logon, or, more usually, to a template called by your W4 logon.

ii) This should be a fairly intuitive interface (feedback welcomed though)

You first select or create a test ..... a test is a definition of how you are planning to test a file, so it will define the input file, any xchgs you plan to run against it, and the format you plan to run.

 

The top section of the left hand pane will describe the parameters you choose, and the links in the bottom selection can be used to select a sequence of xchgs, sorts and formats.

 

Having selected the parameter files to run

3. COPY FLOW

How do you route copy into, and out of, the data formatting module ?

- Using the normal FIP routings !

An simple example is the Horse Racing Cards which arrive via dialup modem from the course administrators :
StageProgramInput QueueParameter tables and Remarks
InputVWIRE-wire/RM
RoutingIPROUTE2broutedroute/RM
Actual routing line is :
1       z="*HORSE RACING*"      +horses
ie make an 2nd copy of the file and send to destination 'horses'
Dist'nIPWHEEL2gosys/USERS 
horses= DP:localhost DQ:form SC:NO DC:NO
ie process on whichever system it arrived on, send the file directly to spool/form with NO chr translation (ipxchg).
Format SelectionIPFORMDformform/PROCESS
Selection of actual Format prarameter file 
; Selection criteria for Horses
DU=horses       >atexhorses
ie if the DU field is exactly equal to 'horses' do the 'atexhorses' job.
Actual ProcessingIPFORMAT-form/text/ATEXHORSES
The original file is deleted at the end while the new, formatted file is sent back to ipwheel with a new destination (DU) of 'atexhorses'.
Dist'n of outputIPWHEEL2gosys/USERS 
atexhorses= DP:atex1 DQ:junk-wir SC:HORSES DC:ATEX DF:DATAFORM
ie send file to 2atex queue via 'xchg'.
Chr XchgIPXCHGxchgxchg/HORSES2ATEX
Clean up the data
Send to AtexIPGTWY2atexgateway/DATAFORM
Send it to junk-wir

EXAMPLE 2 : STOCKS : More complicated examples are for the various Stocks.

The large Hong Kong tables follow the same path as 'horses' except that the names of the parameter files are obviously different.


Selection in PROCESS is on Filename only
Format text fileform/text/HKSTOCKS
output xchgxchg/HKSTOCKS

For the Regional Stocks, there are two different input formats (plus Manilla which is different but simpler) which need to be used to create almost the same output.

The Input variations are :
First type :
RICDISPLAY NAMELASTTODAY'S HIGHTODAY'S LOWHISTORIC CLOSE
AAH.ASABN-AMRO HLDGS59.96059.359.5
ACHN.ASACF HOLDING35.735.735.535.5
AEGN.ASAEGON NV112.411312.2112.8

Second Type :
*FASTCLOSERIC9503011900KLS
SECURITYDATEHIGHLOWLAST TRADEPREVIOUS CLOSE
AYER HITAM TIN STK9503014.1003.9604.1003.960
AYER HITAM PLANT STK95030114.30014.30014.30013.500
ACIDCHEM STK9503016.3506.1006.3506.000

Manilla Stocks :
NAMECLOSEHIGHLOWPRECLOSE
A Soriano3.23.23.153.2
A Soriano B3.23.23.23.15

There are two outputs which are identical EXCEPT London and New York prices are quoted in fractions NOT decimals and so they require a different Character Xchg.

The processing is more complicated than for 'atexhorses' as some of the names of the Stocks are changed AND the output is sorted alphabetically. For example 'INTL BUS MACHINE' needs to be 'IBM', but as 'IBM' it will start the 'I's and NOT appear after 'Inland Steel' as in the input feed.

So we use the 'job:' keyword in the PROCESS file to get IPFORMD to run a series of jobs rather than just start IPFORMAT as in the example above.

Look at the processing for London and New York :

While the selection remains similar to 'atexhorses' :

SN=LON.TXT	>fracstocks	; London
we select on the filename this time.

But At the top of the PROCESS file there are a series of parameters for the job 'fracstocks' :

;
; Job sequence for London and New York - Fractions
job:fracstocks  /bin/rm -f formsave/FRACSTOCKS*
job:fracstocks  /fip/bin/ipformat -p fracstocks -i $i -D -S FRACSTOCKS
job:fracstocks  /fip/bin/ipxchg -1 formsave/FRACSTOCKS -D fracstocks -F -o formsave
job:fracstocks  /bin/sort +0 -3 -o formsave/FRACSTOCKS.s formsave/FRACSTOCKS
job:fracstocks  /bin/mv formsave/FRACSTOCKS.s 2go/#SN:\SN#DU:atexstocks

So the copy flow for London Stocks is that the FORMAT stage is replaced by ALL these job lines in sequential order :

- Remove any files starting FRACSTOCKS in formsave
- Run IPFORMAT with the input file and parameter file FRACSTOCKS leaving the output in formsave/FRACSTOCKS
- Run IPXCHG once on formsave/FRACSTOCKS overwriting the input with the xchged file. This is to get the correct StockNames eg:
x/Hong Kong Telecm/HK Telecom
- Sort formssave/FRACSTOCKS on the first three words, creating an output file called formsave/FRACSTOCKS.s
- Move formsave/FRACSTOCKS.s to 2go with destination (DU) 'atexstocks' and preserving the original filename (SN)

Please refer to the documentation on IPFORMD in the programs section for more information on jobs.

One further not on the Stocks is that all the output files pass through the STOCKS2ATEX xchg.

This is used to add column headers before certain Stock names. eg :

x/Northern Elec/\n{M1Stock\rClose\rHigh\rLow\rPrev\r\n{M0Northern Elec

4. PARAMETER FILE REFERENCE GUIDE

This is the reference and hints section describing the main Parameter file used for processing.

Overview

Part 1. Define what the input file looks like plus a general section covering fixed information.

define the type of file.
define the record and field (if any) separators or sizes.
define the record and field keytypes (if any).
define all 'sets' - short forms.
define all partial field structures - 'partial'.
define all localised searches - 'match'.
define strings placed before and/or after the data.
define alternate names and headers.
Keywords:
filtyp: recsep: reckey: recsiz: fldsep: fldkey: fldsiz:
stripeol: number: startkey: keycasesens: wild: wchr: set:
include: calc: fraction: base: date: style: partial:
match: hdr: nohdr: name: chrset: before: after:

Part 2. Output Section

Overview

Record Processing Lines

This is flagged as beginning with the 'output:' keyword. It describes what processing should be done for each input record.output

 r=X
lines to describe the output.

Record lines can have system variables, input fields, tests, builtin formatting etc.

Each one of the keywords is described below.

Each line is a self-contained item ending with a NewLine (Unix) or CR LF (PC) or CR (Mac). The text parameter file can be edited by any word-processor on any (normal) platform AS LONG AS the end result is a pure, raw ascii file with no Presentation or fancy graphics embedded.

Comments are the usual semi-colon in front.

	; comment

Reserved Names

The list of keywords is the list provided above plus a series of tests and builtins described below.

Note that it is possible - but not advised - to override some of these keywords. So these names should be considered reserved. In addition a few other names have been reserved for future use :

blksep:
blkkey:
blksiz:
The Structure of the Parameter File

The text file is split into 2 main parts as described above. The OUTPUT section must the second and is marked by the keyword 'output:' on a line on its own.

By common consent, the first part starts with the definitions of the file, records and fields but this is NOT strictly necessary. The advice is - do whatever is easiest for you.

Comments and the binary version of the Parameter file

Comments are lines STARTING with a semi-colon. You can have millions of comment lines and, except for the first run, they will have no effect on run-time speeds.

This is because 'ipformat' uses a compact, binary version of the Text Parameter file which is built automatically by 'ipformbl' every time you modify the Text version.

The only time you need touch the binary versions (in tables/form/bin) is that they should be deleted during every software upgrade of the DF module.

Comments are encouraged !

The Processing Loop is...

The actual processing cycle is :

For each input file hitting queue spool/form :

- 'ipformd' will select the correct Parameter file using form/PROCESS
Normally this will mean starting 'ipformat' with the chosen file

Once started by 'ipformd', 'ipformat' will go through the following steps :

- Preprocessing
- Create a Fip style header unless not required with 'nohdr'
This can be the standard one or can have extra fields added using 'hdr'
- Add Data at the beginning of the output file if the 'before' keyword has been specified.
- Processing input file
- Split the input file into records and for each record :
- Start at the 'output' section of the Parameter file
- Check each record specification line. If it is for that record type, process it
- Loop around for the next record
- Postprocessing
- Add Data at the end of the output file after the processed data if the 'after' keyword has been specified.
- Create a Fip filename
This can be the standard one or can be replaced if the 'name' keyword is specified.
- Send the file spool/2go for 'ipwheel' to distribute usually via 'ipxchg'.

That's it !

Syntax of Each Keyword

In the Output section - Record processing lines

The actual data is processed and output using the Record processing lines.

The Syntax of each line is that the first bit specifies which record type or key the rest of the line applies to :

		r=(key)		(output)
For example :
	r=abc	"Fried fish starts " s1 spc f4 " and the rest ..."
There can be multiple lines for the same record type. The following two lines will give the same result as the one above :
		r=abc	"Fried fish starts " s1 
		r=abc	spc f4 " and the rest ..."
For lines where you want to process for all EXCEPT a particular record type/key, use teh syntax 'r#' :
		r#35	"Nobody wants record 35s !"

How to specify you want to use, format and/or output zones ?
records r3or r="abc" if not numeric
fieldsf99or f=Z if not numeric
partial fieldsp22- always numeric
save zoness1 - always numeric
flagsx199 - always numeric
calculationsc4- always numeric
countersz4- always numeric
blocksb77- always numeric
set name---specify name as in the 'set'
fixed text---" some fixed text "

A '*' can be used in certain cases to signify 'ALL' zones ie
clrflag=*clear all flags.
f*output all fields from this record.

Use double quotes for alphabetic keys and those with embedded spaces.

You should try not to use 'sets', 'partial's or 'match's with names in the form 'z999' where z is one of the single letters above and 999 is a number in the rangle 1-999.

Note that blocks are 'super records' but should be ignored for now.

Note that case is IGNORED in keys in the current version .

When 'ipformat' finds a name in the record processing line, it does the following sequence :

- check to see if it is an already specified constant 'set' name.
- if not, is it a zone - record, field, partial or block (eg p44, f3)
- if not, is it a builtin command (see below)
- if not, is it a save zone, flag or counter (eg s1, x33, c7)
- otherwise it is considered some FipSeq string and saved as such.

Spaces, End-Of-Lines and Double Quotes in the output

One common failing when putting together a new parameter file is to completely forget about spaces (or other separators) and end-of-lines (CR or NL or CR NL or whatever) in the output file.

The point is - you have to specify them as NOTHING is implicit in the output file. There is no hidden magic which suddenly realises that you want an end-of-line when you need it. You have to state where and when you want them.

Generally this will be done by either putting them as constant/'set's or specifying them in the record processing line. The following are exactly the same : Either

	set	spc	\s
	set	ql	\n
	output:
	r="BIG"		f5 spc f3 spc f99 spc f5 ql
Or
	output:
	r="BIG"		f5 \s f3 " " p99 \s f5 \n
As you can specify a space as either '\s' or in double quotes, to output a double quote character, you need to specify it as an number : \000.

Builtins

There are a number of builtin conversion routines for formatting zones - records, fields, save areas etc.

These are called by placing the name of the conversion BEFORE the name of the zone eg :

		zapspcextra p5
which means :
'zap all leading, trailing and multiple spaces from partial field 5'

A single zone can be subject to several builtins :

zappunc zapspc caps f=Z
which means :
take field "Z" and zap all punctuation and zap all spaces and force uppercase before outputting.
Builtins for case conversion :
caps - force zone uppercase
lwrcase - force zone lowercase
idicase - force zone idiot upper and lowercase
upper1 - force first letter of every word uppercase
initial- only display first letter of each word followed by a full stop
Builtins for removing spaces:
zapspc - remove all spaces from zone
zapspcextra - remove all leading, trailing and multiple spaces from zone
zapspclead - remove all leading spaces from zone
zapspctrail - remove all trailing spaces from zone
Builtins for removing punctuation:
zappunc- remove all punctuation from zone
zappuncextra - remove all leading, trailing and multiple punctuation from zone
zappunclead - remove all leading punctuation from zone
zappunctrail - remove all trailing punctuation from zone
Builtins for Counters:
setctr- set a counter
incctr- add one to a counter
decctr- subtract one from a counter
clrctr- clear a counter or set it to zero
Builtins for Calculations:
savnum- save a printable number in a variable
savbyte- save a single byte in a variable
savint- save a binary integer (2 bytes) in a variable
savswint- save a binary integer (2 bytes swapped) in a variable
savlong- save a binary long (4 bytes) in a variable
savswlong- save a binary long (4 bytes swapped) in a variable
Miscellaneous:
strlen- returns the length of the string which can be output or saved or tested
zapleadzero- removes leading zeros from zone
zapctl- remove all control characters from zone
incfile - include standing file at this point
	r=99	incfile /home/standing/ s4
newfile - finish this file, send it and start another
	r=abc	newfile
if any more information is specified AFTER the 'newfile' on the record processing line, it will be added to the FIP Hdr unless 'nohdr' has been specified. eg:
	r=abd	newfile	#DF:newform#QQ:\$Z
log- log message in the Item Log
continue- ignore all other tests for this record and continue with the next data record
stop!- stop processing now. If there is an 'after' section it is done before the program finishes. (please note the exclamation mark !)
reckey- output the actual record key. This is useful where wild cards are used for all records but you still need to output what the key was.

Tests

There is a further selection of tests which can be made one zones inside the date.

These enable you to select even finer some processing depending on actual data. If and ONLY if the test is true is the rest of the line continued with.

Syntax for Tests

	(ifxxx) (first string) (second string if required)
where strings can be fields, partials, saves or fixed text
Actual tests can be :
ifprv/ifnprv - test previous record type/key or not
ifeq/ifne - test if 2 zones are equal or not
ifgt/iflt - test if a zone is greater than another or not
ifflag/ifnflag - test if a flag is ON or OFF
ifnul/ifnnul - test if a zone is empty or not
ifspc/ifnspc - test if a zone only contains spaces or not
ifalpha- test if a zone only contains letters a-Z or not
ifnum- test if a zone only contains number/digits 0-9 or not
ifcon/ifncon- test if a string is (not) found within another
ifpunct- test if a zone only contains punctuation or not

Note that sequence is important for comparing two fields that may be different lengths as ifeq will be true if the first field is complete ie : 

		1st=AAA		2nd=AAABC	will be true 
		1st=AAABC	2nd=AAA		will be false
Example 1 :
	r=24	ifprv r=35	"Last record was type 35 and this is 24"
Only if the previous record type was "35" will the string be output

Example 2 :

	r=24	 f3 ifnul f3 " _ " x99
For record type 24, output field 3 and if there was nothing in it, output a (spc) (dash) (spc). Flag 99 will also be set if there was nothing there.

Example 3 : When using numeric data, please ensure that all extraneous characters are stripped from the zone before the test. In particular strip commas, plus signs, currency symbols etc. For example, if field 7 has data like p9300.0007 and save field 9 has 10,000 compare the two by :

	match:mnop	/p//
	match:mnocomma	/,//
	output:
	r=99	ifgt mnop mnocomma f7 mnop mnocomma s9      "Field 7 >  Save 9"

Using Flags

Flags are a really useful means for deciding type of processing to do - or NOT to do.
Commands for setting, clearing and testing flags are :
- To set a flag: x999 where 999 is the flag number
- To clear a flag: clrflag=999
- To clear all flags: clrflag=*
- To test a flag is ON: ifflag x3 (rest of the commands on line are done ONLY if true)
- To test a flag is OFF: ifnflag x5 (rest of the commands on line are done ONLY if false)

For example, let's use flag 3 to test if record type 'abc' has Richard, Helen or George in the first field. Print out 'New name is (name) (newline)' if it does :

	r=abc	clrflag=3 
	r=abc	ifeq "Richard" f1	x3 
	r=abc	ifeq "Helen" f1		x3 
	r=abc	ifeq "George" f1	x3 
	r=abc	ifflag x3		"New name is " f1 nl

Save areas

Save areas may be used to store strings - either in their original state or after conversion/formatting by other built-ins. The maximum save number is 299.
Commands for setting, clearing and testing save areas are :
- To output a save area: s299 -- where 299 is the save number
- To clear a save area: clrsave=299
- To clear all saves: clrsave=*
- To save data in a save area: save=299 (string) 
	eg	save=1	f3 
		save=5	caps f7
save the contents of field 7 in save area 5 AFTER forcing to Uppercase
- To append data to a save area: savcat=88 (string) 
	eg	save=77 "ABC"
save zone 77 holds ABC 
		savcat=77 "DEF"
save zone 77 now holds ABCDEF

Save areas may be used in the normal 'if' tests, eg :

	ifeq "AAA" s1	x88
if the contents of save area 1 starts "AAA., set flag 88 ON

Using Counters

Counters are integers (ie proper numbers with no decimals or fractions in the range -32000 to +32000.

They are signalled by 'zX' where X is a number.

They can be used to count the number of occurences of a record or field or even types of data and act accordingly.

All counters are set to zero when the program starts and by using the builtins :

incctr
decctr
setctr
clrctr
in the Record processing lines, you can manipulate them.

For example, to add some random markup every 10th line of a record type AB using counter 26 :

	r="AB"	incctr=26	ifeq 10 z26	clrctr=26	"[pt9][font99]"
ie : For all records type AB, add 1 to counter 26, then test if ctr 26 is equal to 10; if so reset ctr 26 back to zero and output string '[pt9][font99]'.
The syntax for 'setctr' is
setctr=99 345- set ctr 99 to a fixed number 345
setctr=297 p3- set ctr 297 to the contents of partial field 3.

In the second example, if the p3 is NOT a number, ctr 297 is set to zero. Also if p3 is a decimal number like '123.456', only the main number is saved.

Using Calculations

Calculations are defined in the first part of the parameter file and used in the record processing part :

For example :

	calc:mktcap	c1*c2
	output:
	r=BC	savnum=1 f5	savnum=2 f7	mktcap
In this example we define 'mktcap' to be variables 1 and 2 multiplied together. Then in the output section, for record type BC. field 5 is saved in variable 1 and field 7 in variable 2 before we do the calculation and output the result.

A quick word about BINARY numbers.

Normally fields will hold printable data - such as in the example above - and we use the builtin 'savnum' to take that number for use in the calculation(s).

However some data is already in a binary form. Use builtins 'savbyte, savint, savswint, savlong and savswlong' to load these numbers. Often these will be derived from a partial field using the 'b' for binary field type. eg:

	partial:bindata	b:2 b:4 b:2 b:4
What is a swapped integer or long ? Some computers - like the PDP-11 and most Intel 16+ bit chips - hold the data in reverse byte mode.

- So if the data has been generated on a SPARC OR rs6000 or a Mac the data is 'normal' - use savint or savlong.

- While data from PDP-11s or Intel based PCs could well need to be swapped.

Loading Variables :

Save a printable zone as a number variable - use 'savnum'
savnum=5 p4
- save the contents of p4 as a number. So if p4 held the string '789', c5 would be the number '789.
Save a fixed number in a number variable - use 'savnum' again
savnum=7 1234
- loads the number '1234' into c7.
Save the contents of a single byte - use 'savbyte'
savbyte=33	p7

Note that the contents of the variables, c1, c2 etc are not amended by the calculation UNLESS you specifically save it, eg :

	r=BC	savnum=1 f5	savnum=2 f7	savnum=3 mktcap
will load c3 with the result of the 'mktcap' calculation. Examples of Builtins :
STRLEN
	; test the field 2 is greater than 44 chrs (ie 44 is less than strlen of f2)
	r=HH    ifnnul f2       iflt 44 strlen f2       "Big Field 2 here over 44 chrs long" \n
	r=KK    "Save Field for Name (s55) is " strlen s55 " chrs long"
ZAPLEADZERO
	; data - field 99 is 00000330303, field 101 is 00000000.00
	r=3     "This outputs 330303=" zapleadzero f99 ", while this is
0.00=" zapleadzero f101

Putting it all together - some examples

EXAMPLE ONE 
; file is variable text type
filtyp:t
; each record is separated by CR NL 2 letter type
recsep  \r\n
; There are NO fldsep - we will use partials
; There are NO reckey or fldkey - we will test strings for the type of processing
; allow wild cards
wild:*
; 
set     qc      \004\n
set     topbit  \n{M2Processing Date :
set	dash	" _ "

;Partial a Class line which contains the Class/Name/Length of race
; eg : Class 2 - ATV Anniversary Hcp. - 1000 M
partial:pclass  p:0::\s s:0 n:0  s:0 t:1 p:0::- t:1 p:0

; localised matchs - search and replace
match:mhcp	/(Hcp.)//
match:mhcp2	/Hcp.//
; replace M with meters
match:mmeters	?M?meters?
;
;******************** output section ***********************8
output:
; Start by clearing flags 99 and 1 for each input record...
r=*	clrflag=99 clrflag=1

; Now test for ONLY those lines which match our needs...
;| all  |if field1 start|partial field1 |if partial    |set    |set flag 99 on
;| recs |with Class     |according using|field 3 is not|flag 1 |too
;|      |	       |pclass	 |empty	 |one    |
r=*     ifeq "Class" f1 pclass f1       ifnnul p3       x1      x99

; Print out only the names of a new race - only process if flag 1 is ON
; Use flag x101 to output [rf3] for the FIRST race only - which is the 1st class
r=*     ifflag x1       ifnflag x101    [rf3]   x101
; partial f1 again using pclass, if partial field 6 is NOT empty, remove extra
;	spaces, Do the two search and Replaces and output followed by a 004 NL
r=*     ifflag x1       pclass f1       ifnnul p6 zapspcextra mhcp mhcp2 p6 qc
; remove extra spaces from partial 1 and output it, output partial 2 and 3, then
;	if partial field 8 is NOT empty, add (spc) (dash) (spc) etc
r=*     ifflag x1       zapspcextra p1 p2 p3 ifnnul p8 dash mmeters zapspc p8
r=*     ifflag x1       qc



© FingerPost Ltd. 1996