New DataFormatting Administrators Manual

this version: 003 26may12

Overview

The new version of the DataFormatting module extends and replaces the existing version by adding :

Separate Test and Production environments
- allowing significant changes to be made without interfering with Production
Upgrade and Rollback of jobs in Production
Concept of Stages
- Multiple stages rather than the old xchg->format->xchg model
- Stages can be of any accepted program
- Each stage can be tested separately
Multiple Publications working (semi) independently on the same (pair of) servers
Much better security

This guide aims to explain and detail what steps are needs to install and run.

%TOC%

Links to other DataFormat bits

Users Guide – DataFormattingv2user
Help on Syntax of data formats
XSLT
Others scripts or programs written in perl, csh, VB etc
Fip Program Documenatation

HowTo …

Initial Setup

Preparation

Name of Publication(s)
A three, four or five letter code plus the name is required
Will you be running multiple publications and do you need to block access between them ?
If so, you will need the logon lists version of the w4 logon stuff
Do you have up-to-date binaries and scripts from FingerPost – the minimum versions are :
```
 the default form.setup
 ipformat 54t
 ipformbl 37w
 ipformd 22h
 ipxchg 47z
 sfflogon 02k
 form_job.pl 00x
 form_admin.pl 00s
 sffform.pl 01f
 sffbase.pl 02h
 admin_logon_radius.pl 06b
 
```
Download the relevant WEBSHOW xchgs from FingerPost

If running Solaris or Win2K, you will also need the ‘sdiff’ and ‘cmp’ from the gnu ‘diffutils’ package (pre-built binaries are on the FingerPost site)

create customer.setup

This is used to control the behaviour of the whole w4 system. It is placed in /fip/web/setup.

The data formatting bits are to define the publication(s) used and (for multiple publications) the logon list.

There should be at least one ‘logon-pub’ keyword:-
```
	
logon-pub:DB|The Daily Bugle
logon-pub:MEGA|Mega Entertainments Weekly
```
The logon list file is either FIP.INTERNAL or FIP.EXTERNAL
```
logon-list-file:FIP
```
form.setup

This is used to control the behaviour of the system and to allow access to the various aspects. It is placed in /fip/web/setup

Do we allow Perl and/or non-standard programs/scripts
```
allow-perl:yes
allow-nonstds:yes
```
Restrict access to parameter files of other publications
```
	 NONE		= No restriction
	 PUB		= Between publications
	 SECTION	= Between publications and sections
	 restrict-access:pub
```
Balance – this needs to match the entry in ~/tables/sys/BALANCE for all other fip systems running dataFormats
```
	 balance-group:formadmin
```
What views of the data are required ? These map to exchanges e.g. ZWIRE requires ~/tables/xchg/WEBSHOW.ZWIRE
```
view:SWIRE,SWIRE2|AP S-Wire Data
view:SWIREBLANK|AP S-Wire Data (no formatting)
view:ZWIRE|AP Z-Wire Data
view:PADF|PA Data Formatted Fields
view:INCOPY|InCopy Markup
view:QUARK|Quark XPress Tags
view:SETTER|Fip Setter Tags
```
For Solaris and Win2K, use the GNU versions of ‘diff’ once you have added where the programs are
```
bin-cmp:/usr/bin/cmp -s
bin-sdiff:/fip/gnu/diff -y
```
Logon List

Changes to other bits of Fip

~/tables/sys/SYSTEM file

If using multiple publications, add an ‘ipformd’ process for each publication, with its own input directory and parameter file
DO NOT START YET
Note the parameter file is always ‘PROCESS.(code)’ where code is the publication code.
```
abform	local	ipformd -Q -i 2form_ab -p process.ab
cdform	local	ipformd -Q -i 2form_cd -p process.cd
etc
```
If using a single publication
```
dpform	tcform	ipformd -Q -i dp_form -p process.dp
```
For balancing file over multiple systems add ‘ipbalan’ and ‘ipsvrd’ to the SYSTEM file
```
balan	local	ipbalan
svrd	local	ipsvrd -l
```
Normal Fip routing to get the data into the relevant directories

All other Fip systems

Remember to copy (manually) any files you have changed

Converting from an old Fip system

Adding a new Publication

This is currently a manual process :

Here is an example for a publication called ‘piff’. Please note the case in the following : (i.e. it is important to make folders lowercase and files UPPERCASE) :
- Add the data folder
```
mkdir -p /fip/data/form/piff
```
- Add an entry in the setup file – /fip/web/setup/form.setup
pub:(short name) (full name)
The full name should have ‘\s’ in place of spaces
```
pub:piff	The\sDaily\sPiff
```
- If the publication already exists and there is an existing PROCESS file, copy it twice to
```
/fip/web/form/templates/PROCESS.EXTRA.PIFF
/fip/tables/form/PROCESS.PIFF
```
- Change the main /fip/tables/sys/SYSTEM (carefully)
```
fpiff	local	ipformd -Q -i form_piff -z process.piff
```
or for non-English, add the default locale
```
fpiff	local	ipformd -Q -i form_piff -z process.piff -E fr_FR@euro
```
- Add a destination in /fip/tables/sys/USERS for incoming, raw data
```
		
dformin_piff=	DP:localhost	DQ:form_piff	SC:no	DC:no
```
- Add the relevant destinations in /fip/tables/sys/USERS for output
  (this is site/system dependent of course)
Route incoming data to DU:formin_piff
e.g. in /fip/tables/route/ files :
```
; Sport (Category R)
	1	 c=r
		2	c=rdi	>w4sport
		2	c=rdr	>w4racing_results+dformin_pub1+dformin_piff
```
- Create a logon that uses ‘piff‘
- Copy relevant parameter files to the correct tables folders
  Sorry this is a manual task at the moment
  Remember, if you copy ‘xchg’ files, that they may reference ‘chrset’ files too
- Add a logon which can access/create/manage jobs for that publication
  Note a single logon can only access a single publication
  Use the ZADMIN logon (or any other you have created with ADMIN access) to add or change an existing logon and select FORMAT for the Radius type and that publication
Adding a new Program for use in any DF stage
In the main setup file, web/setup/form.setup, there should be five parameter lines added plus your comment lines
Firstly think of a name to call it
p-program:(name) (full pathname to that program)
```
p-tables:(name)	(full pathname to the directory where the parameter files are kept or '-' if none)
p-help:(name)		(URL to the help page)
p-prod:(name)		(Command line to run the program in production)
p-test:(name)		(Command line to run the program in test)
```
e.g.
```
p-program:ipformat	/fip/bin/ipformat
p-tables:ipformat	/fip/tables/form/text
p-help:ipformat	http://195.13.83.33/twiki/bin/view/FipDoc/Ipformat
p-prod:ipformat	-i^i -D -S^o -p^p ^s
p-test:ipformat	-i^i -xob -D -S^o -p^p -cformsave/^u^j^xT.^n ^s
```
Note the syntax for the command lines is :
```
^i - Input file
^o - Output file
^p - Parameter file
^s - Custom switches as specified in the 'Other Switches' panel on the UserInterface
^n - Stage number
^j - Job name
^u - Publication name
^x - a single ^
Also ^ followed by any other character is a single ^
```
Note also, where possible, you want a copy of the output file for that section.
For ‘ipformat’ this is the ‘-cformsave/^u^j^xT.^n’ which will create a file in spool/formsave, called ‘pub^^job^T.9’ which enables the user to click on ‘output file’ on the UI and display that file !

Cleaning out old projects

Sorry this is a manual task at the moment. See the [Under the lid] section for where files reside

Installation

Dependencies
- Fip base, in particular at least these version levels:
Preparation
- Beforehand – Back everything up. Preferably upgrade on a spare or test system first
form.setup
pub (s)
balance-group

logons

Upgrading programs and scripts

Create a temporary folder and unzip the two files – binaries and tables under it

Copy the following files from the release media :

a. web/templates
Templates and buttons
cp -p web/templates/* /fip/web/templates
b. fip-cgi
Main user interface scripts
cp -p web/fip-cgi/* /fip/web/fip-cgi

c. pages/
Cascading Style Sheets
cp -p web/pages/css/* /fip/web/pages/css

d. setup
Setup files – customer.setup and form.setup
cp -p web/setup/* /fip/web/setup

e. xchgs

Make sure you have the correct versions of the WEBSHOW
xchgs
cp -p tables/xchg/WEBSHOW* /fip/tables/xchg

f. logons

TAKE CARE If you are already running RADIUS_* versions, please take care that you do not overwrite anything here
```
	 mkdir /fip/web/logon/lists
	 mkdir /fip/web/logon/templates
	 cp -p logon/list/		/fip/web/logon/lists
	 cp -p logon/templates/form_	/fip/web/logon/templates
	 cp -p logon/names/RADIUS	/fip/web/logon/names
```
g. bins
While only the major binaries are required, you may want to upgrade all fip programs. Please turn all fip programs off beforehand!
cp -p bin/ /fip/bin
or
```
	 cp -p bnew/sfflogon 	/fip/bin
	 cp -p bnew/sfflogon	/fip/bin
	 cp -p bnew/ipformd	/fip/bin
	 cp -p bnew/ipformat	/fip/bin
	 cp -p bnew/ipformbl	/fip/bin
	 cp -p bnew/iproute	/fip/bin
	 cp -p bnew/ipxchg	/fip/bin
	 cp -p bnew/ipsgml	/fip/bin
```
h. mkdir
/fip/data/form
/fip/web/form
/fip/web/form/templates
/fip/web/form/history
Sort and test are ignored and can be removed

Setup

a. web/setup files
Merge the existing customer.setup with the new one
Please see the documentation for ‘sfflogon’ at http://www.fingerpost.co.uk/wp/programme-documentation/sfflogon/

The parameter ‘logon-list-file’ should match the name of the file in logon/lists
e.g. for the following, there should be a file called lists/FIPO.INTERNAL
```
; List of standard logons
logon-list-file:fipo
```
b. logon
Tune RADIUS_FORMAT for your wires
Use the logon ZADMIN (default password hih2) to generate logons with

c. basic system – add ‘ipbalan’ and ‘ipsvrd’ if they are not already running
So in tables/sys/SYSTEM, add 2 lines :
```
bal     local    ipbalan
svd     local    ipsvrd -l -P 9140
```
Modify the BALANCE file in /fip/tables/sys for the format balance group
```
	 ;
	 ; 'BALANCE' - Balance files bewteen systems
	 ;
	 group:formadmin	host:(fiphost1)    ignore-localhost:
	 group:formadmin	host:(fiphost2)    ignore-localhost:
```
Replace (fiphost1) with the name of the server of the first Fip box; (fiphost2) the second etc.

Note balance/svrd uses TCP port 9140 to balance data and parameter files, so if the two FingerPost servers are separated by a firewall, data through this port must be allowed.

Remember for any changes to the main SYSTEM file you can either use ‘ip’ and ‘k all’ and ‘a all’ to bring into operation or very carefully add the changes to the file /fip/x/SYSTEM.

d. Add the first publication – see ‘Adding a new Publication’ section above

Under the lid – stuff you dont need !

Where is everything is

Data is in

/fip/data/form/(pub)/SECTION^DESK^JOB^(filename)

Parameter and history files are

/fip/web/form/formats/(pub)/
These files hold for each job :
stages – program, parameter files, extra switches
data files to test against
history
PROCESS file stubs

/fip/web/form/history/(pub)/
These files hold copies of each parameter file at each upgrade point of each job so we can use them to rollback

Folders/Naming Conventions

The carat (^) is used as a divider.

Job names can only have letter, numbers underscore and hyphens.

SECTION^DESK^JOB^P/T
P-Production
T-test

Next Phase
dual/shadow running
able to pull file off w4 directly and run Test
multiple data files for test
(ie be able to run same Test against several files to check)
– for regression testing
locking
better conversion tools
completely isolated parameter files for separate pubs
send live test…
rollback to a test
show differences between versions of jobs – and parameter files.

FingerPost

A documentation and information site for the FingerPost collection of programs : bringing the news to those who bring the news to you.

New DataFormatting Administrators Manual

Overview

Contents

Links to other DataFormat bits

HowTo …

Initial Setup

Preparation

create customer.setup

form.setup

Logon List

Changes to other bits of Fip

All other Fip systems

Converting from an old Fip system

Adding a new Publication

Adding a new Program for use in any DF stage

Cleaning out old projects

Installation

Dependencies

Preparation

Upgrading programs and scripts

Setup

Under the lid – stuff you dont need !

Where is everything is

Data is in

Parameter and history files are

Folders/Naming Conventions

Next Phase