New DataFormatting Administrators Manual

this version: 003 26may12

Overview

The new version of the DataFormatting module extends and replaces the existing version by adding :

  • Separate Test and Production environments
    • allowing significant changes to be made without interfering with Production
  • Upgrade and Rollback of jobs in Production
  • Concept of Stages
    • Multiple stages rather than the old xchg->format->xchg model
    • Stages can be of any accepted program
    • Each stage can be tested separately
    Multiple Publications working (semi) independently on the same (pair of) servers

  • Much better security

This guide aims to explain and detail what steps are needs to install and run.

Contents

%TOC%

Links to other DataFormat bits

HowTo …

Initial Setup

Preparation

  • Name of Publication(s)
  • A three, four or five letter code plus the name is required
  • Will you be running multiple publications and do you need to block access between them ?
  • If so, you will need the logon lists version of the w4 logon stuff
  • Do you have up-to-date binaries and scripts from FingerPost – the minimum versions are :
     the default form.setup
     ipformat 54t
     ipformbl 37w
     ipformd 22h
     ipxchg 47z
     sfflogon 02k
     form_job.pl 00x
     form_admin.pl 00s
     sffform.pl 01f
     sffbase.pl 02h
     admin_logon_radius.pl 06b
     

    Download the relevant WEBSHOW xchgs from FingerPost

    If running Solaris or Win2K, you will also need the ‘sdiff’ and ‘cmp’ from the gnu ‘diffutils’ package (pre-built binaries are on the FingerPost site)

    create customer.setup

    This is used to control the behaviour of the whole w4 system. It is placed in /fip/web/setup.

    The data formatting bits are to define the publication(s) used and (for multiple publications) the logon list.

    There should be at least one ‘logon-pub’ keyword:-

    	
    logon-pub:DB|The Daily Bugle
    logon-pub:MEGA|Mega Entertainments Weekly
    

    The logon list file is either FIP.INTERNAL or FIP.EXTERNAL

    logon-list-file:FIP
    

    form.setup

    This is used to control the behaviour of the system and to allow access to the various aspects. It is placed in /fip/web/setup

    Do we allow Perl and/or non-standard programs/scripts

    allow-perl:yes
    allow-nonstds:yes
    

    Restrict access to parameter files of other publications

    	 NONE		= No restriction
    	 PUB		= Between publications
    	 SECTION	= Between publications and sections
    	 restrict-access:pub
    

    Balance – this needs to match the entry in ~/tables/sys/BALANCE for all other fip systems running dataFormats

    	 balance-group:formadmin
    

    What views of the data are required ? These map to exchanges e.g. ZWIRE requires ~/tables/xchg/WEBSHOW.ZWIRE

    view:SWIRE,SWIRE2|AP S-Wire Data
    view:SWIREBLANK|AP S-Wire Data (no formatting)
    view:ZWIRE|AP Z-Wire Data
    view:PADF|PA Data Formatted Fields
    view:INCOPY|InCopy Markup
    view:QUARK|Quark XPress Tags
    view:SETTER|Fip Setter Tags
    

    For Solaris and Win2K, use the GNU versions of ‘diff’ once you have added where the programs are

    bin-cmp:/usr/bin/cmp -s
    bin-sdiff:/fip/gnu/diff -y
    

    Logon List

    Changes to other bits of Fip

    ~/tables/sys/SYSTEM file

    If using multiple publications, add an ‘ipformd’ process for each publication, with its own input directory and parameter file
    DO NOT START YET
    Note the parameter file is always ‘PROCESS.(code)’ where code is the publication code.

    abform	local	ipformd -Q -i 2form_ab -p process.ab
    cdform	local	ipformd -Q -i 2form_cd -p process.cd
    etc
    

    If using a single publication

    dpform	tcform	ipformd -Q -i dp_form -p process.dp
    

    For balancing file over multiple systems add ‘ipbalan’ and ‘ipsvrd’ to the SYSTEM file

    balan	local	ipbalan
    svrd	local	ipsvrd -l
    

    Normal Fip routing to get the data into the relevant directories

    All other Fip systems

    Remember to copy (manually) any files you have changed

    Converting from an old Fip system

    Adding a new Publication

    This is currently a manual process :

    Here is an example for a publication called ‘piff’. Please note the case in the following : (i.e. it is important to make folders lowercase and files UPPERCASE) :

    • Add the data folder
    mkdir -p /fip/data/form/piff
    
    • Add an entry in the setup file – /fip/web/setup/form.setup

    pub:(short name) (full name)
    The full name should have ‘\s’ in place of spaces

    pub:piff	The\sDaily\sPiff
    
    • If the publication already exists and there is an existing PROCESS file, copy it twice to
    /fip/web/form/templates/PROCESS.EXTRA.PIFF
    /fip/tables/form/PROCESS.PIFF
    
    • Change the main /fip/tables/sys/SYSTEM (carefully)
    fpiff	local	ipformd -Q -i form_piff -z process.piff
    

    or for non-English, add the default locale

    fpiff	local	ipformd -Q -i form_piff -z process.piff -E fr_FR@euro
    
    • Add a destination in /fip/tables/sys/USERS for incoming, raw data
    		
    dformin_piff=	DP:localhost	DQ:form_piff	SC:no	DC:no
    
    • Add the relevant destinations in /fip/tables/sys/USERS for output
      (this is site/system dependent of course)

    Route incoming data to DU:formin_piff
    e.g. in /fip/tables/route/ files :

    ; Sport (Category R)
    	1	 c=r
    		2	c=rdi	>w4sport
    		2	c=rdr	>w4racing_results+dformin_pub1+dformin_piff
    
    • Create a logon that uses ‘piff
    • Copy relevant parameter files to the correct tables folders
      Sorry this is a manual task at the moment
      Remember, if you copy ‘xchg’ files, that they may reference ‘chrset’ files too

    • Add a logon which can access/create/manage jobs for that publication
      Note a single logon can only access a single publication
      Use the ZADMIN logon (or any other you have created with ADMIN access) to add or change an existing logon and select FORMAT for the Radius type and that publication

    Adding a new Program for use in any DF stage

  • In the main setup file, web/setup/form.setup, there should be five parameter lines added plus your comment lines
  • Firstly think of a name to call it
    p-program:(name) (full pathname to that program)

    p-tables:(name)	(full pathname to the directory where the parameter files are kept or '-' if none)
    p-help:(name)		(URL to the help page)
    p-prod:(name)		(Command line to run the program in production)
    p-test:(name)		(Command line to run the program in test)
    

    e.g.

    p-program:ipformat	/fip/bin/ipformat
    p-tables:ipformat	/fip/tables/form/text
    p-help:ipformat	http://195.13.83.33/twiki/bin/view/FipDoc/Ipformat
    p-prod:ipformat	-i^i -D -S^o -p^p ^s
    p-test:ipformat	-i^i -xob -D -S^o -p^p -cformsave/^u^j^xT.^n ^s
    

    Note the syntax for the command lines is :

    ^i - Input file
    ^o - Output file
    ^p - Parameter file
    ^s - Custom switches as specified in the 'Other Switches' panel on the UserInterface
    ^n - Stage number
    ^j - Job name
    ^u - Publication name
    ^x - a single ^
    Also ^ followed by any other character is a single ^
    

    Note also, where possible, you want a copy of the output file for that section.
    For ‘ipformat’ this is the ‘-cformsave/^u^j^xT.^n’ which will create a file in spool/formsave, called ‘pub^^job^T.9′ which enables the user to click on ‘output file’ on the UI and display that file !

    Cleaning out old projects

    Sorry this is a manual task at the moment. See the [Under the lid] section for where files reside

    Installation

    Dependencies

    • Fip base, in particular at least these version levels:
    • 		 sfflogon	version 02f
      		 ipformd	version 22d
      		 ipformat	version 54m
      		 ipformbl	version 37w
      		 iproute	version 161u
      		 ipxchg	version 47l
      

      Web interface for a new build server

      • Apache/IIS/Domino – Please check with FingerPost if you have lost the documentation on how to setup these.
      • No changes are required at all for an existing FingerPost site, already using the W4 web interface
      • Fip W4 with incoming wires/email etc. for raw data

      Or at least some means of getting raw data into the DF modules.
      If the W4 version is still using /fip/spool/w4data, an upgrade will be needed

    Preparation

    • Beforehand – Back everything up. Preferably upgrade on a spare or test system first

    form.setup
    pub (s)
    balance-group

    logons

    Upgrading programs and scripts

    Create a temporary folder and unzip the two files – binaries and tables under it

    Copy the following files from the release media :

    a. web/templates
    Templates and buttons
    cp -p web/templates/* /fip/web/templates
    b. fip-cgi
    Main user interface scripts
    cp -p web/fip-cgi/* /fip/web/fip-cgi

    c. pages/
    Cascading Style Sheets
    cp -p web/pages/css/* /fip/web/pages/css

    d. setup
    Setup files – customer.setup and form.setup
    cp -p web/setup/* /fip/web/setup

    e. xchgs

    Make sure you have the correct versions of the WEBSHOW

  • xchgs
    cp -p tables/xchg/WEBSHOW* /fip/tables/xchg

    f. logons

    TAKE CARE If you are already running RADIUS_* versions, please take care that you do not overwrite anything here

    	 mkdir /fip/web/logon/lists
    	 mkdir /fip/web/logon/templates
    	 cp -p logon/list/		/fip/web/logon/lists
    	 cp -p logon/templates/form_	/fip/web/logon/templates
    	 cp -p logon/names/RADIUS	/fip/web/logon/names
    

    g. bins
    While only the major binaries are required, you may want to upgrade all fip programs. Please turn all fip programs off beforehand!
    cp -p bin/ /fip/bin
    or

    	 cp -p bnew/sfflogon 	/fip/bin
    	 cp -p bnew/sfflogon	/fip/bin
    	 cp -p bnew/ipformd	/fip/bin
    	 cp -p bnew/ipformat	/fip/bin
    	 cp -p bnew/ipformbl	/fip/bin
    	 cp -p bnew/iproute	/fip/bin
    	 cp -p bnew/ipxchg	/fip/bin
    	 cp -p bnew/ipsgml	/fip/bin
    

    h. mkdir
    /fip/data/form
    /fip/web/form
    /fip/web/form/templates
    /fip/web/form/history
    Sort and test are ignored and can be removed

    Setup

    a. web/setup files
    Merge the existing customer.setup with the new one
    Please see the documentation for ‘sfflogon’ at http://www.fingerpost.co.uk/wp/programme-documentation/sfflogon/

    The parameter ‘logon-list-file’ should match the name of the file in logon/lists
    e.g. for the following, there should be a file called lists/FIPO.INTERNAL

    ; List of standard logons
    logon-list-file:fipo
    

    b. logon
    Tune RADIUS_FORMAT for your wires
    Use the logon ZADMIN (default password hih2) to generate logons with

    c. basic system – add ‘ipbalan’ and ‘ipsvrd’ if they are not already running
    So in tables/sys/SYSTEM, add 2 lines :

    bal     local    ipbalan
    svd     local    ipsvrd -l -P 9140
    

    Modify the BALANCE file in /fip/tables/sys for the format balance group

    	 ;
    	 ; 'BALANCE' - Balance files bewteen systems
    	 ;
    	 group:formadmin	host:(fiphost1)    ignore-localhost:
    	 group:formadmin	host:(fiphost2)    ignore-localhost:
    

    Replace (fiphost1) with the name of the server of the first Fip box; (fiphost2) the second etc.

    Note balance/svrd uses TCP port 9140 to balance data and parameter files, so if the two FingerPost servers are separated by a firewall, data through this port must be allowed.

    Remember for any changes to the main SYSTEM file you can either use ‘ip’ and ‘k all’ and ‘a all’ to bring into operation or very carefully add the changes to the file /fip/x/SYSTEM.

    d. Add the first publication – see ‘Adding a new Publication’ section above

    Under the lid – stuff you dont need !

    Where is everything is

    Data is in

    /fip/data/form/(pub)/SECTION^DESK^JOB^(filename)

    Parameter and history files are

    /fip/web/form/formats/(pub)/
    These files hold for each job :
    stages – program, parameter files, extra switches
    data files to test against
    history
    PROCESS file stubs

    /fip/web/form/history/(pub)/
    These files hold copies of each parameter file at each upgrade point of each job so we can use them to rollback

    Folders/Naming Conventions

    The carat (^) is used as a divider.

    Job names can only have letter, numbers underscore and hyphens.

    SECTION^DESK^JOB^P/T
    P-Production
    T-test

    Next Phase

  • dual/shadow running
  • able to pull file off w4 directly and run Test
  • multiple data files for test
    (ie be able to run same Test against several files to check)
    – for regression testing

  • locking
  • better conversion tools
  • completely isolated parameter files for separate pubs
  • send live test…
  • rollback to a test
  • show differences between versions of jobs – and parameter files.