this version: 003 26may12<\/p>\n
The new version of the DataFormatting module extends and replaces the existing version by adding :<\/p>\n
This guide aims to explain and detail what steps are needs to install and run.<\/p>\n
%TOC%<\/p>\n
\r\n the default form.setup\r\n ipformat 54t\r\n ipformbl 37w\r\n ipformd 22h\r\n ipxchg 47z\r\n sfflogon 02k\r\n form_job.pl 00x\r\n form_admin.pl 00s\r\n sffform.pl 01f\r\n sffbase.pl 02h\r\n admin_logon_radius.pl 06b\r\n <\/pre>\nDownload the relevant WEBSHOW xchgs from FingerPost<\/strong><\/p>\n
If running Solaris or Win2K, you will also need the ‘sdiff’ and ‘cmp’ from the gnu ‘diffutils’ package (pre-built binaries are on the FingerPost site)<\/p>\n
create customer.setup<\/h4>\n
This is used to control the behaviour of the whole w4 system. It is placed in
\/fip\/web\/setup<\/code>.<\/p>\nThe data formatting bits are to define the publication(s) used and (for multiple publications) the logon list.<\/p>\n
There should be at least one ‘logon-pub’ keyword:-<\/p>\n
\t\r\nlogon-pub:DB|The Daily Bugle\r\nlogon-pub:MEGA|Mega Entertainments Weekly\r\n<\/pre>\nThe logon list file is either FIP.INTERNAL or FIP.EXTERNAL<\/p>\n
\r\nlogon-list-file:FIP\r\n<\/pre>\nform.setup<\/h4>\n
This is used to control the behaviour of the system and to allow access to the various aspects. It is placed in
\/fip\/web\/setup<\/code><\/p>\nDo we allow Perl and\/or non-standard programs\/scripts<\/p>\n
\r\nallow-perl:yes\r\nallow-nonstds:yes\r\n<\/pre>\nRestrict access to parameter files of other publications<\/p>\n
\r\n\t NONE\t\t= No restriction\r\n\t PUB\t\t= Between publications\r\n\t SECTION\t= Between publications and sections\r\n\t restrict-access:pub\r\n<\/pre>\nBalance – this needs to match the entry in ~\/tables\/sys\/BALANCE for all other fip systems running dataFormats<\/p>\n
\t balance-group:formadmin\r\n<\/pre>\nWhat views of the data are required ? These map to exchanges e.g. ZWIRE requires ~\/tables\/xchg\/WEBSHOW.ZWIRE<\/p>\n
\r\nview:SWIRE,SWIRE2|AP S-Wire Data\r\nview:SWIREBLANK|AP S-Wire Data (no formatting)\r\nview:ZWIRE|AP Z-Wire Data\r\nview:PADF|PA Data Formatted Fields\r\nview:INCOPY|InCopy Markup\r\nview:QUARK|Quark XPress Tags\r\nview:SETTER|Fip Setter Tags\r\n<\/pre>\nFor Solaris and Win2K, use the GNU versions of ‘diff’ once you have added where the programs are<\/p>\n
\r\nbin-cmp:\/usr\/bin\/cmp -s\r\nbin-sdiff:\/fip\/gnu\/diff -y\r\n<\/pre>\nLogon List<\/h4>\n
Changes to other bits of Fip<\/h5>\n
\t
~\/tables\/sys\/SYSTEM file<\/code><\/p>\nIf using multiple publications, add an ‘ipformd’ process for each publication, with its own input directory and parameter file
\n DO NOT START YET<\/font>
\n Note the parameter file is always ‘PROCESS.(code)’ where code is the publication code.<\/p>\n\r\nabform\tlocal\tipformd -Q -i 2form_ab -p process.ab\r\ncdform\tlocal\tipformd -Q -i 2form_cd -p process.cd\r\netc\r\n<\/pre>\nIf using a single publication<\/p>\n
\r\ndpform\ttcform\tipformd -Q -i dp_form -p process.dp\r\n<\/pre>\nFor balancing file over multiple systems add ‘ipbalan’ and ‘ipsvrd’ to the SYSTEM file<\/p>\n
\r\nbalan\tlocal\tipbalan\r\nsvrd\tlocal\tipsvrd -l\r\n<\/pre>\nNormal Fip routing to get the data into the relevant directories<\/p>\n
All other Fip systems<\/h4>\n
Remember to copy (manually) any files you have changed<\/p>\n
Converting from an old Fip system<\/h3>\n
Adding a new Publication<\/h4>\n
This is currently a manual process :<\/p>\n
Here is an example for a publication called ‘piff’. Please note the case in the following : (i.e. it is important to make folders lowercase and files UPPERCASE) :<\/p>\n
\n
- Add the data folder\n<\/ul>\n
\r\nmkdir -p \/fip\/data\/form\/piff\r\n<\/pre>\n\n
- Add an entry in the setup file – \/fip\/web\/setup\/form.setup<\/strong>\n<\/ul>\n
pub:(short name)\t(full name)<\/code>
\nThe full name should have ‘\\s’ in place of spaces<\/i><\/p>\n\r\npub:piff\tThe\\sDaily\\sPiff\r\n<\/pre>\n\n
- If the publication already exists and there is an existing PROCESS<\/strong> file, copy it twice to\n<\/ul>\n
\r\n\/fip\/web\/form\/templates\/PROCESS.EXTRA.PIFF\r\n\/fip\/tables\/form\/PROCESS.PIFF\r\n<\/pre>\n\n
- Change the main \/fip\/tables\/sys\/SYSTEM<\/strong> (carefully)\n<\/ul>\n
\r\nfpiff\tlocal\tipformd -Q -i form_piff -z process.piff\r\n<\/pre>\nor for non-English, add the default locale<\/i><\/p>\n
\r\nfpiff\tlocal\tipformd -Q -i form_piff -z process.piff -E fr_FR@euro\r\n<\/pre>\n\n
- Add a destination in \/fip\/tables\/sys\/USERS<\/strong> for incoming, raw data\n<\/ul>\n
\t\t\r\ndformin_piff=\tDP:localhost\tDQ:form_piff\tSC:no\tDC:no\r\n<\/pre>\n\n
- Add the relevant destinations in
\/fip\/tables\/sys\/USERS<\/code> for output
\n(this is site\/system dependent of course)<\/i>\n<\/ul>\nRoute incoming data to
DU:formin_piff<\/code>
\ne.g. in\/fip\/tables\/route\/<\/code> files :<\/p>\n\r\n; Sport (Category R)\r\n\t1\t c=r\r\n\t\t2\tc=rdi\t>w4sport\r\n\t\t2\tc=rdr\t>w4racing_results+dformin_pub1+dformin_piff\r\n<\/pre>\n\n
- Create a logon that uses ‘piff<\/strong>‘\n
- Copy relevant parameter files to the correct tables folders
\n\t\t Sorry this is a manual task at the moment
\n\t\t Remember, if you copy ‘xchg’ files, that they may reference ‘chrset’ files too<\/i><\/p>\n- Add a logon which can access\/create\/manage jobs for that publication
\n\t\t Note a single logon can only access a single publication
\n\t\t Use the ZADMIN logon (or any other you have created with ADMIN access) to add or change an existing logon and select FORMAT for the Radius type and that publication<\/i>\n<\/ul>\nAdding a new Program for use in any DF stage<\/h3>\n
- In the main setup file, web\/setup\/form.setup, there should be five parameter lines added plus your comment lines\n
- Firstly think of a name to call it
\n\t\tp-program:(name)\t(full pathname to that program)<\/code><\/p>\n\r\np-tables:(name)\t(full pathname to the directory where the parameter files are kept or '-' if none)\r\np-help:(name)\t\t(URL to the help page)\r\np-prod:(name)\t\t(Command line to run the program in production)\r\np-test:(name)\t\t(Command line to run the program in test)\r\n<\/pre>\ne.g.<\/p>\n
\r\np-program:ipformat\t\/fip\/bin\/ipformat\r\np-tables:ipformat\t\/fip\/tables\/form\/text\r\np-help:ipformat\thttp:\/\/195.13.83.33\/twiki\/bin\/view\/FipDoc\/Ipformat\r\np-prod:ipformat\t-i^i -D -S^o -p^p ^s\r\np-test:ipformat\t-i^i -xob -D -S^o -p^p -cformsave\/^u^j^xT.^n ^s\r\n<\/pre>\n\t\t Note the syntax for the command lines is :<\/p>\n
\r\n^i - Input file\r\n^o - Output file\r\n^p - Parameter file\r\n^s - Custom switches as specified in the 'Other Switches' panel on the UserInterface\r\n^n - Stage number\r\n^j - Job name\r\n^u - Publication name\r\n^x - a single ^\r\nAlso ^ followed by any other character is a single ^\r\n<\/pre>\n\t\t Note also, where possible, you want a copy of the output file for that section.
\n\t\t For ‘ipformat’ this is the ‘-cformsave\/^u^j^xT.^n’ which will create a file in spool\/formsave, called ‘pub^^job^T.9’ which enables the user to click on ‘output file’ on the UI and display that file !<\/p>\nCleaning out old projects<\/h3>\n
\tSorry this is a manual task at the moment. See the [Under the lid] section for where files reside<\/p>\n
Installation<\/h2>\n
\tDependencies<\/h3>\n
\n
- Fip base, in particular at least these version levels:\n<\/li>\n
\r\n\t\t sfflogon\tversion 02f\r\n\t\t ipformd\tversion 22d\r\n\t\t ipformat\tversion 54m\r\n\t\t ipformbl\tversion 37w\r\n\t\t iproute\tversion 161u\r\n\t\t ipxchg\tversion 47l\r\n<\/pre>\n\tWeb interface for a new build server<\/p>\n
\n
- Apache\/IIS\/Domino – Please check with FingerPost if you have lost the documentation on how to setup these.\n
- No changes are required at all for an existing FingerPost site, already using the W4 web interface\n<\/ul>\n
\n
- \tFip W4 with incoming wires\/email etc. for raw data\n<\/ul>\n
Or at least some means of getting raw data into the DF modules.
\nIf the W4 version is still using \/fip\/spool\/w4data, an upgrade will be needed<\/i>\n<\/ul>\nPreparation<\/h3>\n
\n
- Beforehand – Back everything up. Preferably upgrade on a spare or test system first\n<\/ul>\n
form.setup
\npub (s)
\nbalance-group<\/p>\nlogons<\/p>\n
\tUpgrading programs and scripts<\/h3>\n
Create a temporary folder and unzip the two files – binaries and tables under it<\/p>\n
Copy the following files from the release media :<\/p>\n
a. web\/templates
\n\tTemplates and buttons
\n\tcp -p web\/templates\/*\t\/fip\/web\/templates<\/code>
\nb. fip-cgi
\n\tMain user interface scripts
\n\tcp -p web\/fip-cgi\/*\t\/fip\/web\/fip-cgi<\/code><\/p>\nc. pages\/
\nCascading Style Sheets
\ncp -p web\/pages\/css\/*\t\/fip\/web\/pages\/css<\/code><\/p>\nd. setup
\n\tSetup files – customer.setup and form.setup
\n\tcp -p web\/setup\/*\t\/fip\/web\/setup<\/code><\/p>\ne. xchgs<\/p>\n
\tMake sure you have the correct versions of the WEBSHOW<\/p>\n
- xchgs
\n\tcp -p tables\/xchg\/WEBSHOW*\t\/fip\/tables\/xchg<\/code><\/p>\nf. logons<\/p>\n
\tTAKE CARE<\/font> If you are already running RADIUS_* versions, please take care that you do not overwrite anything here<\/p>\n
\r\n\t mkdir \/fip\/web\/logon\/lists\r\n\t mkdir \/fip\/web\/logon\/templates\r\n\t cp -p logon\/list\/\t\t\/fip\/web\/logon\/lists\r\n\t cp -p logon\/templates\/form_\t\/fip\/web\/logon\/templates\r\n\t cp -p logon\/names\/RADIUS\t\/fip\/web\/logon\/names\r\n<\/pre>\ng. bins
\n\t While only the major binaries are required, you may want to upgrade all fip programs. Please turn all fip programs off beforehand!
\ncp -p bin\/\t\/fip\/bin<\/code>
\n\t or<\/p>\n\t cp -p bnew\/sfflogon \t\/fip\/bin\r\n\t cp -p bnew\/sfflogon\t\/fip\/bin\r\n\t cp -p bnew\/ipformd\t\/fip\/bin\r\n\t cp -p bnew\/ipformat\t\/fip\/bin\r\n\t cp -p bnew\/ipformbl\t\/fip\/bin\r\n\t cp -p bnew\/iproute\t\/fip\/bin\r\n\t cp -p bnew\/ipxchg\t\/fip\/bin\r\n\t cp -p bnew\/ipsgml\t\/fip\/bin\r\n<\/pre>\nh. mkdir
\n\t \/fip\/data\/form
\n\t \/fip\/web\/form
\n\t \/fip\/web\/form\/templates
\n\t \/fip\/web\/form\/history
\n\t Sort and test are ignored and can be removed<\/p>\nSetup<\/h3>\n
a. web\/setup files
\n\t Merge the existing customer.setup with the new one
\n\t Please see the documentation for ‘sfflogon’ at http:\/\/www.fingerpost.co.uk\/wp\/programme-documentation\/sfflogon\/<\/p>\n\t The parameter ‘logon-list-file’ should match the name of the file in logon\/lists
\n\t e.g. for the following, there should be a file calledlists\/FIPO.INTERNAL<\/code><\/p>\n\r\n; List of standard logons\r\nlogon-list-file:fipo\r\n<\/pre>\nb. logon
\n\t Tune RADIUS_FORMAT for your wires
\n\t Use the logon ZADMIN (default password hih2) to generate logons with <\/p>\nc. basic system – add ‘ipbalan’ and ‘ipsvrd’ if they are not already running
\n\t So in tables\/sys\/SYSTEM, add 2 lines :<\/p>\n\r\nbal local ipbalan\r\nsvd local ipsvrd -l -P 9140\r\n<\/pre>\n\t Modify the BALANCE file in \/fip\/tables\/sys for the format balance group<\/p>\n
\r\n\t ;\r\n\t ; 'BALANCE' - Balance files bewteen systems\r\n\t ;\r\n\t group:formadmin\thost:(fiphost1) ignore-localhost:\r\n\t group:formadmin\thost:(fiphost2) ignore-localhost:\r\n<\/pre>\nReplace (fiphost1) with the name of the server of the first Fip box; (fiphost2) the second etc.<\/p>\n
Note balance\/svrd uses TCP port 9140 to balance data and parameter files, so if the two FingerPost servers are separated by a firewall, data through this port must be allowed.<\/p>\n
Remember for any changes to the main SYSTEM file you can either use ‘ip’ and ‘k all’ and ‘a all’ to bring into operation or very carefully add the changes to the file \/fip\/x\/SYSTEM.<\/p>\n
d. Add the first publication – see ‘Adding a new Publication’ section above<\/p>\n
Under the lid – stuff you dont need !<\/h2>\n
Where is everything is<\/h3>\n
Data is in <\/h4>\n
\/fip\/data\/form\/(pub)\/SECTION^DESK^JOB^(filename)<\/code><\/p>\nParameter and history files are<\/h4>\n
\t
\/fip\/web\/form\/formats\/(pub)\/<\/code>
\n\t These files hold for each job :
\n\t\t stages – program, parameter files, extra switches
\n\t\t data files to test against
\n\t\t history
\n\t\t PROCESS file stubs<\/p>\n\t
\/fip\/web\/form\/history\/(pub)\/<\/code>
\n\t These files hold copies of each parameter file at each upgrade point of each job so we can use them to rollback<\/p>\n\tFolders\/Naming Conventions<\/h3>\n
The carat (^) is used as a divider.<\/p>\n
Job names can only have letter, numbers underscore and hyphens.<\/p>\n
SECTION^DESK^JOB^P\/T
\n\tP-Production
\n\tT-test<\/p>\nNext Phase<\/h2>\n
- dual\/shadow running\n
- able to pull file off w4 directly and run Test\n
- multiple data files for test
\n\t(ie be able to run same Test against several files to check)
\n\t– for regression testing<\/p>\n- locking\n
- better conversion tools\n
- completely isolated parameter files for separate pubs\n
- send live test…\n
- rollback to a test\n
- show differences between versions of jobs – and parameter files.
\n","protected":false},"excerpt":{"rendered":"this version: 003 26may12 Overview The new version of the DataFormatting module extends and replaces the existing version by adding : Separate Test and Production environments allowing significant changes to be made without interfering with Production Upgrade and Rollback of jobs in Production Concept of Stages Multiple stages rather than the old xchg->format->xchg model Stages […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/www.fingerpost.co.uk\/wp\/wp-json\/wp\/v2\/pages\/1243"}],"collection":[{"href":"https:\/\/www.fingerpost.co.uk\/wp\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.fingerpost.co.uk\/wp\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.fingerpost.co.uk\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fingerpost.co.uk\/wp\/wp-json\/wp\/v2\/comments?post=1243"}],"version-history":[{"count":37,"href":"https:\/\/www.fingerpost.co.uk\/wp\/wp-json\/wp\/v2\/pages\/1243\/revisions"}],"predecessor-version":[{"id":1439,"href":"https:\/\/www.fingerpost.co.uk\/wp\/wp-json\/wp\/v2\/pages\/1243\/revisions\/1439"}],"wp:attachment":[{"href":"https:\/\/www.fingerpost.co.uk\/wp\/wp-json\/wp\/v2\/media?parent=1243"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}