fipman

FingerPost FIP Managers Guide

This Guide is copyright 2013 FingerPost Ltd.

—-

Introduction

1. Overview

The FingerPost Information Processor (Fip) is a family of communication packages specifically created for publishing. It is designed for the varied needs of newspapers, magazines and news agencies, from small weekly newspapers to national magazines and international wire services.

There are FingerPost systems installed in 19 countries

Fip is Media Neutral so most types can be imported and exported, possibly in a different format or using different methods.

Fip is often used to surround an editorial or advertising platform, handing all the inputs and outputs in order to let the Content Management System manage the internal workflow.

2. Clients and Purposes

Clients on the Agency side include TFN, BBC Monitoring, the scandinavian agencies Ritzau, TT, NTB; and Reuters who is currently 10 different systems running on over 40 servers for a variety of tasks.

On the publishing side Fips are running in the biggest – Los Angeles Times, New York Times, Washington Post – to small Advertising freesheets.

Other systems are installed for diverse tasks such as handling Betting feeds and Financial feeds.

Fip handles data in many different languages and formats – handling Russian, Hindu scripts, Arabic, and used for pictures, text, all xml variants, voice and other binary blobs.

3. Different Modules

3.a The Base module

Fip is built on a common base module which handles the majority of the input formats, and routing, minor style changing.

The base module is also used for the resilience, statistics, tracking errors using either the command line interface or W4, the web interface.

As FIP is modular, adding services is simple. As volumes grow and more power is required, new ports and computers can be added to the existing installation.

FIP is easy to use and easy to modify – nearly all setup parameters are in held in text files which can be modified by the system manager quickly and easily. With only a few exceptions, once the parameter files have been modified, the change is recognised immediately.

It handles a variety of input and output methods :

• Web grab and web feed using HTTP, SOAP etc
• FTP get and receive including Secure FTP
• fipremote
• Email – POP3, SMTP and IMAP
• Serial cables
• X25
• Web viewing of all data using the Fip web browser W4
Most data formats are catered for :
• old text services – such as ANPA 1312 and IPTC 7901
• xml variants – such as NewsML, Prism.
• data conversions between proprietary formats (usually using 3rd party software)

3.b NewsAgency/Syndication/Remote Transmission module.

Basically the same source data can be grabbed, reformatted (if required) and then pushed to the client or pulled by the client using whichever is the securest/easiest/quickest/cheapest method.

We find that allowing the customer/end user to define what method and which format they want the data to be in, means that changes in the field are kept to a minimum leading to shorter install times. While the Fips at the central site is normally adaptable to those requirements fairly easily.

The delivery method can be any including normal web transmission type such as FTP, email/SMTP, SOAP, HTTPD

And/or it can send directly to any raw socket such as other Fip systems listening on defined ports.

And/or it also includes a free client program ‘fipremote’ which the remote user can download and use to pull the data directly from your servers – useful in cases where firewalls block most inbound traffic.

And/or the data can be left on the (secure) web site from a human to browse and select manually. This method can be used for both the occasional user and also for requesting resends where the remote site has had a sigificant problem.

3.c Other modules include

– WebGrabber module – one client is currently using this to check for changes on over 2000 external websites. Any new files are grabbed, reformatted and output like a wire feed.

– DataFormatting module – for heavy duty reformatting of tabular data – sports, elections, stox and bonds.

– Postcript and PDF page generator for whole stox and sports pages.

– Manual tables module – for media neutral tables that need to be sent to a variety of different outputs.

– Statistics module for custom financial and sports databases.

– Various Uploaders and Downloaders :
Notes
Oracle, Sybase, Microsoft Sql Server, MySql
Proprietary publishing systems such as CCI, SII, Atex, Mediasystem

– Various tools for sundry uses
Mimic mailbox

4. Hardware and Software

What size of computer would be needed ?

The FingerPost system (Fip) runs on one or more servers – normally two for redundancy. These are used to bring in the data, reformat it and send it to one or more destinations (or none in the case of time checks or other unwanted material !).

The two servers do not need to be in close proximity – so we should be able to replicate the existing setup with one box in the main site and the second at a Business Continuity/DR site. The caveat on this is that the two Fip boxes MUST be able to pass data between themselves with the obvious implications of changes to firewalls to allow this.

Hardware costs depend somewhat on the supplier/OS. Fip runs on Linux (RedHat), Solaris, AIX, Win200X and MacOSX. There is no difference in the features for any of the ports.

We have a slight preference for Linux and then Solaris – mainly for speed, cost and the ability to administer them remotely – and a slight aversion to Win2000 for the all three reasons.

Fip normally runs two or more systems, all running live in an automatic redundant fashion. Although the calculations depend a lot on the actual data, each server can handle up to 100 remote clients. As more users/clients are added, more servers can be added and there is no theoretical maximum number of servers.

While two servers should cope easily with most loads, a third is often useful (but certainly not essential) as a hot spare and allow new test feeds to be added before loading on the production boxes.

We assume you would prefer to source any hardware from an existing (local) supplier and so for sizing purposes we would be looking at two or three servers costing approximately :

While we suggest these are standalone boxes for resilience and redundancy, Fip does NOT have to run on its own boxes and, for example, can be run on the Notes servers themselves.

There are also associated manuals on the following sub-systems/modules :
* News Agency and Syndication and Library modules
* Data Formatting module
* Remote database file retrieval module
* Statistics module

2. Adding new Devices and Services

2.1 Installing new Hardware

When installing new hardware, always use the maunfacturer’s documentation. This section can advise what parameters should be entered.

2.1.1 Installing a new Terminal Server / SpiderPort

FIP expects its ports setup in a certain way.

SpiderPort installation
This is how to install a new SpiderPort. The synatx or procedure will differ for other types of terminal servers but the theory is the same: you need to give it a name and Internet number and you want to configure each port to slave with the correct speed, data bits and flow control (if appropriate). …. See SpiderPort and TerminalServers.

2.1.2 Adding a TTY port

There is little to add to the tty installation procedure.

On the Sun, make sure the /etc/ttytab file has that file to no login.

On the RS6000 this the installation is done via :
smit/tty/add a tty/rs232
port should be S1 or S2
login disabled
Where an agency feed arrives using only pins 3 and 7,
you will need to strap pins 5 and 6.

2.2 Adding any new Service

The same steps have to be taken when adding any type of new service before you make the changes specific to that wire, telex, fax or whatever.

2.2.1 The Physical Connection.

Check the cable specification of the modem, pad or communications device you will connect to. Check for flow control in particular. Note that early versions of the Spider doc had pins 4 and 5 reversed. Many services – especially wires – are broadcast and so a simple 4 wire cable is all that is needed.

2.2.2 Program the Sun port or the Spider.

1. For the Spider :

* Go into the AMIN port (telnet spiderx 2048) and select the CHANGELINE (CL) command for the port. Set to :
* Type : Slave.
* Port No: 1xyy where x is ‘spiderx’ and yy the portno.
* Speed/Parity/DataBits/XonXoff : as per modem or service.
* Password : none

If there is no traffic on the Spider, it is now ready for use.

2. For the IBM/Sun :

* Please follow the instructions as in the Sun manual ‘System and Network Administration’, Part 2 Procedures, Section 11:5 Adding a Printer to your network.

* For SunOS.3, change /etc/ttys while SunOS.4 needs a new line in /etc/ttytab with the getty OFF but set to standard and the correct speed.

Setting up the tty port logically is covered in the FILES/STTY section.

All FIP programs will (re)set the parameters of the /dev/tty using this STTY file except IPOUT where only the speed may be changed.

To make the change you will need to kill init using :

	kill -HUP 1

If you are using a terminal line for testing, you can check what the current settings are by :

	(stty everything) > /dev/ttyZ		- Sun version
	(stty everything) . /dev/ttyZ		- IBM version
	where Z is the particular device/portno.

2.2.3 The SYSTEM file

Add a line to the main SYSTEM file in ‘tables/sys’ (see section 3.1.1) with a unique name, a group and the specific program with ALL switches you need (see section 2 for program switches).

When FIP starts, it makes a copy of the SYSTEM file in /fip/x which it then uses to track programs.

Although not recommended, you can make immediate changes on the fly. First make sure no-oneelse is running IP, and then make the same change to the SYSTEM file in ‘x’ adding ‘-1’ in front of the name (this is the pid number; -1 means not-running). Then restart IP from a root logon and, once the remainder of the following startup procedure has been completed, start the new process in the normal fashion.

The bottom line is take care or you may get a runaway !

2.2.4 The PEEK file

If you have added new TTY ports in the system, there must be a note in ‘tables/sys/PEEK’ (see section 3.1.4) that links Sun ports to Pseudo-Shared memory areas. This is used by all incoming programs (and PEEK itself) and will not start if there is none specified. These numbers must be unique or you will have meaningless information when using PEEK.

2.2.5 Adding a new *WIRE service

1. Firstly consult the Wire Agency as to what message format the feed will be. Then consult section 3 for setup on :

|ANPA/IPTC formats|[[Wire][Wire]]|
|IATA format|[[Zwire][Zwire]]|
|Teletex or Broker feed formats|[[Bkwire][Bkwire]]|
|Precedent character [like Atex CCM] format input program.|[[Pwire][Pwire]]|
|Any other format|[[Vwire][Vwire]]|

2. Create a routing table in ‘tables/route’ (section 3.2) if you are not using an existing one.

3. If you have added any new destinations, then make the changes in ‘tables/sys/USERS’ (section 3.1.2)

4. If you have specified a new source character set or a new destination chrset, then a new XCHG file (section 3.3) is needed. If the character set is different, add in ‘tables/chrset’ too.

2.2.6 Adding a new mailbox

Adding a new mailbox can have one of two meanings. It means either adding a new person who wishes to use an existing mailbox system or it can mean adding another path to the mailbox using [[Ipbox][ipbox]] – the mailbox program. Multiple IPBOXs can access the same database at the same time.

1. Adding a new user mailbox is done by adding a new line in the LOGON file (section 3.6). If the user needs a mailbox then a queue needs to be made in ‘spool/mail/name’ where name is that given to the ‘box:’ keyword. No other changes are necessary and the logon is valid immediately. If the LOGON file does not exist, create it and add the single line.

2. Adding a new communications channel and another copy of IPBOX – the mailbox program with either a modem or x25 connection needs just the physical connection plus the specific progam parameters (section 3.6).

2.2.7 Adding a new fax or telex or EasyLink line.

For a new fax or incoming telex line, the correct program parameters (section 3.7) need to be set plus a routing table in ‘tables/route’ (section 3.2). For telex, the other answerback is placed in the ‘N’ field of the Source Header which can be routed on. Note that the ‘-r’ switch enables you to use the same routing file for all or several telex lines.

For telex outbound and fax, new queues need to be specified for the outbound :

	'spool/telex/xxx/out'
	'spool/telex/xxx/occ'

or the relevant ones for fax, where xxx is the name of this particular telex or fax out. This enables you to have several exit paths depending from where the files have come from, to assist system loading. These queues are not necessary if the telex line is incoming only.

The ‘xxx’ is defined either as ‘telex’ or ‘fax’ or via an Environment variable (qv).

In the USERS file (section 3.1.2), any new destinations need to be specified.

EasyLink files have two sections – one to set up the modem and the second for the ID of the you logon.

2.3 Tuning UNIX

1. For Suns – rebuild VMUNIX – the exec.g
* * FIP is greedy where it comes to processes. Consequently it is usual to rebuild the kernel adding about 100 processes just in case. Your installer will generally do this when installing.

2. CRONTAB
* * Use crontab to clean the LOG files.

* * There is a script : /fip/local/zapfiplog which can be modified for your needs plus a second for syndication – /fip/local/zapfipsyn.

* * Zapfiplog generally runs program IPMAINT to trim the Item Log, zap the Archive Logs and running any accounting required. It can also be used to zap the contents of temp queues etc – do as you please. It is often run in the small hours of the night.

3. RC or RC.LOCAL
* * Please add lines in (Sun)/etc/rc.local or (IBM)/etc/rc to delete the contents of queue ‘/fip/x’ but not the queue !

* * You can also restart processes by running FIP in the same script. The full string : /fip/bin/ip, should be used plus the switch ‘-i’. Any processes with the HOST group identical to the name of that Sun, will be started.

4. XWINDOWS or SUNTOOLS
* * If you are running windows, you may wish to allocate windows to primarily to FIP. The main FIP window should be at least 20 lines deep (preferably slightly longer than your SYSTEM file). The actual size is important for the FIP log command (‘l’) which uses a system parameter to decide the number of lines to display.

* * While a single window is adequate for all, you may find it simpler to have multiple windows for different monitor tasks – possibly on other unixboxes.

* * A second window can be running script ‘ipmon’ which checks the log file for last entries every minute.

* * A third could be used for ‘ipscan’ to review local queues.

* * Syndication and AutomaticDataFormats normally will normally be given another window.

5. ENVIRONMENT
* * FIP will only run if it has a number of environment variables set beforehand.

* * The system manager may want to alter .cshrc to add these.

* * A full list will be in the ‘deliv/system/add.cshrc’ file on your delivery system.

They include :

# include the programs and scripts in the path
set path = ($path /fip/bin /fip/local)
umask 0
alias ips	'cd /fip/spool ;echo $cwd'
alias ipt	'cd /fip/tables ;echo $cwd'

# The main flag - if not present, IP will NOT run
setenv	FIP_OK	yes
# name of the editorial system root :ie ed in edone, edtwo
# this is used for multi network sites
setenv	FIP_EDSYS	"atex"
# edsys net is used by ipout as default system
setenv	FIP_def_EDSYS	"atex1"
# edsys net is used by ipwheel as default network
setenv	FIP_def_EDSYS_NET	"atex"
# Is it an Atex J11 site
setenv	FIP_IPOUT_J11	yes
# Date format - European or USA date format
setenv	FIP_USA_DATE	no
# spiders
setenv	FIP_SPIDER_PASSWD	"hello"
# name of spider for IPOUT and IPSYN
setenv	FIP_SPIDER_OUT	"spider"
setenv	FIP_SPIDER_SYN	"spider"
# screen size
setenv	FIP_SCREEN	32
setenv	FIP_PC_SCREEN	20
# Fax banner etc
setenv	FIP_FAX_QUE	evnews
setenv	FIP_FAX_PAGE	"The Evening News"
setenv	FIP_FAX_BANNER	"\n^j0\n\033EFrom The Evening News, 324 North Pocket Nook Avenue\nGeneral tph : 833-8342 or 837-2114, Telex 76776 ENEWS Z\n\n"
# telex stuff
setenv	FIP_TLX_ANSBCK	"915684"
setenv	FIP_TLX_QUE	evnews
# ed system modes
setenv	FIP_ROMAN	"[m0"
setenv	FIP_BOLD	"[m1"
setenv	FIP_NOTES	"[m2"
setenv	FIP_MODE3	"[m3"
setenv	FIP_ITALIC	"[m4"
setenv	FIP_MODE5	"[m5"
setenv	FIP_MODE6	"[m6"
setenv	FIP_MODE7	"[m7"
setenv	FIP_FLASH	"[m8"
# Writer is MAIN editorial system ??
setenv	FIP_WRITER	no
# Hostname where the Syndication system is running
setenv	FIP_SYN_SYS	* fipsyn
# strip the hh.mm from the headline in IATA files ?
setenv	FIP_ZWIRE_HHMM  no
# Timeout in seconds for inter-unixbox FIPNET traffic (def 8)
setenv	FIP_NET_TIME	5

6. NIS/YELLOW PAGES/TOPS/NFS etcg
* * FIP is totally transparent to NIS (ex yellow pages), TOPS and nearly all other software packages known to man or beast. If you have a problem, contact us.

* * PCs can mount specific FIP queues for sending telexes, faxes etc.

2.4 The Editorial Connection

2.4.1 Atex

The object of the XCHG files (section 3.3) is so that no character translation or further processing is needed on the Atex side.

Note that the precedent character for CCM/RCVFIP should be the same as in the in the tables/out/FORMAT parameter files for program [[Ipout][IPOUT]] (qv).

1. The Async Connection – using TRACCMg
* * The usual connection with Atex is via the TRACCM program or via RCVFIP. They run with very similar tables.

* * Hence for TRACCM, the most basic CCM table is used.
ie:

ct:173		; use curly bracket
de:sca00	; device
df:$12345678-ccm-w06	; default file
hd:wire		; use WIRE.HED
bi:8

* * For the default file, it is advisable to use a group and queue that is fixlsted to that one system to reduce bus traffic to a minimum.

* * So *-w06 would only be on system 6 for example.

* * Remember to build the table afterwards using BLDCCM.

2. The SCPg
* * The SCP parameters should use both transmit and receive, 8 bit (use a non-useable chr like 373 as the ESC chr). Note that there is a version of the SCP software for FIP which is available via Atex UK.

* * Hardware flow control between the SCP and the Spider is advisable as XON/XOFF are valid atex chrs. Although using a speed of 4800 should not need any flow control unless your Atex is seriously overloaded.

3. Special Charactersg
* * Note that Atex does not like special chrs \176 \177 \376 and \377. Any XCHG or CHRSET file should attempt to strip these. Similarly code \373 should be stripped if you are using it as a false escape chr in the SCP CCT table.

4. Using RCVFIP connectiong
* * RCVFIP can be used for either an Async connection or via the Ethernet/FIPNET.

* * For ethernet on the J11 side, ANET must have loaded and started the Excelan card before RCVFIP started. RCVFIP does not use ANET apart from the startup nor does it interfere with other ANET traffic.

A table of the following format is used :

Async			Ethernet
de:scb04		de:eta
			no:9103
ct:[			ct:]
dq:junk3		dq:spike
dg:wir			dg:fea
hd:wires		hd:sys

Note that the ethernet connection needs a port number while the scp is on the device name. The CT field is a single chr only – pick from the ROM if not on the keyboard. The default queue and group must be specified separately.

Remember to build the table afterwards using BLDFIP:
CMD bldfip fip3
this creates a file called fip3.fip-tables-sys.

To run RCVFIP, put in the jstart file :
rcvfip fip3
or it can be run from the command line. In the latter case, if it is unable to run properly, RCVFIP will echo a message why.

rcvfip

RCVFIP program receives from either an SCP port or an ETA (ethernet).

To start either in the command line or in the jstart file :

* rcvfip pramfile

where pramfile is a file in tables-sys as added from BLDFIP program (qv)

TCVFIP may be fixed if necessary, or unfixed.

* /fx or /uf * – error in switch, bomb out

sndfip – Atex J11 communications

SNDFIP sends Atex files from the J11 to a Unixbox either via Ethernet/FIPNET or Serially. On the Unixbox program |ipxnet|ipxnetIPXNET receives on the Ethernet or [[Ipxv24][IPXV24]] serially.

SNDFIP can be run either from the command line on on a wrklsted queue. The syntax for sending is :

	* sndfip DEST<$f-$q-$g/b

where

DEST is both the parameter file in queue FIP-SYS of the same name : DEST-FIP-SYS and the destination on the Unixbox and so there should be an entry of the same name, forced UPPERCASE, in the tables/sys/USERS file on the Unxibox.

The optional switch can be (case insensitive) :
* x for text (default) - strip modes, markup and notes
* b for binary - send text of the file as is
* t for typsetter file - strip the 2 block setter hdr
* f for full text - send modes and markup too
* a for all-text-leave-quads-as-is
* n for all-text-leave-quads-as-is BUT strip modes

The parameter file is an ordinary atex scroll file. It holds the SCP port to use or the Internet address of the unixbox plus the queue and any FIP header information which will precede the filename. Each item is on a line on its own ended by a quad. All modes are valid (and stripped) including notes. Never HNJ or DOC !

Line 1 is either :
* the Internet address in decimal separated by '.'. After
* the last number, a comment can be added.
* OR the scp device and unit (must be even numbered !)

Line 2 is any optional FIP header fields. Any standing text is type 'as is'. Any header field text can be pulled in be specifying *XX when '*' is the presy chr and XX is the header field. Generally Line 2 should finish with the characters #SN: to pick up the atex filename correctly.

Note that FIP hdr fields DU, DA and SA are filled in automatically from the Atex header fields and appended to Line 2 :
* DU is the name of this parameter file
* DA is the contents of the NO field : forced lowercase.
* SA is the contents of the FR or OP field
* The last two are tunable in the CONST section below.

* eg: the contents of FAX-FIP-SYS could be :
* 192.94.43.201
* #FBIN:#SN:
* or: * scb01
* #SU:faxlog#HS*op#SN:

Note that SNDFIP automatically creates a destination filename in the normal FIP header style of :
* #DU:DEST#SA:(6 letter atex logon of 'last chged')#DA:(contents of NOtes hdr field)#SN:(atexfilename)

For ethernet traffic, the port number to send to on the Unixbox is defined as 91yy where yy is the atex system number. So an [[Ipxnet][IPXNET]] -p 91yy -l should be running on the Unixbox.

bldfip

BLDFIP takes an ordinary atex scroll file and builds the table for RCVFIP. The table is in the form :
* de:device * either scp port or eta for the ethernet ie de:scb04
The rest of the parameters are used only if the defaults need changing:
* hd:header * default header file without extention (defaults to sys)
* no:9001 * udp port no for ethernet only: must be unique and > 1024
* If you have 2 or more RCVFIPs waiting on an eta, they
* MUST have different udpnos.
* eq:ak * code for enq - when an enq turns up OUTSIDE a file send a SYN back
* ct:[ * precedent chr ; normally curly bracket/ (or 3/5)
* xx:ms * 2 letter code for broadcast following message
* xo:xo * code for overwrite existing file if found
* xp:xp * code for do not overwrite file (ie increment filename)
* sl:ca * look for slugname after 'ca'
* qu:qq * same for queuename
* bt:bx * begin text
* et:bi * end text
* xa:ac * pls ack this one
* xb:bn * this file is a binary file - do NOT add a 177b at the end
* m0:rn * modes 0 to 7
Note that other header fields are left as is. It is assumed that each paper takes care to send the other hdr fields it needs.

2.4.2 CSI

2.4.3 Others

2.5 File Redundancy

File redundancy is installed by :g
* * - add the program IPREDUN in the main SYSTEM file.
* * - make sure the queue spool/redun exists.
* * - add the file REDUN in tables/sys with the host names of the companion systems.
* * - add the %SZ flag in the /tables/route/ROUTINGFILE

The format of the latter is :g
* * %SZ:mainhost,redhost
where mainhost is the name of the host which normally processes the service and redhost is the redundant one.

The SZ field must be in the same ROUTE file for both machines in the same order.

If a host has several names, the main or boot name should be specified (do a gethostname to check). If you are using 2 ethernet adapters on the same Unixbox, take care that you specify the correct hostname for the network you want to use - otherwise the two unixboxes will never syncronise.

Note that IPREDUN should run on each of the Hosts marked in sys/REDUN.

----

3. Parameter Files

All files in tables are in standard text format and can be changed by any unix wp program. It is advised to keep names etc simple and not to use embedded control characters etc. Note that unix is case-sensitive so 'beans' is completely different to 'BEANS'. Where a space is required, it can be one or many spaces or tabs but not an end-of-line (CR or NL).

Comments unless stated otherwise, can be on single lines or at the end of a line and are preceded by a ';'.

The SYS Directory

3.1.1 SYSTEM

This file defines all programs, their settings and devices used. It is used by IP to control the system. Please use only alphanumeric NAMES (plus the '_'). Host is used to stop and start groups - especially so on startup where it's programs are automatically started - and is not material otherwise.

;;NAME	HOST	PROGRAM
;; comment line - always useful

extspt	roddy	wire	-s spider1 -p 1107 -n extspt
reuters	roddy	wire	-s spider1 -p 1108 -n reuters
pa	roddy	wire	-s spider3 -p 1308 -n pa
afp	roddy	wire	-s spider5 -p 1503 -n afp
uns	roddy	wire	-s spider5 -p 1505 -n uns
coi	roddy	wire	-s spider2 -p 1206 -n coi -x
m7500	roddy	wire	-s spider5 -p 1508 -n m7500 -x
tlx1	lacon	telex	-s spider1 -p 1104 -n 9419611
tlx3	lacon	telex	-s spider3 -p 1304 -n 9419613 -q indpnt
fax	 lacon	fax	-s spider5 -p 1509 -n fax -q indpnt
tph0521	lacon	ropey	-s spider1 -p 1103 -n tph0521 -t
pss1_3	lacon	ropey	-s spider1 -p 1102 -n pss1_3  -x
route	local	iproute
post	local	ippost
wheel	local	ipwheel
toatex	local	ipout

3.1.2 USERS

USERS holds all the destination data and is principly used by program [[Ipwheel][ipwheel]].

The destinations (DU) should be flush to the margin and end in a '='. Any Editorial system destination must be lower-case and end in a single digit (0-9).

See [[FipHdr][Fip Header]] and [[FipHdr2][Fip Header Overview]] doc for description of flags etc.

;	users-destination points for the ip
;
;The contents of the DU field are used to index the
;attributes of where the file is to go and how it
;gets there and in what format
;	DP isdestinationhost	-defaultthishost
;	DQ isdestination	-defaultnone
;	DC isdestinationchrset	-defaultsameassrc
;	plus specific flags for program	-default none
;
;atex wire queues
junk=		DP:atex8	DQ:junk-wir	DC:atex
pahome=		DP:atex8	DQ:pahome-wir	DC:atex
parlns=		DP:atex6	DQ:parlns-wir	DC:atex
poll=		DP:atex4	DQ:elect-wir	DC:atex
elect=		DP:atex4	DQ:elect-wir	DC:atex
m7500=		DP:atex8	DQ:htlx-wir	DC:atex
coi=		DP:atex8	DQ:htlx-wir	DC:atex
tlxlog=		DP:atex0	DQ:tmptlx-wir	DC:atex
faxlog=		DP:atex0	DQ:faxlog-wir	DC:atex
tandy=		DP:atex0	DQ:tandy-wir	DC:atex
mailbox=	DP:atex0	DQ:tandy-wir	DC:atex
;
TELEX=	DP:lacon	DQ:telex/enews/out	DC:unix
FAX=	DP:lacon	DQ:fax/enews/out	DC:unix
BOX=	DP:roddy	DQ:post		DC:sun
LIB=	DP:toby		DQ:lib/in	DC:sun

3.1.3 WAIT

This is a text file that is used by the WAIT function of program IP to check the contents of various queues. Text up to the '*' is displayed and then the queue (/fip/spool is assumed in front):

Looking at Queue:2brouted	  :*/2brouted
Looking at Queue:2go			 :*/2go
Looking at Queue:xchg			:*/xchg
Looking at Queue:2atex		  :*/2atex
Looking at Queue:2net			:*/2net
Looking at Queue:woops		  :*/woops
Looking at Queue:telex-out	 :*/telex/enews/out
Looking at Queue:telex-occ	 :*/telex/enews/occ
Looking at Queue:fax-out		:*/fax/enews/out
Looking at Queue:fax-occ		:*/fax/enews/occ
Looking at Queue:post			:*/post
Looking at Queue:lib-in		 :*/lib/in
Looking at Queue:woops		  :*//woops

If you want to look at queues that are not directly under /fip/spool you can
adda a hyphen after the star, amd then give the full path to the directory you
want ot monitor.

use a syntax like:

Looking at Queue:lib-in		 :*-/Users/lucy
Looking at Queue:woops		  :*-/external_ftp_mount_point/client

3.1.4 PEEK

If any TTY ports are being used, this file assigns UNIQUE numbers in the range 1 to 999 to devices that will be used by various input programs as a Pseudo Shared memory port for PEEK, the non-destructive line monitor to watch incoming traffic.

Note that terminal server (Spider) ports should NOT be in this file at all and start above 1024.

/dev/ttyb	11
/dev/ttya	10
/dev/ttyx00	500

3.1.5 GROUPS

GROUPS is used by IPPOST and the syndication suite MUI, SYN.

All items are on one line, no continuation lines are possible. Comments are allowable at any time using the ';'. Comments extend to the end of the line.

The syntax is :

;	comment
;group:(groupname) (tab) (client1) (tab) (client2)
group:lib	profile		fds	gw

;client: (clientname) (tab) type: (grouptype) (tab) [opt number: (no)]
client:profile	type:lib	; profile
client:boxab	type:pc		; gw from a pc to a pc name.
client:done	type:mail	; mailbox syndication
client:fip	type:fax	number:258-0510
client:tlxaaa	type:telex	number:91568467

All clients in a group must either be specified further on in the GROUPS file or for mailbox only, in the tables/mail/LOGON file.

Valid types are :
|lib|library|
|syn|syndication|
|pc|pc - strips NL to CR NL|
|bdcast|broadcast|
|mail|mailbox - uses the tables/mail/LOGON file otherwise|
|fax|fax|
|telex|telex|

3.1.6 SCAN

For a full description see under the [[Ipscan][IPSCAN] program which is the only user.

3.1.7 MONITOR

For a full description see under the [[Ipmon][IPMON] program which is the only user.

3.1.8 MAINT

For a full description see under the [Ipmaint][IPMAINT]] program which is the only user.

3.1.9 REDUN

REDUN file holds the hostnames of the redundant systems. It is used by [[Ipredun][ipredun]] (qv) to send ping-pong messages to and expect back from. On unixboxes with more than one ethernet adapters connected to different networks, it is important to specify the correct network for the redundancy.

3.1.10 ipvi files - TABLES_HOSTS and TABLES_DIFF_FILES

IPVI is merely an envelope for vi in that it automatically copies parameter files to companion systems. THis is important when modifying routing files for redundant systems - if the same file is different on the two unixboxes, the same message will in fact be considered different !

TABLES_HOSTS is a list of hosts to backup files to; each host should be specified exactly as in /etc/hosts or smit and on a single line.

TABLES_DIFF_FILES is a list of files which are different on each system. Often all parameter files are the same. But some sites have differing SYSTEM files tailored for each unixbox. If there is an entry in TABLES_DIFF_FILES, modifications should be made on the main system for that file and it is copied to its companions with a name of FILENAME.mainsystem . Hence for system failures there is a copy/backup that can be used to recreate the original.

3.2 ROUTE directory

The 'route' queue contains the individual routing files. These can be one per 'wire' input or several inputs can use the same routing file (for example all telexes will use the same routing file).

A semicolon signifies a comment.

The defaults are specified first :

  • '%DU' signifies the default destination user.
  • '%SC' signifies the default character set/xchg file.
  • '%SZ' (optional) is the redudant systems for this wire. The main host is specified first then a comma then the redundant host ie:
	%SZ:mouse,elephant

Then a series of condition lines are specified. Each line contains one and only one test on one of the header fields - category and priority are the usual ones. A '*' signifies any match. The line starts with a number which is the level. When a match is found, the program either desends to the next level OR, if there is a '>xyz', it will take that to be the destination - the DU. Multiple destinations- which imply multiply copies - can be specified by using the '+xyz' beforehand. Whereas IPROUTE stops looking down the table if it finds a '>user', it will save and continue when a '+differentuser' is found.

When a Users is given, you can specify a specific program you have written to be run at this point by using a'!' and the program name. The filename is added after the string you give. The new program should replace the inputfile or the DU should be 'delete' to strip it.

Header fields are dependent on the incoming program. WIRE uses the following fields:
(see the relevant IPTC 7901 or ANPA 1312 document)

  • N = service number
  • S = service designator
  • L = selector field
  • P = priority
  • C = category
  • F = fmt (anpa)
  • K = keyword
  • W = no of words
  • V = version
  • R = reference field

Example :

#	Routing table for PA
%DU:junk
%SC:PA
# home
	1	c=H*	>pahome	# home
	1	c=P*	>hoc	# parly
	1	c=Q*	>pahome	 # advance
# sports results
	1	c=R*	+sptsprt	  # copy to the sportsPR
		2	c=RDA	>swire  # averages
		2	c=RDB	>swire  # performances
		2	c=RDC	>crick  # cricket summaries
		2	c=RSD	>crick  # cricket fixtures
		2	c=R*	+scopy  # one copy to scopy
		2	 c=R*	>swire  # another to swire.
	1	c=S*

# send race cards
		2	s=L	>rac!/fip/race -f AP -c

The latter will start program /fip/race -f AP -c thisfile

3.3 XCHG and CHRSET directories

These two queues contain the chrsets and xchg routines for the [[Ipxchg][ipxchg]] program. [[Ipwheel][IPWHEEL]] passes a file to IPXCHG with the name of the character stripping routine to use. This is made from teh Source character set and destination. If you specified a SC of 'AFP' in your routing files and a destination chrset of 'atex' in the USERS file then IPXCHG will look to for 'AFP2ATEX' in the xchg directory.

See also [[Ipxchg][IPXCHG]] in the programs section.

3.3.1 CHRSET

[[Ipxchg][ipxchg]] knows about several chrsets already - ascii, baudot (actually 2 chrsets : cno2let and cno2num) and ebcdic - plus 'none' which is set to zeros. To create a new one chrset, crate a new file in CHRSET (filenames are always uppercase). The first line should contain the name of a known set or 'none'. Any other lines should contain 2 characters only : a position in the chrset and what you wish to use. These chrs can be specified either :

as letters : abc 67 etc

as octal numbers : \001 \176

or as Unix like escape chrs : \r \n

Please do NOT use any space chrs - Space, tab, form feed - to signify a chr : use either \n notation or spell it out in octal.

Chrset is used for SINGLE chr translations only. Any supplementary characters are ignored.

Note that as the semicolon signifies a comment you must specify its octal number is you want to change this.

3.3.2 XCHG

Each xchg file is of the following format :

	c:ascii
	x:\r:\n			# replace CR with LF
	x:\043: (pounds)	 # replace '#' with pounds
	x:\03601\034:\n*r1*
	x/::::/:/		# replace 4: with :
	z:FCC:

The first section must specify a chrset - default is none. The format of the line must be kept as above. Hashes are trated as comments so if you need one, use \043 which is the octal representation.

Note that IPXCHG uses descending order for priority. Changing the actual sequence of xchgs can change the output significantly.

Note also that the chrs are translated into the correct chrset BEFORE and xchg is done.

When using a 5 bit language, an alternate chrset can be specified eg:

	c:cno2let
	e:\027		# escape to 2nd chrset
	d:cno2num
	f:\031		# escape back

Single character escapes are activated by (the number are for example only) :

	s:\176

or if in the 2nd chrset and want to escape to the first :

	t:\177

Flags are allowed as a 'z' command, followed by the actual flag :

	z:FCC:

This will enable idiot upper and lower case translation.

3.4 OUT and DEVICE directories

Both these directories hold files for us with IPOUT program which sends files out of the FIP system. See also [[Ipout.html][IPOUT]] in the programs section.

3.4.1 OUT directory

The OUT directory holds at least one file - called FORMAT - which describes the message format to be used for transmissions. This is the default file. Other files can be created in the same way and called using the 'DF' (destination format) header field which can be specified usually in the sys/USERS file for that destination (DU).

An example of a format file follows :

;; comment
%DQ junk-wir
%XW 99
%XR no reference available
pre: < this is the preheader>
hdr:[sl \DN [qu \DQ [no \XR [by \$A [wr \XW [ti \$h:\$n [st
tlr: \002 [et [et

A ';' will start a comment. Each line is terminated with a LF or NL; there are no continuation lines.

Defaults for certain fields can be setup using the '%' as a precedence character. In the example above, DQ has been set to ' junk-wir' with a leading space. Only text can be specified in default fields - no variables. If a default is not specified and the header field is NOT found, the field is ignored.

Each file is transmitted with a pre-header, then the header, then the text and finally the trailer. Three letter codes with a ':' are used to specify which section is which - pre:, hdr: and tlr:.

For each of these sections you can insert fixed text, unix escape chrs, system variables and header fields.

  • Fixed text can be any ascii printable chr except '\'.
  • Unix escape chrs are preceeded by a '\' and are lower-case : \n = Linefeed(NL), \r = Carriage Return (CR), \s = a space, \b = backspace, \t is a tab, \f is formfeed.
  • System variables are preceeded by a '\$' : \$A is an Atex-like orig string (SOURCE;10/11,12:31), '\$s' is a system generated sequence number, \$y is the year is '99' format, simularly 'm' is month (Mar), 'd' is day (99), 'h' is hour (99) and 'n' is minutes (99).
  • Header fields are the normal IP fields. Where information from the source header of a wire file is required, then use \Xx notation where x is the field in question. For the Reference field : use \XR, for the keyword : \XK, word length is \XW etc. (See section 3.2 for a list of source header fields)

3.4.2 DEVICE directory

For every physical destination (DP) that is connected via IPOUT, there should be a (series of) files describing which ports should be used for which DP. For example when transmitting to Atex there must be one file called ATEX which states the DP and the actual device :

atex2	* /dev/ttyx00
atex3	* 1309
atex4	* /dev/ttyx03
atex6	* /dev/ttyx04
atex7	* 1109
atex8	* /dev/ttyx06
atex10  /dev/ttyb
atex24	* 1209
; This is Atex.1357 - sending copy to systems 1, 3, 5 and 7

No extra characters or comments should be in these files except the last line may start with a comment (semicolon) and describe the filename and what it routes to.

For spiders, only the port number should be specifed.

For the FIPNET connections via the ethernet to RCVFIP, only the hostname as appears in /etc/hosts should be specified.

There should be several ATEX tables each with a '.99' extension where 99 is any digit, for when various Atexes are down or unhealthy.

*WIRE directory

3.5.1 VWIRE and ANSBCK files

Parameter files for VWIRE are held in this directory. These are the setup files and also any Answerback text files. Please refer to [[Wire][wire]] in the program section for more information.

3.5.2 KEYWORD

Please refer to [[Ipkeyword][IPKEYWORD]] in the program section for more information.

3.6* MAIL directory

Files LOGON and BOX are described in the programs section under [[Ipbox][IPBOX]] which uses them.

3.7 STTY directory

Setup the characteristics of any port using STTY. Normal settings are :

(stty 9600 cs8 -parenb clocal cread cstopb -brkint -istrip -icrnl -ixon -opost -echo -isig -icanon min 0 time 1) > /dev/ttyx
	where x are the actual device number.

3.8 * SETUP directory

3.8.1 EASYLINK

Please refer to [[Ipmer][ipmerc]] in the program section for more information.

3.9 HELP directory

Help files are ordinary text files. The file called IP is the default when the IP 'h' or '?' command is used. Other files may be added at the your discretion.

----

4. Queues

All FIP queues are found under their home directory. This is usually /fip but may be any other path as defined at installation time.

All queues in FIP are lower-case with no special symbols.

Main system queues are :

/fip/bin :

Queue holding all binary programs.

/fip/fix :

Internal queue for system files; pls do not touch.

/fip/help :

Text help files reside in this directory.

/fip/log :

Holds the main system Transaction log (called ALL) plus the logs for the sub-systems such as SYN and IN. There are various queues for archive purposes and supplementary logs.

/fip/log/data :

Holds copies of every data file in large files with the name : day-of-year_source. The day-of-year will actually be day-of-year minus 1, as the 1st January is day 0.
These files can be 'more'd, but should not be 'vi'd as 'vi' can destroy non-ascii characters. The files are used by the [[Ipresend][ipresend]] command in IP.

/fip/spool :

This is the main path to all the queues and data files for all IP-based systems.

/fip/spool/2brouted :

All incoming files from such programs as 'WIRE' and 'TELEX' programs are dumped here for [[Iproute][IPROUTE]] to decide where the files should be sent according to the incoming header.

/fip/spool/2go :

[[Ipwheel][IPWHEEL]] scans this queue and will attempt to send the each file to its destination(s) which may be on this system (file is sent directly to the queue specified), on another Sun or an Atex. If not correctly addressed, IPWHEEL sticks the file in 'woops', the intercept queue. If the file needs translating into atexese or telexese, it is sent from [[Ipxchg][IPXCHG]] to work its magic.

/fip/spool/xchg :

[[Ipxchg][IPXCHG]] will translate files for the Atex according to the character set and stripping algorithm you have specified as the source and destination chrset

/fip/spool/2atex :

Files arriving here are processed by either [[Ipout][IPOUT]] or IPCCM which checks the 'DP' field for which Atex system to send it to; and uses ' to a particular destination or device (that means usually a destination like 'swire' or a device like 'atex6').

/fip/spool/fax/xxx/out:

Faxes being sent/to be sent are in this queue.

/fip/spool/fax/xxx/occ:

Faxes go here when the remote fax is occupied or busy, they are re/fipsent 10 mins later.

/fip/spool/telex/xxx/out:

Telexes being sent/to be sent are in this queue.

/fip/spool/telex/xxx/occ:

Telexes go here when the remote telex machine is occupied or busy, they are re-sent 10 mins later.

/fip/spool/in :

Various queues used by the Remote-database automatic access sub-system. Please refer to the documentation on that system for full details.

/fip/spool/syn :

Various queues used by the Syndication sub-system. Please refer to the documentation on that system for full details.

/fip/spool/woops :

The Intercept queue - Any file that any program has problems with, will be sent here for manual intervention. Usually this will be either [[Iproute][IPROUTE]] or [[Ipwheel][IPWHEEL]] which have found a slight irregularity in one of the routing or destination parameter files. Normally there should be no items at all in this queue, as all incoming messages should be routed to their default destinations.

/fip/spool/redun:

Spooled queue for all the redundant files that need to be processed by [[Ipredun][IPREDUN]].

/fip/spool/pc:

Working queues for all pc traffic

/fip/tables :

These directories contain the main site-dependent parameter files that are described in the rest of the MANAGERS GUIDE.

/fip/x :

An internal queue which for FIP programs. Pls ignore. It is cleared each time the system is re-booted. (part of /etc/rc or /etc/rc.local).