Fip - Internet Delivery Module - Features and Options

Fip Internet Delivery Module - Features and Options

This document covers the Features and Options available for clients receiving data from the Fip Internet Delivery Module

Introduction

The FingerPost Web Delivery Module allows delivery of any type of data files over TCP/IP via a combination of internet, intranet, VPN etc.

A selection of Delivery Methods are available that cover many of the different techiques used to get data from a supplier to a client.

The main purpose is to find the easiest and quickest method, hopefully allowing the client to use existing programs to receive the data.

Where existing programs do not exist, the FipRemote program may be downloaded and used to start testing quickly while the proper Feed Handler is written.

The actual Message Format, File Type an/or Character Sets can also be tuned/modified for each individual client. So the data should arrive 'ready-to-use'.

What is included and What will the Client have to do

Depending on the actual Delivery Method chosen, the responsibility for the feed ends at the point of entry for Client.

For FTP, it ends with the file in the folder or directory chosen by the client.

For SMTP/email, it ends with the reception of the data in the Clients's email server.

For a TCP feed, it ends with the socket connection that the Clients's software has attached.
If using the FipRemote program, the data will be stored as separate files in a folder on the Client's system (BUT please note that there is NO implied responsibility for FipRemote from either FingerPost or the Supplier).

What it is not.... is any software for subsequent processing the files at the Clients.

Standard tools for uploading data into a DataBase or an Editorial System or a Content Managment System are available from a variety of software vendors - including FingerPost - but are NOT part of this package.

If you are interested in the FingerPost Base Module for handling the incoming feed and the DataBase Uploader (ip2db), please check the web site www.fingerpost.co.uk for further information.

Options

Push or Pull
Delivery Methods
Permanent Connection or Start-Stop
Envelopes and Formats
Allowable File Types and Character Sets for the different types of traffic
Logon and Password
Encryption
Compression
Encoding
Check Messages
Redundancy - duplicated data flows
Statistics
Test Channels
Repeats
Acknowledging Receipt of Files
Note on TCP connections

1. Push or Pull

Who Starts the connection - the Client's system or the Supplier's ?

If both systems are on a VPN or are outside their respective FireWalls, there is no problem - either side can 'see' the other and either can start the process.

However in practise either one is labelled the initiator and ONLY that system can start. However if one system is behind FireWall ..

either a hole must be punched thru to that system
or that system is used to start the connection

2. Delivery Methods

A wide range of delivery methods are available.

These include

TCP socket - this is sometimes called 'Telnet Binary with no escapes' or 'Raw Telnet'
HTTPD - where the data is sent to a Web Server like Apache as a 'POST'.
A CGI program or script, usually in Perl, is used to accept, process and forward each file.
FTP
SMTP - email
ICE (soon)
Postman Pat

The most common are

TCP sockets for large data volumes
email (SMTP) or FTP for lower volumes.

See the Note at the end of this document on TCP sockets if that is applicable.

3. Permanent Connection or Start-Stop

Depending on the volume of data - number and size of files - the Supplier and/or Client may decide NOT to dedicate a 24 hour dedicated port but offer access between certain times/on certain days or force the client to connect only occasionally.

In this case the Client gathers any data and drops the connection immediately after.

This option is generally set by the Supplier.

4. Envelopes and Formats

Virtually all known Transmission Formats can be handled.

Some standard TEXT formats are available to be used immediately, including :

Note that any other format can be created quickly.

Any format, including the ones itemised above, can be tuned for any client who needs specific and particular changes without affecting any other client.

5. File Types and Character Sets

For TEXT files, all data can be forced to a specific character set and style as specified by the Client - usually that of his main computer or application.

GRAPHIC and other BINARY files may also be converted to a type and/or resolution the Client can handle easily.

6. Logon Sequence and Passwords

Certain Formats and data flows are on available once a logon/password routine has been completed.

Please Note that the default is that all connections to the Supplier's Server require a Logon.

For FTP, ICE where there is an implicit Logon sequence, then that is used.

However for general broadcast newswires, a small handshake is used.
Immediately after connection, the client sends a string in the format :
(pipe) (logon) (pipe) (password) (CR or NL)
where

pipe is the bar character (dec 124)
logon is the string given by the Supplier
password is the string given by the Supplier
CR or NL is a Carriage Return and/or a NewLine

The Server will reply either of the two strings :

(ack) Logon Accepted
(nak) Logon NOT Accepted

Where ACK is a single chr (dec 006) and NAK is (dec 021)

7. Encryption

Where agreed, the data part (and in some cases, the whole envelope) may be encrypted using standard publicly-available routines or packages.

8. Compression

Where agreed the data part (and in some cases, the whole envelope) may be compressed using standard publicly-available routines or packages. Large BINARY files are almost certainly handled in this way.

9. Encoding

While general TEXT files - especially if using old-style Message Formats - are rarely encoded,

BINARY and XML/HTML files sent within XML Message formats or Delivery Envelopes like NewsML or NITF are usually encoded Base64.

Other encodings are available on request.

10. Check Messages

Check Messages may be optionally sent for open, permanent connections at regular time intervals.

This are used to satisfy the Client that, technically everything is still working, but no files relevant to the Client have been generated in the last time interval.

The actual time interval is normally determined by the Client.

11. Redundancy - duplicated data flows

The Supplier can offer Clients dual, redundant feeds. Where possible, Key Header fields in the Message Format or envelope are marked so give uniqueness.

However it is the Client's reponsibilty to cope with the multiple instances of the data.

12. Statistics

Statistics on connections, failures/errors and data flows will be available in the next release of the software.

13. Test Channels

For Clients requiring Test data - either for tuning an existing feed or perhaps sampling a different envelope or format - may use a second channel setup by the Supplier to handle the tests.

14. Resending current data

To cover for local disasters, Clients may request repeats of all or a selection of already-sent files.

This should be either by ringing/emailing the Supplier OR, where available, using the web browser interface.

Note that for clients with permanent, dedicated connections, when the client reconnects after being down for a period of time, if there are any files that should have been sent, they will be sent before any other traffic.

The default period is from Midnight only but can be tuned at the client's demand.

This option may be disabled completely if the client does not require it.

15. Acknowledging when a File has been Received

Some Delivery Methods like FTP will automatically detect if the client has not received the file correctly.

Others - such as broadcast feeds uch as ANPA 1312, IPTC 7901 or IATA - may be slightly out of sync if the connection drops AFTER the supplier has sent the last packets but BEFORE the client has received them.

In this case, there is an option from the Server to wait after sending for an Acknowledgment.

The syntax for these acks are :
(ack) (Story Name and Number) (CR and/or NL)
Where ACK is a single chr (dec 006) and Story Name and Number is the unique identifier for that file. (Usually this is the Service Designation and Item Number combined).

16. TCP Sockets

Notes on the TCP socket connection :

Use ordinary sockets to open, send, recv and close (WINNT closesocket) the connection.
The code can be in anything which supports sockets including 'C', Java, Perl, VB
If Logons are required, immediately the connection has been made, the logon and passwords must be sent before waiting for data.
For Permanent connections, you MUST handle exceptions such as when the remote server drops the connection with no warning - for example for a reboot or crash! If the client is the passive end, it must immediately go into a Listen state; if active, it should try to re-connect. If the connection fails, wait 5 seconds and try again.
For Start-Stop connections, the server will disconnect automatically when there is no more data to send. Again the client must be able to restore state after disconnection.
Any Statistics will be sent in-band ie in the text Message Format chosen, but will have certain (agreed/documented) Header fields which may be used to stream Stats files from the main data feed.
Repeats will be sent in-band ie in the text Message Format chosen. Note that the Repeats may NOT be flagged for some Message Formats and will be identical to the original data.
Please refer to the Logon section of this document if that is required.
Please refer to the Data Acknowledgement section of this document if that is required.