icewire

useHttp
POST
  icewire

Icewire is an ICE client which connects to a server and collects packages for
the subscriber.

These packages can be updates of weather, financial data, sports results,
backup for Wire services if the satellite is down, graphics, software. In fact
most things.

The program can be used either :
    - on a timed basis to get regular known pages.
    - on demand by sending a file into spool/icepoll with the FipHdr field DF set
to the parameter file required.

There is a TUNING mode to be used for setting up a new link and trying to clean
up the relevant parameter file WITHOUT sending (possibly) live data to the
required destination.
    - This shows the handshaking that taking place
    - Use this in particular when accessing the server via a proxy
    - To run, choose your parameter file in tables/icewire and run 'icewire'
manually in a window:
        icewire -T AUS.STOX | more      for prompt before calls
    or  icewire -A -T AUS.STOX | tee aussies    for no prompting

There are Two (sometimes three) types of parameter file :
    1. Main Parameter file which sets up the polling of certain pages at set times
(if any).
    2. A Page Description file for each site/page accessed.

----- Main Parameter file -----

The syntax of the Main Parameter File - by default tables/icewire/ICE_POLL :
    ; comment line
    poll:(pram file)    day:(MTuWThFSaSu)   time:20:30  mustget:

In detail, the 'poll' keyword :
    Pram file is the name of the Page Description file - see below for its syntax
    day:    Day of week to run the job :
            M   Monday
            Tu  Tuesday
            W   Wednesday
            Th  Thursday
            F   Friday
            Sa  Saturday
            Su  Sunday
            X   Every day.
            Case is NOT important.
            Commas (but NOT spaces) may be used to separate.
            Default is every day.
    either
    time:   Time of the day on 24 hour clock.   Default is 18:00.
    or
    every:  interval between grabs          Default: none
        every: (mins)   [(optional) start:(starttime) end:(endtime)
        every:30    start:07:30 end:19:00
        The minimum interval is 10 mins.
eg:
    poll:AP     day:ALL     time:20:10
        Get the Page file tables/icewire/AP every day at 20:10
    poll:Forex  day:MTuWThF time:16:30
    poll:Forex  day:MTuWThF time:16:40
        Get the Page file tables/icewire/FOREX every week day at 16:30 and 16:40

There can be none or up to 200 polls in the main parameter file.
Note that the page is grabbed ONLY if the program is running.

----- Page Description Parameter files -----

The individual Page description parameter files are also in tables/icewire. The
syntax of these are :
    ; comment start with a semi colon like this

MANDATORY
    url:    Full url of the page.               default: none
        There MUST be one and only one 'url:' specified.
        You can also specify the page, cgi and any subparameters.
        eg  url:www.fingerpost.co.uk
            url:www.big-press-org/sports/baseball/index.htm
            url:www.marketlook.co.uk/scripts/Summary.dll?HandleSummary

    dest:   Fip Destination for the files           default: ICEDATA
        This is the 'DU' FipHdr field as per the USERS file.
        eg  dest:w3saves

OPTIONAL:
    port:   Port number of the Remore Server.       default: 80
    nofiphdr: Do NOT add a Fip Hdr to the file.     default: yes pls
    source: Fip Source of the files. (FipHdr 'SU').     default: ICE_POLL
        Unless 'noarchive' is specified, all data files will be archived
        under this name in log/data.
    noarchive: Do NOT archive these files in log/data.  default: archive
    httphdr: Extra lines of HTTP header you may need.   default: none
        Remember to add a NL at the end of each line.
        eg  httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n 

    post:    Post a Form                    default: get url
        see below for processing a form using method=POST.
    filename: Filename for the output file in FipSeq.   default: ICE\$Z
        If this does NOT start with a '/' it is left under the
        Output Queue as specified on startup (default spool/2go)
        eg  filename:AFP\$d.\$z
    wild: (FipSeq)  Character used as a Wild String for default: '*'
        'matchlinks/ignorelinks'.
        eg  wild:\377
    singlewild: (FipSeq) Character used as a single     default: '?'
        Wild chr for 'matchlinks/ignorelinks'.
        eg  singlewild:!
    number: (o|d|h) Number system for FipSeq        default: octal
        octal, decimal or hexidecimal
        The following are all equivalent :
            number:octal
            before:\040
            number:decimal
            before:\032
            number:hex
            before:\020
    before: FipSeq String to add before any data.       default: none
    after:  FipSeq String to add after any data.        default: none
    script: Script to run on ths data of the incoming file. default: none
    log:    FipSeq custom logging for the item log.
    extra:  Extra FipHdr fields to be added         default: none
        to the output file. To separate FipHdr fields, pls use a '#'.
        eg  extra:ZH:NYNZ#DI:Headline News#QZ:333
    keep-alive: yes/no                  default: no
        Just that ! default:no
    http-version: 1.0 or 1.1                default:1.0
    only-get-if-modified:   (FipSeq message if not found)   default: get
        This will check the remote server for the time the page was
        last modified. This does not work with old servers and some
        set to HTTP/1.0.
        If modified since, the page is read
        If not, the optional message is sent
        If there is no message, no data is sent - just a note in the item log
    one-output-file: Put ALL data in a single output file.
        The default is one file per page/access
        Use this with 'values' to create a single output file.
        This ONLY uses the FipHdr of the first file if 'values' have been specified.
    end-of-document: Where a site is sending really really crap HTML - or XML
        use this to state what the last tag.
        For no checking at all : end-of-document:
        Default:        end-of-document:</HTML>
    proxy-server: If using a proxy, these are the name and port to aim at.
    proxy-port:
    proxy-logon: This is the logon and password to get thru the firewall
        if required. The format is (logon) (colon) (password) and is
        converted to base 64.
        proxy-logon:Y2hyaXMuaHVnaGpvbmVzOnBhbnRoZXIK=

        To generate :
            echo -n "logon:password" | sffb64 -i 
        eg  echo -n "chris:sleekpanther" | sffb64 -i
        gives   Y2hyaXM6c2xlZWtwYW50aGVy
            proxy-logon:Y2hyaXM6c2xlZWtwYW50aGVy=

Note that in the FipHdr - unless the 'nofiphdr' keyword has been requested, the
following fields will be filled in :
    Day and time in the normal HH,HD,HY etc fields
    ST  host
    SP  port
    SF  path/filename

Input Parameters (all optional) :
either  -i : scrolled queue             default: spool/icepoll
        This allow a variety of parameter files to be run
or  -1 : Run a single time and exit         default: spool
        The parameter is the name of the individual parameter file
        in tables/icewire (ie NOT The top or main parameter file)
or  -T : Tuning mode                default: spool
        Display links and data for the page requested. Runs only that page
        and then exits.
        The parameter is the name of the individual parameter file
        in tables/icewire (ie NOT The top or main parameter file)
    -A : In Tuning mode, do NOT prompt      default: prompt
        before searching a link

    -o : output queue in 'spool'            default: spool/2go
        This is ignoring in Tuning mode.
    -w : Wait in seconds between accessing links.   default: 5
    -z : parameter file in 'tables/icewire'.    default: ICE_POLL
    -v : Print the version number and exit

---- Other Notes ----

-- Proxies Proxies Proxies Proxies Proxies Proxies Proxies Proxies Proxies
Proxies

When running through a proxy server, you will need :
    1. hostname of the proxy server
    2. port number on the proxy server if it is NOT port 80
    3. (optionally) a logon and password

If this information is NOT available, normally you can find it easily from any
PC or Mac on the internal network using a browser like Netscape or IExplorer.

Start a NEW copy of either of these.  - It must be a new copy to check on
logons etc.

Under 'Preferences' or 'Internet Options' there should be a 'Connections'
section and under that, the host name or ip address plus host name of any proxy
used.

Note that often the main Fip server is NOT running DNS and will not be able to
resolve external hostnames, so the IP address must be used in this case.

Enter these values in the Fip parameter file as :
    proxy-server:195.13.83.7    (no default)
    proxy-port:412          (this defaults to port 80)

Use the Browser to attempt to access a ice site outside the firewall - like
'www.fingerpost.co.uk'.

If you are asked for a password to get through, you will probably need to add a
'proxy-logon' parameter too unless the keeper of the Firewall has made a hole
through just for you.

The data for 'proxy-logon' is in base64 in the format (logon) (colon)
(password).

Use 'sffb64' to generate this string :
    On a Sparc  echo -n "chris:magicman" | sffb64 -i
    On Linux    echo "chris:magicman" | sffb64 -i
    On Winnt    type "chris:magicman" | sffb64 -i

    proxy-logon:Y2hyaXM6bWFnaWNtYW4===

The actual 'You need to Logon, Pal' message is a '407 Authentication Required'
message.

-- Others Others Others Others Others Others Others Others Others Others Others

--Where a logon and password is requested as part of the Browser - ie a pop-up
from Netscape or IExplorer, NOT an HTML form - you will need to add a
'Authorization' line. This will be true if you get a message like :
    HTTP/1.0 999 Authorization failure
        ... etc etc etc ...
    Assuming you know your logon and password :
    1. Use uuencode or sffb64 to generate a Base64 string
        echo -n "logon:passwd" | sffb64 -i
    2. Add an extra line to the parameter file with the result of the sffb64 line
using 'httphdr'.
        Syntax: Authorization (colon) (spc) Basic (spc) (Base64 of logon:password)
(\n FipSeq for NL)
        Eg  httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n 

(copyright) 2016 and previous years FingerPost Ltd.