useHttp POST icewire Icewire is an ICE client which connects to a server and collects packages for the subscriber. These packages can be updates of weather, financial data, sports results, backup for Wire services if the satellite is down, graphics, software. In fact most things. The program can be used either : - on a timed basis to get regular known pages. - on demand by sending a file into spool/icepoll with the FipHdr field DF set to the parameter file required. There is a TUNING mode to be used for setting up a new link and trying to clean up the relevant parameter file WITHOUT sending (possibly) live data to the required destination. - This shows the handshaking that taking place - Use this in particular when accessing the server via a proxy - To run, choose your parameter file in tables/icewire and run 'icewire' manually in a window: icewire -T AUS.STOX | more for prompt before calls or icewire -A -T AUS.STOX | tee aussies for no prompting There are Two (sometimes three) types of parameter file : 1. Main Parameter file which sets up the polling of certain pages at set times (if any). 2. A Page Description file for each site/page accessed. ----- Main Parameter file ----- The syntax of the Main Parameter File - by default tables/icewire/ICE_POLL : ; comment line poll:(pram file) day:(MTuWThFSaSu) time:20:30 mustget: In detail, the 'poll' keyword : Pram file is the name of the Page Description file - see below for its syntax day: Day of week to run the job : M Monday Tu Tuesday W Wednesday Th Thursday F Friday Sa Saturday Su Sunday X Every day. Case is NOT important. Commas (but NOT spaces) may be used to separate. Default is every day. either time: Time of the day on 24 hour clock. Default is 18:00. or every: interval between grabs Default: none every: (mins) [(optional) start:(starttime) end:(endtime) every:30 start:07:30 end:19:00 The minimum interval is 10 mins. eg: poll:AP day:ALL time:20:10 Get the Page file tables/icewire/AP every day at 20:10 poll:Forex day:MTuWThF time:16:30 poll:Forex day:MTuWThF time:16:40 Get the Page file tables/icewire/FOREX every week day at 16:30 and 16:40 There can be none or up to 200 polls in the main parameter file. Note that the page is grabbed ONLY if the program is running. ----- Page Description Parameter files ----- The individual Page description parameter files are also in tables/icewire. The syntax of these are : ; comment start with a semi colon like this MANDATORY url: Full url of the page. default: none There MUST be one and only one 'url:' specified. You can also specify the page, cgi and any subparameters. eg url:www.fingerpost.co.uk url:www.big-press-org/sports/baseball/index.htm url:www.marketlook.co.uk/scripts/Summary.dll?HandleSummary dest: Fip Destination for the files default: ICEDATA This is the 'DU' FipHdr field as per the USERS file. eg dest:w3saves OPTIONAL: port: Port number of the Remore Server. default: 80 nofiphdr: Do NOT add a Fip Hdr to the file. default: yes pls source: Fip Source of the files. (FipHdr 'SU'). default: ICE_POLL Unless 'noarchive' is specified, all data files will be archived under this name in log/data. noarchive: Do NOT archive these files in log/data. default: archive httphdr: Extra lines of HTTP header you may need. default: none Remember to add a NL at the end of each line. eg httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n post: Post a Form default: get url see below for processing a form using method=POST. filename: Filename for the output file in FipSeq. default: ICE\$Z If this does NOT start with a '/' it is left under the Output Queue as specified on startup (default spool/2go) eg filename:AFP\$d.\$z wild: (FipSeq) Character used as a Wild String for default: '*' 'matchlinks/ignorelinks'. eg wild:\377 singlewild: (FipSeq) Character used as a single default: '?' Wild chr for 'matchlinks/ignorelinks'. eg singlewild:! number: (o|d|h) Number system for FipSeq default: octal octal, decimal or hexidecimal The following are all equivalent : number:octal before:\040 number:decimal before:\032 number:hex before:\020 before: FipSeq String to add before any data. default: none after: FipSeq String to add after any data. default: none script: Script to run on ths data of the incoming file. default: none log: FipSeq custom logging for the item log. extra: Extra FipHdr fields to be added default: none to the output file. To separate FipHdr fields, pls use a '#'. eg extra:ZH:NYNZ#DI:Headline News#QZ:333 keep-alive: yes/no default: no Just that ! default:no http-version: 1.0 or 1.1 default:1.0 only-get-if-modified: (FipSeq message if not found) default: get This will check the remote server for the time the page was last modified. This does not work with old servers and some set to HTTP/1.0. If modified since, the page is read If not, the optional message is sent If there is no message, no data is sent - just a note in the item log one-output-file: Put ALL data in a single output file. The default is one file per page/access Use this with 'values' to create a single output file. This ONLY uses the FipHdr of the first file if 'values' have been specified. end-of-document: Where a site is sending really really crap HTML - or XML use this to state what the last tag. For no checking at all : end-of-document: Default: end-of-document:</HTML> proxy-server: If using a proxy, these are the name and port to aim at. proxy-port: proxy-logon: This is the logon and password to get thru the firewall if required. The format is (logon) (colon) (password) and is converted to base 64. proxy-logon:Y2hyaXMuaHVnaGpvbmVzOnBhbnRoZXIK= To generate : echo -n "logon:password" | sffb64 -i eg echo -n "chris:sleekpanther" | sffb64 -i gives Y2hyaXM6c2xlZWtwYW50aGVy proxy-logon:Y2hyaXM6c2xlZWtwYW50aGVy= Note that in the FipHdr - unless the 'nofiphdr' keyword has been requested, the following fields will be filled in : Day and time in the normal HH,HD,HY etc fields ST host SP port SF path/filename Input Parameters (all optional) : either -i : scrolled queue default: spool/icepoll This allow a variety of parameter files to be run or -1 : Run a single time and exit default: spool The parameter is the name of the individual parameter file in tables/icewire (ie NOT The top or main parameter file) or -T : Tuning mode default: spool Display links and data for the page requested. Runs only that page and then exits. The parameter is the name of the individual parameter file in tables/icewire (ie NOT The top or main parameter file) -A : In Tuning mode, do NOT prompt default: prompt before searching a link -o : output queue in 'spool' default: spool/2go This is ignoring in Tuning mode. -w : Wait in seconds between accessing links. default: 5 -z : parameter file in 'tables/icewire'. default: ICE_POLL -v : Print the version number and exit ---- Other Notes ---- -- Proxies Proxies Proxies Proxies Proxies Proxies Proxies Proxies Proxies Proxies When running through a proxy server, you will need : 1. hostname of the proxy server 2. port number on the proxy server if it is NOT port 80 3. (optionally) a logon and password If this information is NOT available, normally you can find it easily from any PC or Mac on the internal network using a browser like Netscape or IExplorer. Start a NEW copy of either of these. - It must be a new copy to check on logons etc. Under 'Preferences' or 'Internet Options' there should be a 'Connections' section and under that, the host name or ip address plus host name of any proxy used. Note that often the main Fip server is NOT running DNS and will not be able to resolve external hostnames, so the IP address must be used in this case. Enter these values in the Fip parameter file as : proxy-server:188.8.131.52 (no default) proxy-port:412 (this defaults to port 80) Use the Browser to attempt to access a ice site outside the firewall - like 'www.fingerpost.co.uk'. If you are asked for a password to get through, you will probably need to add a 'proxy-logon' parameter too unless the keeper of the Firewall has made a hole through just for you. The data for 'proxy-logon' is in base64 in the format (logon) (colon) (password). Use 'sffb64' to generate this string : On a Sparc echo -n "chris:magicman" | sffb64 -i On Linux echo "chris:magicman" | sffb64 -i On Winnt type "chris:magicman" | sffb64 -i proxy-logon:Y2hyaXM6bWFnaWNtYW4=== The actual 'You need to Logon, Pal' message is a '407 Authentication Required' message. -- Others Others Others Others Others Others Others Others Others Others Others --Where a logon and password is requested as part of the Browser - ie a pop-up from Netscape or IExplorer, NOT an HTML form - you will need to add a 'Authorization' line. This will be true if you get a message like : HTTP/1.0 999 Authorization failure ... etc etc etc ... Assuming you know your logon and password : 1. Use uuencode or sffb64 to generate a Base64 string echo -n "logon:passwd" | sffb64 -i 2. Add an extra line to the parameter file with the result of the sffb64 line using 'httphdr'. Syntax: Authorization (colon) (spc) Basic (spc) (Base64 of logon:password) (\n FipSeq for NL) Eg httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n (copyright) 2016 and previous years FingerPost Ltd.