webwire

webwire

FOR HTTPS/port 443, please use the 'webwiressl' version of this program.

Webwire goes and gets pages of data from Other people's web sites automatically
and then sends those pages to your destination - usually the editorial system -
in the normal Fip fashion.

These can be updates of weather, financial data, sports results, backup for
Wire services if the satellite is down, graphics, software. In fact most
things.

It can be used either :
    - on a timed basis to get regular known pages.
    - on demand by sending a file into spool/webpoll with the FipHdr field DF set
to the parameter file required.

What it can do -
    - drill down links to several layers deep,
        optionally ignoring the data on the top levels.
    - select only certain links - either in XML, HTML, JSON or CSV
        - you set masks to filter which to get and which to ignore.
    - logon automatically to protected sites
        and save Cookie information for use in later accesses.
    - fill in standard form data to get make on-demand searches.
    - strip or rework HTML tags to make the data more presentable.
        This is meant for reasonably simple pages while more complicated ones
        will be routed through 'ipsgml' and/or 'ipxchg'.
    - Use an external list of values to make several grabs to the same
site/page/script
        but varying the search data for each hit. eg to pull all the values of a
financial index. (This we call a 'values-file')
    - Grab a 'id' from a web service and then sequentially call all pages using
intermediate ids from the last to the new one.

What it cannot do -
    - play tunes.
    - run javascripts or any other applet type affairs. (yet..)
    - run FTP, GOPHER or whatever (for these and especially FTP, see program
'ipftp' and 'iptimer').

The current version is primarily for getting text data but can be used for
images etc if required.

There is a TUNING mode to be used for setting up a new link and trying to clean
up the relevant parameter file WITHOUT sending (possibly) live data to the
required destination.
    - This shows the data with escaped unprintables and '$' at the end of a line.
    - All links and forms are also displayed.
    - Any pages saved in Tuning mode are NOT sent to the normal output queue
(spool/2go) but are left in spool/webtest for future perusal and/or deletion.
    - To run, choose your parameter file in tables/webwire and run 'webwire'
manually in a window:
        webwire -T AUS.STOX | more      for prompt before calls
    or  webwire -A -T AUS.STOX | tee aussies    for no prompting

There are Two (sometimes three) types of parameter file :
    1. Main Parameter file which sets up the polling of certain pages at set times
(if any).
    2. A Page Description file for each site/page accessed.
    3. Optional lookup file of values where you want to repetitively hit a site
changing certain values each time. (eg a sport site for several divisions or a
list of stox to get)

----- Main Parameter file -----

The syntax of the Main Parameter File - by default tables/webwire/XWEB :
    ; comment line
    poll:(pram file)    day:(MTuWThFSaSu)   time:20:30  mustget:

In detail, the 'poll' keyword :
    Pram file is the name of the Page Description file - see below for its syntax
    day:    Day of week to run the job :
            M   Monday
            Tu  Tuesday
            W   Wednesday
            Th  Thursday
            F   Friday
            Sa  Saturday
            Su  Sunday
            X   Every day.
            Z   Weekdays M-F.
            Case is NOT important.
            Commas (but NOT spaces) may be used to separate.
            Default is every day.
    either
    time:   Time of the day on 24 hour clock.   Default is 18:00.
    or
    every:  interval between grabs          Default: none
        every: (mins)   [(optional) start:(starttime) end:(endtime)
        every:30    start:07:30 end:19:00
        The minimum interval is 1 min and maximum is 3 hours (ie every:180 mins)
        You may also specify in seconds using 'secs' or 'seconds'
        immediately after the number (with no spaces)
            every:10secs    start:09:30 end:09:50
eg:
    poll:AP     day:ALL     time:20:10
        Get the Page file tables/webwire/AP every day at 20:10
    poll:Forex  day:MTuWThF time:16:30
    poll:Forex  day:MTuWThF time:16:40
        Get the Page file tables/webwire/FOREX every week day at 16:30 and 16:40

There can be none or up to 200 polls in the main parameter file.
Note that the page is grabbed ONLY if the program is running.

----- Page Description Parameter files -----

The individual Page description parameter files are also in tables/webwire. The
syntax of these are :
    ; comment start with a semi colon like this

MANDATORY
    url:    Full url of the page.               default: none
        There MUST be one and only one 'url:' specified.
        You can also specify the page, cgi and any subparameters.
        eg  url:www.fingerpost.co.uk
            url:www.big-press-org/sports/baseball/index.htm
            url:www.marketlook.co.uk/scripts/Summary.dll?HandleSummary

    dest:   Fip Destination for the files           default: WEBDATA
        This is the 'DU' FipHdr field as per the USERS file.
        eg  dest:w3saves

OPTIONAL:
    use-tls: no/yes
    use-ssl: no/yes
    use-https: no/yes
        Use Secure Sockets Layer (TLS/SSL) - also called HTTPS  default: no
        If the url starts 'https://....' then this command is NOT needed.
    ssl-method: (1,2,3,11,12,23,999)
        Version number to use for TLS/SSL       default: 999 for current default (2 or 3)
        Pls do NOT use sslv2 !
            1=tls1, 11=tls1_1, 12=tls1_2
            2=sslv2, 3=sslv3, 23=sslv2 and 3
            999 is default (currently 23 which on a modern server is sslv3 and tls1_2 !)
    ssl-password: (password)
    ssl-passwd: (password)                default: none
        Optional password if the handshake requires a shared secret
    ssl-pem: (name of a PEM certificate file)       default: none
    ssl-cert: (name of a PEM certificate chain)     default: none
    ssl-root-cert: (name of a root PEM certificate file)    defaunt: none
        Optional certificates are in tables/ssl unless name starts with '/'
    ssl-verify: yes/no  verify certificates     default: yes
    ssl-ciphers: (list) acceptable ciphers
        (use 'openssl ciphers' to list)
        default:  "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS"

    port:   Port number of the Remore Server.       default: 80
        This forces the port to be this if none is specified.
    nofiphdr: Do NOT add a Fip Hdr to the file.     default: yes pls
    source: Fip Source of the files. (FipHdr 'SU').     default: XWEB
        Unless 'noarchive' is specified, all data files will be archived under this
name in log/data.
        This can be in FipSeq so that 'combie' can be used to set a default..
    noarchive: Do NOT archive these files in log/data.  default: archive
    maxlevel:3  Maximum no of levels to drill down. default: 1
        Normally the URL you have requested is the data you want.
        However if that is an index page with links that may change,
        it may be these lower-level pages that are needed. 'maxlevel'
        states how many levels of link the actual data pages are.
        Default is 1 = do NOT drill down any of the links.
        Note that level 1 is the first page.
    ignorelevel: Used with 'maxlevel' where the information     def: no
        required is on a linked page and NOT on the first page,
        use 'ignorelevel' to ignore all those pages on intermediate
        levels.  Note that level 1 is the first page.
        eg  ; ignore levels 1, 2, 4 and 6
            ignorelevel:1,2,4,6
    matchlinks: Only follow links which match this mask.    def: all links
        Used only if 'maxlevel' is greater than 1.
        There can be many 'matchlinks'.
        Use the '*' as a wild card string and '?' as a wild chr.
        eg  ; get all links ENDING 'html'
            matchlinks:*html
    matchforms: Only process forms which match this mask.   default:no forms
        Used only if 'maxlevel' is greater than 1.
        There can be many 'matchforms'.
        Use the '*' as a wild card string and '?' as a wild chr.
            eg  ; get all forms ENDING 'asp'
                matchforms:getfile.asp
    matchframes: Only follow frames which match this mask.  def: all frames
        Used only if 'maxlevel' is greater than 1.
        There can be many 'matchframes'.
        Use the '*' as a wild card string and '?' as a wild chr.
        eg  ; get all links ENDING 'html'
            matchframes:*.top
    matchkeys: Only follow links which match this test. def: all links
        Used only if 'maxlevel' is greater than 1.
        Used only or 'linktag' where an attribute MUST be set for the link to be
valid
        There can be many 'matchkeys.
        Use the '*' as a wild card string and '?' as a wild chr.
        eg  ; <hotel id=33 name="Fawlty Towers" url="http://www.ohnonotagain.com"
status="current" />
            linktag:hotel@url
            matchkeys:hotel@status=current
            matchkeys:hotel@status=ready
    match-case-sensitive: yes/no
        all matches and ignores can be case sensiive or in-sensitive
        DEFAULT changed 05u to INsensitive - previously sensitive.
    force-lower-levels: (levelNumber)
        When data is on more than one level - maybe a text page has a link to a PDF
and you need both bits, use this to get all bits of this element before
continuing with the next element.
        The default (without this parameter) is to get all this level and then all
the next lowest level etc.
        ; force the lower levels below level 2
        force-lower-levels:2

    skip-links: Name of a file in /fip/fix/webwire holding names of links
        and forms already accessed; so that only new ones are tried.
            eg  skip-links:webwirelinks.\$d
            default: none
    skip-details-tag: (tagname) extra details (such as a publishdate) for check if
existing links have been updated
            see below on the section for RSS feeds
            default: none
    skip-purge-after: (hours) Number of hours to keep the skip entry
        default is 24.  You might want to tune this :
            make bigger if sites add/take off old material
            reduce the time if the same link is used for different data
    skip-save-data: (FipSeq field)
        Sometimes there is some data in the link which changes for every access -
such as a Cookie or SessionId
        eg the first access might get
            search.do;jsessionid=A9823A4622A23C10C4EC7F1825BF9E26.node1?messageId=268482
        and the second
            search.do;jsessionid=FCC18E9582E77C2AD9EFE6C68CA0F0A2.node1?messageId=268482
        But they both happen to be the same file - messageId=268482
        Use FipSeq to just get the data that contains ONLY the information you want
to save.
        Certain FipHdr fields hold relevant info:
            WX is the field marker '^'
            WS is the skip details tag (optional - see above)
            WT is the type - 'a'-anchor
            WL is the level no
            W$ is the actual link - anchor, form etc
            WH is the associated display text from an anchor tag
        In the above example :
            ; split on the '?' - get the second field
            repeat:Q1   W$,?.2
            ; skip string is now 'messageId=268482' - note the FipSeq needs a backslash
            skip-save-data:\Q1

    skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute
        the skip file when changed (see doc on 'ipbalan')
        This is often used where a second system could be used as a redundant server
        if the main system fails. (see also -B input switch)
    ignorelinks:    Of the Links found, skip any matching this mask. default: all
links
        Used only if 'maxlevel' is greater than 1.
        There can be many 'ignorelinks'.
        Use the '*' as a wild card string and '?' as a wild chr.
        eg  ; ignore any links pointing at any 'netscape' or 'microsoft' site
            ignorelinks:*microsoft*
            ignorelinks:*netscape*
            ; ignore any links requiring 'ftp:'
            ignorelinks:ftp:// *
            ; ignore any links to other sections
            ignorelinks:../ *
            ; ignore any links to any index
            ignorelinks:*index*
    httphdr: Extra lines of HTTP header you may need.   default: none
        Remember to add a NL at the end of each line.
        There can be multiple httphdr lines but pls remember to add a '\n' at the
        end of each one. (or you can try to force all on one httphdr line!)
        eg  httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n
            httphdr:User-Agent: Mozilla/4.0\n
            httphdr:Host: wibble.wobble.com\n
        see below for 'useful, common header lines'
        ** ALL basic-authentication MUST BE HIGHER IN THE PARAMETER FILE THAN httphdr
OR proxy-logon
    httphdr-on-all-grabs:yes/no
        Normally the httphdr is only for a single host. So if the 2nd or subsequent
level is to a different host, by default, nothing defined as 'httphdr' will be
added.
        if 'yes', the option adds the httphdrs to all grabs
    httphdr-on-proxy:yes/no
        Normally the httphdr is only for data grabs NOT for getting thru the Proxy.
        if 'yes', the option adds the httphdrs to the proxy call

    basic-authentication: (fiphdr field) (logon:password)
        Build a FipHdr field with the BasicAuthentication formatted logon:password
        Pls remember to escape any funny chrs - like backslashes
        ** ALL basic-authentication MUST BE HIGHER IN THE PARAMETER FILE THAN httphdr
OR proxy-logon
        eg  basic-authentication:BA DOMMY\\zipple:Ardvark99
            httphdr:Authorization: Basic \BA\n

    method: POST/GET/DELETE/PUT etc             default: GET unless 'post:' is specified
        normally this is a single UPPERCASE action - with NO spaces.
    post:    Post a Form                    default: get url
        see below for processing a form using method=POST.
    filename: Filename for the output file in FipSeq.   default: WEB\$Z
        If this does NOT start with a '/' it is left under the
        Output Queue as specified on startup (default spool/2go)
        eg  filename:AFP\$d.\$z
    striptags:(yes|no) Strip tags and attributes        default: no
    wild: (FipSeq)  Character used as a Wild String for default: '*'
        'matchlinks/ignorelinks'.
        eg  wild:\377
    singlewild: (FipSeq) Character used as a single     default: '?'
        Wild chr for 'matchlinks/ignorelinks'.
        eg  singlewild:!
    number: (o|d|h) Number system for FipSeq        default: octal
        octal, decimal or hexidecimal
        The following are all equivalent :
            number:octal
            before:\040
            number:decimal
            before:\032
            number:hex
            before:\020
    before: FipSeq String to add before any data.       default: none
    after:  FipSeq String to add after any data.        default: none
    script: Script to run on ths data of the incoming file. default: none
    outque: Output folder (in FipSeq)           default: spool/2go
        This overrides both the default and the '-o' output switch
        except for Testing/Tuning mode where the file is forced
        to spool/webtest.
    log:    FipSeq custom logging for the item log.     default:\SN \SU \EF : \EH,\EP
        This logs each Page grabbed
        Note that
            EH or ST    remote site host
            EP or SP    remote site port
            EN or SF or SG  remote site url     SG is the actual link, the others are the
link used to grab
            EF      parameter file used
        The default is that no incoming files are logged by webwire
    custom-log: FipSeq custom logging for the item log.     default: none
        This can be used to log link details in a custom log
/fip/log/webwire/(date)_(paramfile).fip

custom-log:pnac.\YN|date.\YT|procdate.\T7|taketime.\T9|source.\TU|take.\TZ|head.\TH
    log-https-errors:warn
        Any failure to go secure in https connections are flagged as warnings
        The transmission is always aborted. This parameter affects only the logging.
        default:  !x for failures

    extra:
    extra-pre: Extra FipHdr fields to be added to the output file.  default: none
        To separate FipHdr fields, pls use a '#'.
        extra-pre is added as soon as the file is read - so may be used for
information in the URL
        extra is only used for any output file and is not used at all for any other
purpose.
        eg  extra:ZH:NYNZ#DI:Headline News#QZ:333
    tag:    FipSeq String to replace the start tag      default: none
        such as <H1>. There can be many 'tag's.
        eg  tag:P       {Para}\n
    endtag: FipSeq String to replace the End tag        default: none
        such as </P>, </TITLE>. There can be many 'endtag's.
        eg  endtag:TITLE    \n
    getimages: Also get all the images
        By default all images - *.gif or *.jpeg are ignored.
    keep-alive: yes/no                  default: no
        Just that ! default:no
    http-version: 1.0 or 1.1                default:1.0
    only-get-if-modified:   (FipSeq message if not found)   default: get
        This will check the remote server for the time the page was
        last modified. This does not work with old servers and some
        set to HTTP/1.0.
        If modified since, the page is read
        If not, the optional message is sent
        If there is no message, no data is sent - just a note in the item log
    ignore-key:PHPSESSID
        When matching for skip files, ignore this key-value pair.
        see the section below on Repeat Offenders
    max-items: (number)                 default: 0 for all
        Max number of items to grab per session
        Some sites only allow you to read 5 or 10 items before blocking you.
        Use this to creep under that total.
    pause-between-files: (secs)
        Gap/wait/pause between grabs    default is 5 for standalone, 1 for iptimer
        This is overridden by the -w input switch
    one-output-file: Put ALL data in a single output file.
        The default is one file per page/access
        Use this with 'values' to create a single output file.
        This ONLY uses the FipHdr of the first file if 'values' have been specified.
    end-of-document: Where a site is sending really really crap HTML - or XML
        use this to state what the last tag.
        For no checking at all : end-of-document:
        Default:        end-of-document:</HTML>
        See below for a standard-fingerpost-rant on crap HTML.....
    end-of-cookie-page: end text which signifies the end of a logon or cookie page
        This is rarely changed.
        default is </HTML>
    connection-timeout: (secs)
    wait-end-timeout: (secs)
        For slow, busy sites, data - especially big files - may take a lot longer
than normal to be
        retreived. Use this to expand that time. Default is 120 (it should be
divisible by 5 for some arcane reason)
    pretend-301: (3 digit number)
    pretend-302: (3 digit number)
        Ignore redirects (HTTP return code 301 or 302) and assume they are this
return code
        pretend-301:200
        this will take a 301 and save the data as through it was an incoming file.
    no-data: (FipSeq string in place of data)
        Do not get/send the data - just this string
    data-is-binary:(yes/no/maybe)
        Data files at the lowest level are binary or not
        default is check for <?xml, Tiff, Jpeg, MsWord/Office, EPS and PDF
automaticatlly
            otherwise it is treated as text
    ignore-mime-if-binary: (yes/no)
        if yes = Strip the MimeHeader off binary files
        default is no to leave it on - so you know what the file really is !
    proxy-server: If using a proxy, these are the name and port to aim at.
    proxy-port:
    proxy-logon: This is the logon and password to get thru the firewall if
required. The format is (logon) (colon) (password) and is converted to base 64.
        proxy-logon:Y2hyaXMuaHVnaGpvbmVzOnBhbnRoZXIK=

        ** ALL basic-authentication MUST BE HIGHER IN THE PARAMETER FILE THAN httphdr
OR proxy-logon
        To generate use basic-authentication or:
            echo -n "logon:password" | sffb64 -i
        eg  echo -n "chris:sleekpanther" | sffb64 -i
        gives   Y2hyaXM6c2xlZWtwYW50aGVy
            proxy-logon:Y2hyaXM6c2xlZWtwYW50aGVy=
    proxy-is-squid:yes/no   Is the proxy a Squid ?  default: no
    proxy-handshake:yes/no  Does the proxy need to say hello first ?    default: no
        If the proxy is a Squid, this MUST be NO

    logeachfile:(dest) Send a Success/failed msg to this destination
            for each file. There is no default. This log file is
            just a FipHdr with the following extra fields :
                DR-File Sent OK     DR:ok or DR:error
                DG-Will Retry later DG:retrying, DG:stopped
                DT-Some message text    DT:No connection
            default: no log created.
        The text for the DR and DG can be in FipSeq and so can contain
        FipHdr and other variables. As they are FipHdr fields, please
        do NOT put NL, CR etc in the fields.
        Note that System Variable \$q holds the time taken for transmission.
    DRgood:(text)   Message for the FipHdr field DR on a   successful tx
            default: ok
    DRbad: (text)   Message for the FipHdr field DR on a unsuccessful tx
            default: error
    DGcont:(text)   Message for the FipHdr field DG if, after an
            unsuccessful tz, another attempt will be made.
            default: retrying
    DGstop:(text)   Message for the FipHdr field DG if no further
            attempts will be made as the file was sent successfully
            or the maximum no of attempts has been tried.
            default: stopped
    fiphdr-for-logeachfile: (FipSeq) or
    msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
            'logeachfile' or 'loglasterrfile' msg. This should be in FipHdr
            format and be in FipSeq. It can be used to pass FipHdr fields
            in the outgoing file into the log file.
            eg  msgeachfile:    DF:logdial\nSS:\SS\n
            default: nothing added

    convert-CDATA-sections:
        convert-CDATA-sections:no   - no dont ! (default)
        convert-CDATA-sections:zap  - no but zap the '<!CDATA[' and ']]>'
        convert-CDATA-sections:yes  - yes pls and zap the '<!CDATA[' and ']]>'
        convert-CDATA-sections:preserve - yes pls and leave the '<!CDATA[' and ']]>'
        Normally a CDATA section like :
            <![CDATA[ Vongerful Vondafool C&oe;penh&areing;gen <99thisIsAnon-compliant
XMLtag> ]]>
        is considered a single, raw string of XML/SGML data. And all the tags and
entities (like &lt;) are not changed either. Use this parameter to convert
them.
        Note that you should use this option CAREFULLY if any tag in the CDATA is the
same as a tag in the main envelope. See below for more comments.

To save the contents of a particular Tag or TagAttribute, use the 'fiphdr'
keyword :
    fiphdr:(FipHdr field)  (optional subkeywords)
        Either  tag:(name of tag)
                specify the tag name which contains the data required.
        Or  data:(FipSeq)
                for adding FipHdrs with standing data.
                fiphdr:TT   data:\$e\$y\$i\$d
                will create a FipHdr field DT with the current date in it
        Or  tag:(name of tag)@(name of attribute)
                specify the tag name and the attribute name which contains the data
required.
        Or there can also be a 'key' parameter for selecting the data ONLY if there
is Key attribute with its data equal to a certain string:
            eg: if the tag is <meta name="category" content="f"/>
                fiphdr:NC   tag:meta@content key:meta@name=category
                Get the contents of the content attribute of 'meta' where another attribute
called 'name' has the value 'category'
            or  fiphdr:NC   tag:meta    key:meta@name=category
            or  fiphdr:NC   tag:meta@name=category
                Get the data for the 'meta' tag that has an att 'name' = 'category'
            Double quotes around the Key Data are optional unless there are embedded
spaces. The Key Data can be in FipSeq.

        For any of the tag options, use 'dup' to flag duplicated fields.
            dup:(optional separator)
                This field may be duplicated. Duplicate fields are separated
                with a space unless a separator chr is also specified.

        Where there might be embedded tags inside the main tag, use 'repxml' to
specify a replace string
            repxml:(FipSeq)
            eg fiphdr:AL    tag:TD  repxml:+\s+
                and the data is <td>abc<br>efg<br>line3</td>
                will give   AL:abc+ +efg+ +line3

        As some FipHdr fields have distinct meanings - SN, DU, DP etc - please use
other 2 letter codes starting N or Q.
        In the current version of webwire, you CANNOT specify trees of tags ie
fiphdr:AA tag:entry/id.

    eg  fiphdr:NA   tag:itemid  dup:+
            get the data from each <ITEMID> field. If there is more than one,
            they are separated by a '+'.

    fiphdr-save:(FipSeq)
    fiphdr-file:(Filename in /fip/fix/webwire/fiphdr)
        This allows data to be stored as FipHdrs at the end of the session - and read
at the begining of the next
        So items like Sequence numbers and time-of-access can be passed between
attempts.
            ; default name
            combie:QA   WA,default
            ; save and possibly reuse the FipHdrs ....
            repeat:JQ   J1,+,1
            repeat:JD   J2,+,1
            fiphdr-save:BQ:\JQ\nBD:\JD\nXX:some comment\n
            fiphdr-file:websave_\QA
        ** This must be lower down the parameter file than any FipSeq if you are
using FipHdr fields as the example above !
        There can be multiple 'fiphdr-file' - all of which are read as the parameter
file is read.
        But if there is a fiphdr-save, ONLY the last 'fiphdr-file' is stored to.

    fiphdr-on-all-levels:
        Add the FipHdr to each file on every level - default: no
    fiphdr-hash: (single chr in FipSeq)
        This will replace a Hash '#' in a FipHdr field (as Hashes are normally
end-of-fiphdr field)

    meta-to-save:(FipSeq)
    meta-save-file: (Filename)
    meta-save-on-tag: (tag name)
        This meta file is appended to on the End-of-tag specified (or end-of-file if
no tag specified)
            ; save these fields to the lookup file
            meta-to-save:\J3|\J5|\J6|\J1|\J4|\$h:\$n:\$b\n
            meta-save-file:/fip/data/blob/\$e\$y\$i\$d/\WA
            meta-save-on-tag:LINK
    reset-fiphdr-on-tag: (tagName)
        Trim the FipHdr - and extra, added fields - on the end of this tag to the
same position when the tag started
        This can be used in meta-save to make sure that FipHdr fields from one group
of tags to not linger and are not used for the second or subsequent
        default: not used.
    grab-on-tag: (tagName)
    grab-on-endtag: (tagName)
        Any links should be grabbed at the start or end of this Tag
        default: all links are grabbed at the end of the page
        An extra parameter may be specified on the same line for level eg
        grab-on-endtag:VALUE    level:3
        grab-on-endtag:params/param/value/struct/member
    retry-404-max:3
    retry-404-gap:1
    retry-404-error:abort/ignore/move
    retry-404-queue:2go
    retry-404-fiphdr:#CE:300#DU:nextstage
        Retry links which return a 404 Not Found error. Max is the number of retries
and Gap is the pause in seconds between the retries
        Use this for those sites which are a bit slow to add the data files the links
point to.
        If the files really are not there - and you do NOT want to abort the
transmission - use 'retry-404-error:ignore' to continue with the next grab
        OR you can use retry-404-error:move and retry-404-queue:(queue in spool) and
retry-404-fiphdr:(FipSeq) to send a item
    retry-500-code:505
    retry-500-max:5
    retry-500-gap:1
    retry-500-error:abort/ignore/move
    retry-500-queue:2go
    retry-500-fiphdr:#CE:300#DU:nextstage
        Retry links which return this system error - code can be any 3 digit number
above 400.
        Max is the number of retries and Gap is the pause in seconds between the
retries
        Use this for those sites which are a bit slow to add the data files the links
point to.
        If the errors continue - and you do NOT want to abort the transmission - use
'retry-500-error:ignore' to continue with the next grab
        OR you can use retry-500-error:move and retry-500-queue:(queue in spool) and
retry-500-fiphdr:(FipSeq) to send a item
    save-data-path: (Fipseq pathname for data)
        This puts the data of the incoming file into this folder and creates a FipHdr
        file that contains 2 FipHdrs containing the full path/filename
            SX: and FTP_EXTERNAL_FILE:
        (ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE)
            eq  save-data-path:/fip/data/jpegs/\$e\$y\$i\$d/
        Use this for big files that you do not want to copy around the Fip Spool
area.
    save-data-filename: (FipSeq name)
        Use this to specify exactly what the the 'save-data-path' name should be
        default is (incoming filename).(time).(seqno)
        eg  save-data-filename:\HR-\SU.raw
    save-data-balance-group: (Balance group)
        Balance all save-data files to the following group. default: do not balance
    save-data-balance-folder: (Balance folder)
        If balancing, put the token in this folder under spool. default: 2balance

More Complex sites ------

-- The links are not in the normal Anchor or Frame tags.
If the Site returns an XML feed rather than HTML, you can specify what the
contents of which tags you want to play with. There can be up to 10 tags
specified.
    linktag:(tagname)
or  linktag:(tagname)@(attribute)   (for version 05u onwards)
    linktag:TEXT
    linktag-2:Slavver
    linktag-3:Bone

or to imitate the defaults :
    linktag-1:a@href
    linktag-2:frames@src

- sites which return other data which is not xml - such as CSVs
    data-type:CSV   (can be CSV for comma sep format, JSON, PSV for Pipe sep, TXT)
    data-type-sep:|
    data-type-eoln:
    data-link-idx:2
        define the column containing the link to the data
    headline-link-idx:3
        define the column containing the headline
    skipdetails-link-idx:1
        define the column containing the skipdetails

- RSS feeds
Sometimes a link can point to data which gets updated and there is a second tag
which gives either a unique-id or a date/time which you need to track for any
changes. Use the 'skip-details-tag' to specify the second tag - it is the
combination of the 'linktag' and 'skip-details-tag' which should be unique.
For general RSS 2.0 feeds, this can either be 'pubDate' or 'guid' :
    linktag:link
    skip-details-tag:pubDate
In RSS feeds there is often a fake 'link' at the top which is the channel.
Usually you do not want this one - often it is a URN not a real URL, so use
'matchlinks' or 'ignorelinks' to bypass it.
if more than one skip details are needed, up to 9 skip-details-tag-X can be
specified.

-- If the data in the link is not complete..
Use templates to slot data from a link into another call. This is again used
extensively for XML work - like soap.

It uses either just a template (in FipSeq so you can add Header Fields etc) or
a template AND a template file if there is a lot of data.
    level2template:/query.dll?src=\QD
    level3template:/getFile.dll?file=\W$
    level3template-file:soap-getfile.xml

There are 4 templates  for level2, 3, 4 and 5. 'maxlevels:' and 'ignorelevels'
must always be used with these to specify which one you need the data from.

A levelXtemplate on its own will generate a GET.
To POST something, you will also have to specify a 'levelXdata: (FipSeq)' eg
    ; level 3 - To get THAT file is always a POST of
FileManager1%24gvwFiles%24ctl03%24gvlnkNam
    ; .. using the different EVTVAL and VWSTAT
    level3template:/proximity/Admin/FileManager.aspx

level3data:__EVENTTARGET=\A2&__EVENTARGUMENT=&__VIEWSTATE=\G8&__EVENTVALIDATION=\G9
will force as
    POST /proximity/Admin/FileManager.aspx
will data filled in for fiphdrs A2, G8 and G9 eg

_EVENTTARGET=FileManager1%24gvwFiles%24ctl03%24gvlnkName&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUJMjQyMzY1MzEX%3D&__EVENTVALIDATION=%2FwEWAwKVxPyBCgLc

Note there is no level1template as that is the same as the URL:.. BUT there is
a 'level1template-file' version. In this case the URL: should be just that.

The template-files are normally in /fip/tables/webwire. They are NOT force
uppercase.

The default Content-Type for POSTing data or forms,
'application/x-www-form-urlencoded', sometimes needs to be changed for
templates.
It can be changed with the 'levelXmime' parameter. For example, soap normally
likes a content-type of 'application/soap+xml':
    level1mime: application/soap+xml
unless you are Microsoft of course who usually/sometimes want
    level1mime: text/soap

The 'W$' in the example is because each link is put into a temporary FipHdr
field called W$ as it is being used. If the link data is too much or too
little, use FipSeq to chop/add/replace.
Eg  if the data in the link is "nine:/rt/newsart/Id="z1jit4":text"
    And you want a link like
        /searchDB?database=nine&link=/rt/newsart/Id="z1jit4"&format=text
    use repeat:R1   W$,:,1
        repeat:R2   W$,:,2
        repeat:R3   W$,:,3
        ; if there is no 3rd field, use 'xml' instead
        combie:W4   R3,xml
        level2template:/searchDB?database=\R1&link=\R3&format=\W4

Values -------
Values can be -
    - EITHER a file containing lines of values to be used to repeatedly grab data
for a single file.
        using values-file:(filename in tables/webwire)
    - OR a sequential number
        using values-seqno:(min value):(max value):(incremental value)
        plus    values-seqno-fiphdr-from: (FipHdr field containing the From seqno - ie
start grabbing from the NEXT id after this)
            values-seqno-fiphdr-to: (FipHdr field containing the To seqno - ie each
seqno until and INCLUDING this one)
    values-get-url:
    values-post-url:
    values-post-data:
        Fipseq to POST a form or GET a link from a line in the
        values file. See below for a description.
    values-sep: Separator chr for splitting fields in the values file.
        default is a pipe - '|'
    values-leave-spaces: Normally leading spaces are trimmed from each
        field in the values file. Use this to preserve them.
    values-parallel: (Number of Simultaneous Hits)
        For 'values' the default is to run the hits serially, one after the other has
finished. Use this to send out a number of hits
        at the same time which should reduce the total time by a large factor.
However, you should check with the remote and test
        what the number should be. For Apache sites for example, 8 is a common
default setting.
        eg  values-parallel: 10
    values-fiphdr: Normally fipHdr W1 will contain the first field of the values
file, W2 the second etc.
        So data can be specified by \W1
        Use this parameter to specify another field - ie if W1 is being used
elsewhere.
        ** Note that if you are using iptimer to start webwire running a values file,
the Wx fields will be zapped in the output file.
        So in this case, always use 'value-fiphdr:' with a different FipHdr if you
want to use the Values in iproute or another downstream program.
        eg  values-fiphdr:R1
    values-pause: (secs)
        Gap/wait/pause between Grabs using the next value   default is 0 for none
    zap-values-file: (yes/no)
        Delete the values file after it has been used.  default no
        Only files in /fip/x/ starting TMP.. can be deleted.

Note that in the FipHdr - unless the 'nofiphdr' keyword has been requested, the
following fields will be filled in :
    Day and time in the normal HH,HD,HY etc fields
    ST  host
    SP  port
    SF  url - path/filename being grabbed
    SG  url - path/filename with is the link
Where webwire is sitting on a scrolled queue (using -i), the folder name is in
EQ and the filename EN (with all '#' replaced by the chr chged by
'fiphdr-hash')

Input Parameters (all optional) :
either  -i : scrolled queue             default: no default
        This checks the folder and for each file, checks the FipHdr for 'DF' which is
used for the name of the parameter file to run against
        This allow a variety of parameter files to be run
or  -1 : Run a single time and exit         default: spool
        The parameter is the name of the individual parameter file in tables/webwire
(ie NOT The top or main parameter file)
or  -T : Tuning mode                default: spool
        Display links and data for the page requested. Runs only that page and then
exits.
        The parameter is the name of the individual parameter file in tables/webwire
(ie NOT The top or main parameter file)
    -A : In Tuning mode, do NOT prompt before searching a link  default: prompt
    -a : log the actual link of each accesses in the FipLog     default: no
        This can be quite a lot of logging if you are grabbing lots of files !
        But is quite useful when starting/adding a new feed.
    -B : default balance group for skip files           default: none
        (see skip-balance-group parameter)
    -C : warm restart for cookies/api-keys      default: always ask for new
cookies/api-keys for logon
        ie do NOT re-logon if the previous session logged on and saved the cookie or
api-key
        if any apikey is missing or has timed out, all cookies and api-keys are wiped
and webwire needs to be re-run to logon and download.
        see note below
    -d : done folder for -i scrolled queue      default: none
        This can be overwritten by the 'doneque:' parameter
    -D : display the Request and Response       default: do not
    -e : exit with the Result Code of the last grab.    default: normal program exit
        The Normal exit is 0 if ok, negative number if not
        With -e this will be 0 for ok, and -1 (timeout) but 4XX or 5XX for page
errors.
    -F : do NOT add a FipHdr to the output file default: do
        this can be overridden by the 'nofiphdr:no' parameter
    -h : extra FipHdr information           default: none
        This is in FipSeq and should normally be quoted
        Note this is the means that 'iptimer' sends variable information to webwire
        eg : -h"SN:hello#TC:200401031"
    -H : display the Request and Response in fancy HTML default: do not
    -I : wire id                        default: 0
        used to track which instance of a multi-webwire system a file arrived/logged
    -k : ignore the Skip list (used mainly in tuning)   default: use skip-links:
    -K : Do NOT save or process any data, just build up a skip file.
        This can be used before putting sites into production so that
            all old links are ignored and only new links will be tracked.
        ie run 'webwire -1 (name) -K' once beforehand.
    -l : no logging to the FipLog except for errors default: log all
    -L : log new files and errors to the FipLog default: log all
    -N : path and filename of the output file   default: fip standard
        use this to leave the file(s) in a non-std folder.
    -o : output queue in 'spool'            default: spool/2go
        This can be overwritten by the 'outque' parameter
        This is ignoring in Tuning mode.
    -O : force ALL output to this queue in 'spool'  default: spool/2go
        This overwrites the 'outque' parameter
        This is ignoring in Tuning mode.
    -s : generate statistics for bandwidth usage    default: no
        using Hour_group files
    -S : generate statistics for bandwidth usage    default: no
        using name of group_client files
    -t : track status               default: no
        this can be overwridden by the parameter
            track-status:no
    -V : if using spool-a-folder (-i) then stop when it is empty    default: keep
spooling
    -w : Wait in seconds between accessing links.   default: 5
    -x : Proxy server host or IP address        default: none
    -X : Proxy server port              default: 80
    -y : Proxy logon                default: none
    -Y : Proxy server is Squid          default: no
    -z : parameter file in 'tables/webwire'.    default: XWEB
    -v : Print the version number and exit

---- Other Notes ----

-- Netiquette --

Pls note if you are grabbing data off another site, then you should contact the
webmaster of the remote and let them know. Certainly if you are accessing every
few seconds, then there is a good chance they will put you on some refuse list.
So it pays to be nice !

-- How to find out the actual url....

Sometimes it is quite difficult to find out the real path to use for the url.

Especially so for script-driven gets and puts.

NetScape or Iexploiter is invaluable in this case..
 - use either 'View Source' or 'History' normally gives the game away!

Snooping using tcpdump or windump
    0. Open a Terminal/Cmd window and start you browser - without hitting the site
yet
    1. Find out which interface
        tcpdump -D
    2. Leave tcpdump running in background
        On Mac OSX you will need to be sudo
        tcpdump -i1 -w remo.tdmp -X host www.remote.host

    3. On the browser, do the absolute minimum ..
        .. do a simple logon and grab ne file using Firefox, Mozilla, IExp, Safari
etc
    4. CntrlC to stop tcpdump
    5. run tcpdump to show data
        rcpdump -r remp.tdmp > remo.fip
    6. call up remo.fip in an editor.

-- Cookie Cookie Cookie Cookie Cookie Cookie Cookie Cookie Cookie Cookie

Cookies are neat but nasty.
If you already know the cookie you need, just make a file in /fip/fix/webwire
with the name of the cookie (case is important on Unix boxes) and slap in the
whole of that cookie which has the syntax
    (key)=(data)
ie  zumzum=hungryTummy

Before grabbing data pages we can attempt to logon to a box and get its cookies
!!
This uses from 1 to 9 GETs or POSTs

    add-cookie:\C1; \C2 Add the Cookie on to the end of the HTTP headers in this
form
    get-cookie-1:   Command to send to get a cookie or to logon.
    get-cookie-data-1: Optional data usually required for a POST
    get-cookie-http-1: more HTTP headers used ONLY for this GET/POST
    cookie-fiphdr-1: name of the cookie to use as a FipHdr field C1 to C9
            ie if there are several cookies returned but only one
            is needed, put the key as the cookie-fiphdr
            ie Set-Cookie: ABC=12345
                add-cookie:\C1; perm=yes
                cookie-fiphdr-1:ABC
            will result in a Cookie: ABC=12345; perm=yes
            If you  want all the cookies to be saved, use '*'
                cookie-fiphdr-1:*
    cookie-ignore-redirect-1: just that
        Ignore any redirects (like 302 Moved).
    follow-cookie-redirect: just that !
        ie if you get a 302 Moved Temporarily status Plus a Location from a cookies
-request,
        use that rather than the 'url:..' specified.
            HTTP/1.1 302 Moved Temporarily$
            Date: Fri, 29 Oct 2010 00:17:19 GMT$
            Cache-Control: max-age=3$
            Location:
http://fippo.fip.fip/palio/html.run?_Instance=cms_csi&_PageID=1&_SessionID=1068051&_SessionKey=922432532&_CheckSum=328747502$
    cookie-form-1: find and save an input tag in a form and put the data in a
FipHdr field starting F*
        use this to add hidden form zones into a reply for a logon for example.
        ; csrf_token will go into F1
        cookie-form-1:csrf_token
        ; then send it back
        get-cookie-2:POST /login/

get-cookie-data-2:csrf_token=\F1&email=dot%40sniggerfrost.com&pwhash=somut&caform=1&submit=Login

There can be up to 9 of these.
    eg  add-cookie:\C1
        get-cookie-1:GET /
        get-cookie-2:POST /logon.pl
        get-cookie-data-2:logon=helpme&password=iamswimming

Rarely are any  get-cookie-http-1 fields needed as
    Host, Content-type, and Content-length are added automatically
    Referer is added if you have specified a 'referer'
        which you should if running 'http-version:1.1'
    Keep-alive is added if you secify 'keep-alive:yes'
    Others 'httphdr' fields should be specified as normal..

As a general rule, some Microsoft IIS sites (who else!) have problems if you
HTTP headers are in the wrong order. Basically, make sure your CONTENT* lines
are last.

Example 1
; ------------------------------------------------------
; we need to go and get a cookie for this service
; we will call it C1 - so the httphdr will be 'Cookie: (contents of C1)'
add-cookie:\C1
; C1 will hold the contents of an incoming 'WebLogicSession=.....'
cookie-fiphdr-1:WebLogicSession
; this is the URL to hit (with parameters) to trigger the Cookie
get-cookie-1:GET /servlet/com.login.DispatchServlet?Login=&User=guest&Pwd=guest

Example 2
; ----------------------------------------------
; in this case we have 3 cookies C1, C2 and a fixed one 'b'
; C1 is SID=..
; C2 is ASP...=...
; add the fixed 'b=b' on the end
add-cookie:\C1 ;\C2 ;b=b
; just one grab at a cookie - and Logon and the same time
get-cookie-1:POST /login/Login.asp
; one logon string
get-cookie-data-1:u=%2Findex.asp%3F&l=letmein&p=ohpleaseplease&x=0&y=0
; ignore the 302 return - it is only trying to send us to index.asp
cookie-noredirect-1:
; Save the two cookies as C1 and C2
cookie-fiphdr-1:SID
cookie-fiphdr-2:ASPSESSIONIDASDQCAAD

This will POST - ie pretend to be a filled out html FORM - the logon back.

Note that the cookie-data is 'URI escaped' ie if it is a special chr - like
/?&+ - and is in the data bit, you must use the '%xx' notation (where xx is
the HEX value). But hopefully you would have seen that in your tcpdump/snoop
anyway.

-- Proxies Proxies Proxies Proxies Proxies Proxies Proxies Proxies Proxies
Proxies

When running through a proxy server, you will need :
    1. hostname of the proxy server
    2. port number on the proxy server if it is NOT port 80
    3. (optionally) a logon and password
    4. Is the proxy SQUID ?
        If so headers are slightly different.

If this information is NOT available, normally you can find it easily from any
PC or Mac on the internal network using a browser like Netscape or IExplorer.

Start a NEW copy of either of these.  - It must be a new copy to check on
logons etc.

Under 'Preferences' or 'Internet Options' there should be a 'Connections'
section and under that, the host name or ip address plus host name of any proxy
used.

Note that often the main Fip server is NOT running DNS and will not be able to
resolve external hostnames, so the IP address must be used in this case.

Enter these values in the Fip parameter file as :
    proxy-server:195.13.83.99   (no default)
    proxy-port:412          (this defaults to port 80)

Use the Browser to attempt to access a web site outside the firewall - like
'www.fingerpost.co.uk'.

If you are asked for a password to get through, you will probably need to add a
'proxy-logon' parameter too unless the keeper of the Firewall has made a hole
through just for you.

The data for 'proxy-logon' is in base64 in the format (logon) (colon)
(password).

Use 'sffb64' to generate this string :
    On a Sparc  echo -n "chris:magicman" | sffb64 -i
    On Linux    echo "chris:magicman" | sffb64 -i
    On Winnt    type "chris:magicman" | sffb64 -i

    proxy-logon:Y2hyaXM6bWFnaWNtYW4===

The actual 'You need to Logon, Pal' message is a '407 Authentication Required'
message.

-- Repeat Offenders -----------------

Some sites add a session-id into each and every link. And this Id changes on
each access.

To 'webwire' this appears to be a new file and so it is grabbed every time -
falsely.

There is an 'ignore-key' command to isolate and ignore the relavany parameter.
eg Take a site like :
    url:http://www.fingerdong.com/
    matchlinks:*&news=yes&newsid=*
    ignorelevel:1

which returns links like
    /en/pressrelease.php?date=20080910&news=yes&PHPSESSID=11bf21&newsid=7866

If value of PHPSESSID changes each access, they you will get a copy of newsid
7866 every time.

Use :
    ignore-key:PHPSESSID

Do NOT specify the '=' or '?' etc.

-- Others Others Others Others Others Others Others Others Others Others Others

--Where 'webwire' is used to drill down links, there is a wait of about 5
seconds between accesses which, hopefully, is enough time for other people to
use that server.

--Where a logon and password is requested as part of the Browser - ie a pop-up
from Netscape or IExplorer, NOT an HTML form - you will need to add a
'Authorization' line. This will be true if you get a message like :
    HTTP/1.0 999 Authorization failure
        ... etc etc etc ...
    Assuming you know your logon and password :
    1. Use uuencode or sffb64 to generate a Base64 string
        echo -n "logon:passwd" | sffb64 -i
    2. Add an extra line to the parameter file with the result of the sffb64 line
using 'httphdr'.
        Syntax: Authorization (colon) (spc) Basic (spc) (Base64 of logon:password)
(\n FipSeq for NL)
        Eg  httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n

-- Valid links are :
    - The HREF tag atttibute in A for Anchor    <a href="www.fingerpost.co.uk>
    - The SRC  tag attribute in FRAME       <frame src="ax1000.html">
    - The URL in a META/Refresh         <META HTTP-EQUIV="Refresh" CONTENT="0;
url=go4thAndMulitply.com">

-- For 'matchlinks', the term LINK is the contents of the <a href="THISONE">,
NOT the associated text
    ie matchlinks:*boonies*
        will find   <a href="/rubbo/boonies/tunies.html">This is a Wonderful Page</a>
        BUT not     <a href="/tunies.html">This is the boonies Wonderful Page</a>

-- Note that 'ignorelinks' refers to both Links and Forms.

-- If you want to ignore all links and only get forms, use a weirdo name in
mathclinks
        matchlinks:gobbLedeGook9981

-- What are reasonable HTTP headers ?
1. If you are using HTTP Version 1.1, you MUST add a line in the headers which
specifies the actual host you are trying to access (ie the REMOTE hostname or
IP address):
    httphdr:Host: www.theirsite.com\n
or if DNS is a problem
    httphdr:Host: 123.456.789.012\n

2. Most servers would like to know what you are and what you can do - so lie !
    Try this for starters :
    httphdr:Accept: \052/\052\n
    httphdr:Accept-Language: en\n
    httphdr:User-Agent: Mozilla/4.0 (compatible; MSIE 4.01)\n
Note the syntax is httphdr:(Keyword) (colon) (space) (Parameter) (NL)
    Keyword is case-INsensitive
    There MUST a Colon-Space beteween the Keyword and Parameter.
    The line MUST finish with a single NL (which webwire will handle correctly)
    as Double NLs mean end of header.

3. If the data on a lower level is being served from a different host, if you
need authentication or some other httphdr, use the 'httphdr-on-all-grabs:yes'
parameter to add them for that server too.

-- ValuesFile ValuesFile ValuesFile ValuesFile ValuesFile ValuesFile --

Take the case where you need to get the 10 foreign exchange rates every 20
minutes from a site like Yahoo.

The normal way would be to test using one forex rate and, when ready, just
duplicate that parameter file another 9 times, just changing the forex
name/search string in the 'url' or 'post'.

The classy way is to pput all the search values (ie the bits that change) into
a single 'values-file' and reference them using FipHdr fields W1 to W9.

To Do this :
If the original url is :
  http://finance.yahoo.com/m5?a=1&s=USD&t=LAK

1. Create a values-file in /fip/tables/webwire - lets call ir VALUES_4_FOREX
    This can have the normal Fip-style comments of ';' at the start of line
    ;
    ; Values file for Forex
    ;
    USD|LAK
    USD|YEN
    USD|MYR
    ; end of values file

2. In the WebWire parameter file - lets call it FOREX.
    ;
    ;   FoREX
    ;
    port:8080
    url:http://finance.yahoo.com

    values-file:VALUES_4_FOREX

    values-get-url:/m5?a=1&s=\W1&t=\W2

... and let rip.....

Note that W1 is the first field, W2 the second etc. If you are already using W1
for something else, specify another FipHdr field to start on with the
'values-fiphdr' parameter.

Note that the FipHdr fields are useable for filename and other Fippy things.
  filename:Forex-\W1-\W2.fip

will give filenames (and/or FipHdr SN) for our example of
    Forex-USD-LAK.fip
    Forex-USD-YEN.fip
    Forex-USD-MYR.fip

-- Standard-FingerPost-Rant on bad HTML ----------------------
-- Using Webwire to pull off other file formats

Sometimes, 'webwire' seems to only grab part of a page and never returns
errors. Well, if you use a browser to look at the page and then 'View Source'
or 'View Frame Source', lo and behold there is probably a random </HTML> at
that point.

</HTML> is of course the End Tag of an HTML document. So we SHOULD stop there
really.

But a lot of web sites do not care how awful their stuff is - or maybe a
conversion program has been set up wrongly (a well-known news agency in New
York uses </html> in place of </image> to end pictures for example)

So use the keyword 'end-of-document' to track either nothing - just timeout -
or the REAL end of document.

If the data is NOT html - some XML variant for example - use 'end-of-document'
to track that.

By the way, did you know you can immunise yourself from fingerpost-rants; pls
contact the sales dept.

-- Wrinkles with Ports and RSS

Some RSS servers like to service the initial list from one port - but you have
to grab the data from another
    port:8080
    url:http://finance.yahoo.com

-- using warmrestarts and cookies to keep a logon current

1. Add '-C'

if using iptimer, add as ' switch:-C '
client:abc type:w template:abc.fip days:X  every:1s fiphdr:'#' switch:-C

2. run manually to see what the response utl is for a BAD ie logon again pls
eg good is normally a 200 code :    HTTP/1.1 200 OK
    bad is something like
HTTP/1.1 303 See Other$
Date: Fri, 24 Jul 2015 15:53:18 GMT$
Server: Apache$
Expires: Thu, 19 Nov 1981 08:52:00 GMT$
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0$
Pragma: no-cache$
Location: /login/$

2. to the parameter file
; YES or FIPHDR= logon returns data; COOKIE=logn returns a cookie in the
mimeheader
need-logon-token:yes
; look for this in the incoming data
matchlogon:/login/
add-cookie:\C1
cookie-fiphdr-1:*
get-cookie-1:POST /login/
get-cookie-data-1:csrf_token=\F1&email=shouta%40bingbong.com&pwhash=N1&caform=1&submit=Login

when it goes bad, you will get this message in the item log
Fri Jul 24 16:58:22 webwiressl !z : Zap cookies for Logon NZXIS.APAC

-- API keys in the data

basic-authentication:JA whomee:never
need-apikey:yes

apikey-host-1:https://auth.weather.gods
apikey-fiphdr-1:*
apikey-url-1:POST /oauth/token
apikey-httphdr-1:Authorization: Basic \JA\n
apikey-postdata-1:grant_type=client_credentials&scope=ukmo-warning-read
apikey-data-type-1:json
apikey-save-1:\VZBZ:\JZ\nXX:as at \$h:\$n:\$b\nXX:expires \J1\nXX:scope
\J2\nXX:domain \J3\n\$o

; READ last token
fiphdr-file:\R2.\R3.\R4.BZ
combie:RZ   JZ|BZ

; ; ..
{"timestamp":1482758750452,"status":401,"error":"Unauthorized","message":"Not
authorized","path":"/active"}
; if status code =401-Unauthorized .. redo the logon ..
matchlogon-fiphdr:J0=401
; .. and if the test is invalid, zap the FipHdr file
matchlogon-invalid:zap

-------------------------------------------------------
Version Control
;05z60  10may05 hourly bandwidth stats files rather than per client
    ;a 13may05 balance skiplists if changed
    ;b-c 25jun05 added -M and -K
    ;d-g 05aug05 added fiphdr:XX data:abc\A3 and wait-end-timeout
    ;h-k 04sep05 changed -x-X to force not default
    ;l-m 07nov05 added 24hour+ skip files
    ;n-p 25sep06 added ssl at last
    ;q-t 17oct06 added skip-details-tag
    ;u 29apr07 major change to linktag, added matchkeys and match-case-sensitive
    ;v-w14 21may07 add rest of path if 3rd+ level and no starting '/' (w14 -
tweaks to stuff_cookie)
    ;x1-6  8may08 added save-fiphdrs ;3 added -N newname ;6 bugette with VALUES
file and port != 80
        ;7 added -e and -E errname ;8 balance fiphdr fields ;9 meta-files ;10-12
minor
        ;13-14 note_balance_action ;15-16 spc in url ;17 added pretend-301:200 ;19
allow feed:
        ;20-23 finally added basic-authentication: and redid ssl
        ;24 bugette/modette - allow multiple spaces in mime headers
        ;25 allow intergap of zero
        ;26 bugette - save_metadata missing if one and only one found
        ;27-29 25jun10 bugette when proxy is a Squid and host changes
    ;y1-9 26jul10 added grab-on-tag/endtag (major release) ;10-11 6sep10 bugette
with 302-move and http://...
        ;12-14 added matchlogon, bug (bg) with data-type:CSV, plus tom bug :
retry-404-max:3 retry-404-gap:1
        ;15-17 14oct10 added skip-save-data and days:Z for weekdays
        ;18 15nov10 added use-cookie-redirect: ; 19 able to parse VALUES-FILE: ;20
added nofiphdr
        ;21-25 mess if too many 404 plus added -D and fiphdr-hash
        ;26-27 16mar11 added repxml for fiphdr: / include fiphdr-file in start of
hdr..
        ;28-29 31mar11 added zap-values-file:yes
        ;30-32 poll.every secs bugette ;32 added need-proxy-cookie
        ;33 6jul11 better skips handling now allow 15000 skips and zap olds with
different skipdetails
        ;34 29jul11 added need-logon-token and cookie-host-X for rconnect
        ;35-36 added dbl-dblqtes in links plus Bugette in Chunks and redid outque for
speedy
        ;37-41 added CONNECT for proxy https plus started minitracking and sleep
between polls for XWEB
        ;42 allow multiple spaces in custom tag link and added filter ;43
null_next_link added
        ;43-45 added retry-404-error
    ;z1-8 15mar12 added eventvalaidation and viewstate and level5* and json
        ;9-10 allow multiple grabs, added level to grab-on-tag and matchlinks etc
        ;11-12 redid 302 moved to handle full paths better ;13 ;14 bugettes -
proxy/do NOT output file for cookies
        ;15 28feb13 tuning for level1template-file:
        ;16  4apr13 bug in skips if no headline
        ;17-23 11apr13 added trees, levels and keys to fiphdr:,  grab-on*tab: and
linktag:
        ;24-28 17may13 added retry-500 kwds and better proxy handling ;27 added
level1mime and -I wireId
        ;29-31 17mar14 added 404/500action=move, que and FipHdr ;31 modette-repxml
for all tags
        ;32 14apr14 for custom logging ;33 4aug14 added -Z force DF ;34 bugette with
fiphdr.. key:
        ;35-36 12nov14 added httphdr-on-all-grabs
        ;37 17dec14 bugette WINNT Only, cookie=* ;38 ;39 fiphdr-hash for W$ too ;40
CDATA
        ;41 28dec15 new apache does not like 443 on the end of Host:..
        ;42 13jan16 bug with https and proxy
        ;43-45 22mar16 allow 302/301 with Values and Bug with skipDetailsTag ;46
httphdrs on proxy
        ;47 10jun16 pullapartJson better
        ;48-52 14jun16 proxy and TLS1_2 and httphdr on proxy
        ;53  7sep16 allow same tag or tag@att to be in multiple fiphdrs
        ;54-55 19sep16 added SX/save-data-pathname
        ;56-57 cleanups bugettes - SU/DU if in extra, proxy and 302 handling
        ;58-59 16oct16 for ANP_FOTO fiphdrs and keydepth
        ;60 30dec16 added apikey stuff
;004z   07jul04 tweaks...
    ;b 01aug04 added fiphdr:....
    ;c 10aug04 added levelXtemplate: where X is 2->4
    ;d-k 01sep04 -9 speedy and timing stats (f-maxlevel and values bugette)
    ;l-n 07oct04 added skps2, fixed one-file,
        fixed HTTP results with no messages
    ;o 28oct04 redid skps2
    ;p 01dec04 buglette with spaces in URLS - need to be stripped.
        plus lvl1file-lvl5file added
    ;s 31dec04 added -x proxy-host, -X proxy-port plus -y/-Y
    ;t-u 01feb05 added bandwidth-stats
    ;v-w 19feb05 added -u testPid and -U singlelevel only and split into files
        plus bugette with Chunking
    ;x-z 18apr05 added -O for rpt-offenders/small-diffs flag
;003z   15dec00 added one output file, tracking sents, only-get-if-modified
    ;a 20dec00 added watch on XWEB
    ;b/c 22jan01 allow hrefs to be NOT in dbl quotes
            plus added end-of-document
    ;d/e 19mar01 started proxies
    ;f 17sep01 proxies again
    ;g 29oct01 proxies again
    ;h 13dec01 minor mods - allow http:name:port in url and proxy
    ;i 08jan02 values-fiphdr and bugette with values
    ;j 08apr02 bug with one output file - core dump
    ;k 01jan03 MACOSX
    ;l-p 21jan04 added -h and allows secs for 'every'
        and 'no-data:'
    ;q-u 08jun04 added matchlinks/ignorelinks/url and now FipSeq
    ;u 27jun04 added -H html, -k ignore skipfile
    ;w-z 30jun04 proxy-is-squid added
;002b   24oct00 added values-file
    ; 06nov00 added 'every' and Chunks

(copyright) 2017 and previous years FingerPost Ltd.