poss - merge RESET and TRIM ??

NOTE - These FipHdrs are used internally by webwire
    C0-9 for cookies
    F0-9 for forms
So pls do NOT used for 'fiphdr:XX ....' etc


FOR HTTPS/port 443, please use the 'webwiressl' version of this program.

Webwire goes and gets pages of data from Other people's web sites automatically
and then sends those pages to your destination - usually the editorial system -
in the normal Fip fashion.

These can be updates of weather, financial data, sports results, backup for
Wire services if the satellite is down, graphics, software. In fact most

It can be used either :
    - on a timed basis to get regular known pages.
    - on demand by sending a file into spool/webpoll with the FipHdr field DF set
to the parameter file required.

What it can do -
    - drill down links to several layers deep,
        optionally ignoring the data on the top levels.
    - select only certain links - either in XML, HTML, JSON or CSV
        - you set masks to filter which to get and which to ignore.
    - logon automatically to protected sites
        and save Cookie information for use in later accesses.
    - fill in standard form data to get make on-demand searches.
    - strip or rework HTML tags to make the data more presentable.
        This is meant for reasonably simple pages while more complicated ones
        will be routed through 'ipsgml' and/or 'ipxchg'.
    - Use an external list of values to make several grabs to the same
        but varying the search data for each hit. eg to pull all the values of a
financial index. (This we call a 'values-file')
    - Grab a 'id' from a List-of-items from a REST web service and then
sequentially call all items

What it cannot do -
    - play tunes.
    - run javascripts or any other applet type affairs. (yet..)
    - run FTP, GOPHER or whatever (for these and especially FTP, see program
'ipftp' and 'iptimer').

The current version is primarily for getting text data but can be used for
images etc if required.

There is a TUNING mode to be used for setting up a new link and trying to clean
up the relevant parameter file WITHOUT sending (possibly) live data to the
required destination.
    - This shows the data with escaped unprintables and '$' at the end of a line.
    - All links and forms are also displayed.
    - Any pages saved in Tuning mode are NOT sent to the normal output queue
(spool/2go) but are left in spool/webtest for future perusal and/or deletion.
    - To run, choose your parameter file in tables/webwire and run 'webwire'
manually in a window:
        webwire -T AUS.STOX | more      for prompt before calls
    or  webwire -A -T AUS.STOX | tee aussies    for no prompting

There are Two (sometimes three) types of parameter file :
    1. Main Parameter file which sets up the polling of certain pages at set times
(if any).
    2. A Page Description file for each site/page accessed.
    3. Optional lookup file of values where you want to repetitively hit a site
changing certain values each time. (eg a sport site for several divisions or a
list of stox to get)

----- Main Parameter file -----

The syntax of the Main Parameter File - by default tables/webwire/XWEB :
    ; comment line
    poll:(pram file)    day:(MTuWThFSaSu)   time:20:30  mustget:

In detail, the 'poll' keyword :
    Pram file is the name of the Page Description file - see below for its syntax
    day:    Day of week to run the job :
            M   Monday
            Tu  Tuesday
            W   Wednesday
            Th  Thursday
            F   Friday
            Sa  Saturday
            Su  Sunday
            X   Every day.
            Z   Weekdays M-F.
            Case is NOT important.
            Commas (but NOT spaces) may be used to separate.
            Default is every day.
    time:   Time of the day on 24 hour clock.   Default is 18:00.
    every:  interval between grabs          Default: none
        every: (mins)   [(optional) start:(starttime) end:(endtime)
        every:30    start:07:30 end:19:00
        The minimum interval is 1 min and maximum is 3 hours (ie every:180 mins)
        You may also specify in seconds using 'secs' or 'seconds'
        immediately after the number (with no spaces)
            every:10secs    start:09:30 end:09:50
    poll:AP     day:ALL     time:20:10
        Get the Page file tables/webwire/AP every day at 20:10
    poll:Forex  day:MTuWThF time:16:30
    poll:Forex  day:MTuWThF time:16:40
        Get the Page file tables/webwire/FOREX every week day at 16:30 and 16:40

There can be none or up to 200 polls in the main parameter file.
Note that the page is grabbed ONLY if the program is running.

----- Page Description Parameter files -----

The individual Page description parameter files are also in tables/webwire. The
syntax of these are :
    ; comment start with a semi colon like this

    url:    Full url of the page.               default: none
        There MUST be one and only one 'url:' specified.
        You can also specify the page, cgi and any subparameters.
        eg  url:www.fingerpost.co.uk

    dest:   Fip Destination for the files           default: WEBDATA
        This is the 'DU' FipHdr field as per the USERS file.
        eg  dest:w3saves

    use-tls: no/yes
    use-ssl: no/yes
    use-https: no/yes
        Use Secure Sockets Layer (TLS/SSL) - also called HTTPS  default: no
        If the url starts 'https://....' then this command is NOT needed.
        (There is also a setup option for openssl s/r to use either Bio or SSL
functions for the secure connection
    ssl-method: tls1.3 tls tls1 tls1.1 tls1.2 sslv2 sslv3 sslv2and3
        Version number to use for TLS/SSL       default: 999 for current default (2 or 3)
        (only the digits are significant, so add other text to make it readable)
        For 'modern' connection, pls do NOT use sslv2 ! as it is deemed insecure
        If default it will check the available list and pick the highest.
        The default is currently 23 which on a modern server is sslv3 and tls1_2 !)
    ssl-password: (password)
    ssl-passwd: (password)                default: none
        Optional password if the handshake requires a shared secret
    ssl-key: (name of a certiticate key file)       default: none
    ssl-cert: (name of a certificate file)      default: none
    ssl-root-cert: (name of a root PEM certificate file)    defaunt: none
        Optional certificates are in tables/ssl unless name starts with '/'
    ssl-verify: yes/no  verify server certificates  default: yes
    ssl-ciphers: (list) acceptable ciphers
        (use 'openssl ciphers' to list)
        pre 2021oct default: 
        pre 2017 default:  "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS"
    ssl-display: yes/no display SSL connection details  default: no

    port:   Port number of the Remore Server.       default: 80
        This forces the port to be this if none is specified.
    nofiphdr: Do NOT add a Fip Hdr to the file.     default: yes pls
    source: Fip Source of the files. (FipHdr 'SU').     default: XWEB
        Unless 'noarchive' is specified, all data files will be archived under this
name in log/data.
        This can be in FipSeq so that 'combie' can be used to set a default..
    noarchive: Do NOT archive these files in log/data.  default: archive
    maxlevel:3  Maximum no of levels to drill down. default: 1
        Normally the URL you have requested is the data you want.
        However if that is an index page with links that may change, it may be these
lower-level pages that are needed. 'maxlevel' states how many levels of link
the actual data pages are.
        Default is 1 = do NOT drill down any of the links.
        Note that level 1 is the first page.
    ignorelevel: Used with 'maxlevel' where the information     def: no
        required is on a linked page and NOT on the first page,
        use 'ignorelevel' to ignore all those pages on intermediate levels.  Note
that level 1 is the first page.
        eg  ; ignore levels 1, 2, 4 and 6
    matchlinks: Only follow links which match this mask.    def: all links
        Used only if 'maxlevel' is greater than 1.
        There can be many 'matchlinks'.
        Use the '*' as a wild card string and '?' as a wild chr.
        eg  ; get all links ENDING 'html'
    matchforms: Only process forms which match this mask.   default:no forms
        Used only if 'maxlevel' is greater than 1.
        There can be many 'matchforms'.
        Use the '*' as a wild card string and '?' as a wild chr.
            eg  ; get all forms ENDING 'asp'
    matchframes: Only follow frames which match this mask.  def: all frames
        Used only if 'maxlevel' is greater than 1.
        There can be many 'matchframes'.
        Use the '*' as a wild card string and '?' as a wild chr.
        eg  ; get all links ENDING 'html'
    matchkeys: Only follow links which match this test. def: all links
        Used only if 'maxlevel' is greater than 1.
        Used only for 'linktag' where an attribute MUST be set for the link to be
        There can be many 'matchkeys.
        Use the '*' as a wild card string and '?' as a wild chr.
        eg  ; <hotel id=33 name="Fawlty Towers" url="http://www.ohnonotagain.com"
status="current" />
    match-case-sensitive: yes/no
        all matches and ignores can be case sensiive or in-sensitive
        DEFAULT changed 05u to INsensitive - previously sensitive.
    match-dedup: (FipSeq)
        Check and ignore Sequencial duplicate items with (possibly) diff urls -
        It can be used the same as skip-save-data if that else use \W$ for the normal
grab url

    force-lower-levels: (levelNumber)
        When data is on more than one level - maybe a text page has a link to a PDF
and you need both bits, use this to get all bits of this element before
continuing with the next element.
        The default (without this parameter) is to get all this level and then all
the next lowest level etc.
        ; force the lower levels below level 2
    mime-type-fiphdr:(2 letter FipHdr field)
        if the MimeType is present, add the mime-type  to the fiphdr as  this fiphdr
    level-fiphdr:(2 letter FipHdr field)
        add the Level of this file to the fiphdr as this fiphdr field
        This can be used for option inside the parameter file:
            option:V1   AL,,,,1,
            option:V2   AL,,,,2,
    level-link-fiphdr: (FipSeq - 2 letter FipHdr)
        This gives access to the top level link in force-lower-levels
        eg  force-lower-levels:2
        so C1 is for level 1 link, C2 for level 2 etc
        If you want only a part of the link use FipSeq to pull apart
        there is no default

        Some BIG feeds will only return the first 'n'100 items of a list of items -
eg S3 is up to 1000
        Use this to FipSeq that an input tag is saved into a FipHdr field and the
webwire loops for more
        eg  ; FH for next page
            ; save the contents of NextContinuationToken tag at the top level 1
            fiphdr:JY   level:1 tag:NextContinuationToken
            ; only add 'continuation-token=' and the Token if the Token HAS data
            option:VY   JY

    skip-links: Name of a file in /fip/fix/webwire holding names of links
        and forms already accessed; so that only new ones are tried.
            eg  skip-links:webwirelinks.\$d
            default: none
    skip-details-tag: (tagname) extra details (such as a publishdate) for check if
existing links have been updated
            see below on the section for RSS feeds
            default: none
    skip-purge-after: (hours) Number of hours to keep the skip entry
        default is 24.  You might want to tune this :
            - make bigger if sites add/take off old material
            - reduce the time if the same link is used for different data
    skip-save-data: (FipSeq field)
        Sometimes there is some data in the link which changes for every access -
such as a Cookie or SessionId
        eg the first access might get
        and the second
        But they both happen to be the same file - messageId=268482
        Use FipSeq to just get the data that contains ONLY the information you want
to save.
        Certain FipHdr fields hold relevant info:
            WX is the field marker '^'
            WS is the skip details tag (optional - see above)
            WT is the type - 'a'-anchor
            WL is the level no
            W$ is the actual link - anchor, form etc
            S$ is the actual hostname or IP address
            WH is the associated display text from an anchor tag
        In the above example :
            ; split on the '?' - get the second field
            repeat:Q1   W$,?.2
            ; skip string is now 'messageId=268482' - note the FipSeq needs a backslash

    skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
        the skip file when changed (see doc on 'ipbalan')
        This is often used where a second system could be used as a redundant server
        if the main system fails. (see also -B input switch)
    ignorelinks:    Of the Links found, skip any matching this mask. default: all
        Used only if 'maxlevel' is greater than 1.
        There can be many 'ignorelinks'.
        Use the '*' as a wild card string and '?' as a wild chr.
        eg  ; ignore any links pointing at any 'netscape' or 'microsoft' site
            ; ignore any links requiring 'ftp:'
            ignorelinks:ftp:// *
            ; ignore any links to other sections
            ignorelinks:../ *
            ; ignore any links to any index
    httphdr: Extra lines of HTTP header you may need.   default: none
        Remember to add a NL at the end of each line.
        There can be multiple httphdr lines but pls remember to add a '\n' at the
        end of each one. (or you can try to force all on one httphdr line!)
        eg  httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n
            httphdr:User-Agent: Mozilla/4.0\n
            httphdr:Host: wibble.wobble.com\n
        see below for 'useful, common header lines'
        ** ALL basic-authentication MUST BE HIGHER IN THE PARAMETER FILE THAN httphdr
OR proxy-logon
        Normally the httphdr is only for a single host. So if the 2nd or subsequent
level is to a different host, by default, nothing defined as 'httphdr' will be
        if 'yes', the option adds the httphdrs to all grabs
        Normally the httphdr is only for data grabs NOT for getting thru the Proxy.
        if 'yes', the option adds the httphdrs to the proxy call

    basic-authentication: (fiphdr field) (logon:password)
        Build a FipHdr field with the BasicAuthentication formatted logon:password
        Pls remember to escape any funny chrs - like backslashes
        ** ALL basic-authentication MUST BE HIGHER IN THE PARAMETER FILE THAN httphdr
OR proxy-logon
        eg  basic-authentication:BA DOMMY\\zipple:Ardvark99
            httphdr:Authorization: Basic \BA\n

    method: POST/GET/DELETE/PUT etc             default: GET unless 'post:' is specified
        normally this is a single UPPERCASE action - with NO spaces.
    post:    Post a Form                    default: get url
        see below for processing a form using method=POST.
    filename: Filename for the output file in FipSeq.   default: WEB\$Z
    newname: ditto
        If this does NOT start with a '/' it is left under the Output Queue as
specified on startup (default spool/2go)
        eg  filename:AFP\$d.\$z
        eg  newname:#SN:\JC.\JK.\JZ#XX:\$u.z\$z.v\@v
        Note \@v is no of items in this file
        It is ignored if a -N (forcename) is specified as an input parameter
    supercede:(FipSeq which should resolve to yes|no)   default: no
        if supercede:yes, the contents of any existing file is overwritten
    striptags:(yes|no) Strip tags and attributes        default: no
    wild: (FipSeq)  Character used as a Wild String for default: '*'
        eg  wild:\377
    singlewild: (FipSeq) Character used as a single     default: '?'
        Wild chr for 'matchlinks/ignorelinks'.
        eg  singlewild:!
    number: (o|d|h) Number system for FipSeq        default: octal
        octal, decimal or hexidecimal
        The following are all equivalent :
    before: FipSeq String to add before any data.       default: none
    after:  FipSeq String to add after any data.        default: none
    script: Script to run on ths data of the incoming file. default: none
    outque: Output folder (in FipSeq)           default: spool/2go
        This overrides both the default and the '-o' output switch
        except for Testing/Tuning mode where the file is forced to spool/webtest.
    log:    FipSeq custom logging for the item log.     default:\SN \SU \EF : \EH,\EP
        This logs each Page grabbed
        Note that
            EH or ST    remote site host
            EP or SP    remote site port
            EN or SF or SG  remote site url     SG is the actual link, the others are the
link used to grab
            EF      parameter file used
        The default is that no incoming files are logged by webwire
    custom-log: FipSeq custom logging for the item log.     default: none
        This can be used to log link details in a custom log

    log-errors:w (warn)         - for all communication errors
    log-https-errors: w (for warning)   - for all HTTPS comms errors
        Any failure to go secure in https connections are flagged as warnings
        The transmission is always aborted. This parameter affects only the logging.
        default:  !x for failures

    extra-pre: Extra FipHdr fields (in FipSeq) to be added to the output.   default:
        To separate FipHdr fields, pls use a '#' or a newline.
        extra-pre is added as soon as the file is read - so may be used for
information in the URL
        extra is only used for any output file and is not used at all for any other
        extra-grab is added before each grab
        eg  extra:ZH:NYNZ#DI:Headline News#QZ:333
    tag:    FipSeq String to replace the start tag      default: none
        such as <H1>. There can be many 'tag's.
        eg  tag:P       {Para}\n
    endtag: FipSeq String to replace the End tag        default: none
        such as </P>, </TITLE>. There can be many 'endtag's.
        eg  endtag:TITLE    \n
    getimages: Also get all the images
        By default all images - *.gif or *.jpeg are ignored.
    keep-alive: yes/no                  default: no
        Just that ! default:no
    http-version: 1.0 or 1.1                default:1.0
    only-get-if-modified: (FipSeq for yes or no or etag)        default: NO for get data
each time
        This will check the remote server for the time the page was last modified.
This does not work with old servers and some set to HTTP/1.0.
        If remote data has been modified since, data is grabbed and processed
        If not - it is ignored (unless logging is lon)
        If the parameter is 'etag' then any incoming ETag tag is saved and subsequent
request use 'If-None-match: (Etag)'
    if-modified-suffix: (FipSeq)
        'only-get-if-modified' uses a save file named by the parameter file (or the
poll name)
        If there are several grabs using the same parameter file but need their own
separate times.
        (Otherwise they would all use the one, latest time for all grabs ! - not good
        This adds a suffix to the saved time file
        combie:QJ   AJ|BJ,json

        When matching for skip files, ignore this key-value pair.
        see the section below on Repeat Offenders
    max-items: (number)                 default: 0 for all
        Max number of items to grab per session
        Some sites only allow you to read 5 or 10 items before blocking you.
        Use this to creep under that total.
        (from 6a20) The number can be in FipSeq.
        Note this is the number of files produced - ignoring Skipped files
        So it the number of linked grabs is 2 * a FipHdr field, use FipSeq 'sum' to
        eg is AL:7 is a FipHdr field and 2 files per link are generated - THUMBNAIL
            sum:Q7  (\AL * 2)
        There can be a subparameter - level:(number) - where there are multiple
levels and you want to grab the all the items on the lowest level BUT need to
track the previous level
    pause-between-files: (secs)
        Gap/wait/pause between grabs    default is 5 for standalone, 1 for iptimer
        This is overridden by the -w input switch
    one-output-file: Put ALL data in a single output file.
        The default is one file per page/access
        Use this with 'values' to create a single output file.
        This ONLY uses the FipHdr of the first file if 'values' have been specified.
    end-of-document: Where a site is sending really really crap HTML - or XML
        use this to state what the last tag.
        For no checking at all : end-of-document:
        Default:        end-of-document:</HTML>
        See below for a standard-fingerpost-rant on crap HTML.....
    end-of-cookie-page: end text which signifies the end of a logon or cookie page
        This is rarely changed.
        default is </HTML>
    connection-retries: (number)
        No of retries that a connection or a  broken connection (ie before a response
is received)
        Some slow sites are throttled and will kick the n+1 th connection off before
servicing it.
        Use this to retry. Default is 1 connection - ie NO retries.
    connection-timeout: (secs)
        Slow, busy sites, may take a lot longer than normal to connect to. Use this
to adjust the time to connect.
        Default is 90
    wait-end-timeout: (secs)
        For slow, busy sites, data - especially big files - may take a lot longer
than normal to be retreived. Use this to expand that time.
        Default is 120 (it should be divisible by 5 for some arcane reason)
    pretend-301: (3 digit number)
    pretend-302: (3 digit number)
        Ignore redirects (HTTP return code 301/307 or 302/308) and assume they are
this return code
        this will take a 301 and save the data as through it was an incoming file.
        Save /Dump a copy of the each request and response and data in a dump file in
/fip/dump default:no
    dump-filename: (FipSeq)
        Name to be appended to the dump filename in /fip/dump   default: none
    no-data: (FipSeq string in place of data)
        Do not get/send the data - just this string
    data-is-binary:(yes/no/maybe - can be FipSeq)
        Data files at the lowest level are binary or not
        default is check for <?xml, Tiff, Jpeg, MsWord/Office, EPS and PDF
automatically; otherwise it is treated as text
    ignore-mime-if-binary: (yes/no - can be FipSeq)
        if yes = Strip the MimeHeader off binary files
        default is no to leave it on - so you know what the file really is !
For Socks 4/5 - use these parameters to control
    use-socks:4/5 yes/no (yes is same as 5)
    socks-host: (hostname of the socks proxy)   no default
    socks-port: (port number of the socks proxy)    default: 1080
    socks-user: (user name for the socks proxy) no default
        if nothing specified, assumed that there is none
    socks-pwd: (password for the socks proxy)   no default

For old-style HTTP Proxies :
    proxy-server: If using a proxy, these are the name and port to aim at.
    proxy-logon: This is the logon and password to get thru the firewall if
required. The format is (logon) (colon) (password) and is converted to base 64.

        ** ALL basic-authentication MUST BE HIGHER IN THE PARAMETER FILE THAN httphdr
OR proxy-logon
        To generate use basic-authentication or:
            echo -n "logon:password" | sffb64 -i
        eg  echo -n "chris:sleekpanther" | sffb64 -i
        gives   Y2hyaXM6c2xlZWtwYW50aGVy
    proxy-is-squid:yes/no   Is the proxy a Squid ?  default: no
    proxy-handshake:yes/no  Does the proxy need to CONNECT first ?  default: no
        If the proxy is a Squid, this MUST be NO

    logeachfile:(dest) Send a Success/failed msg to this destination
            for each file. There is no default. This log file is
            just a FipHdr with the following extra fields :
                DR-File Sent OK     DR:ok or DR:error
                DG-Will Retry later DG:retrying, DG:stopped
                DT-Some message text    DT:No connection
            default: no log created.
        The text for the DR and DG can be in FipSeq and so can contain
        FipHdr and other variables. As they are FipHdr fields, please
        do NOT put NL, CR etc in the fields.
        Note that System Variable \$q holds the time taken for transmission.
    DRgood:(text)   Message for the FipHdr field DR on a   successful tx
            default: ok
    DRbad: (text)   Message for the FipHdr field DR on a unsuccessful tx
            default: error
    DGcont:(text)   Message for the FipHdr field DG if, after an
            unsuccessful tz, another attempt will be made.
            default: retrying
    DGstop:(text)   Message for the FipHdr field DG if no further
            attempts will be made as the file was sent successfully
            or the maximum no of attempts has been tried.
            default: stopped
    fiphdr-for-logeachfile: (FipSeq) or
    msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
            'logeachfile' or 'loglasterrfile' msg. This should be in FipHdr
            format and be in FipSeq. It can be used to pass FipHdr fields
            in the outgoing file into the log file.
            eg  msgeachfile:    DF:logdial\nSS:\SS\n
            default: nothing added

        convert-CDATA-sections:no   - no dont ! (default)
        convert-CDATA-sections:zap  - no but zap the '<!CDATA[' and ']]>'
        convert-CDATA-sections:yes  - yes pls and zap the '<!CDATA[' and ']]>'
        convert-CDATA-sections:preserve - yes pls and leave the '<!CDATA[' and ']]>'
        Normally a CDATA section like :
            <![CDATA[ Vongerful Vondafool C&oe;penh&areing;gen <99thisIsAnon-compliant
XMLtag> ]]>
        is considered a single, raw string of XML/SGML data. And all the tags and
entities (like &lt;) are not changed either.
        Use this parameter to convert them.
        Note that you should use this option CAREFULLY if any tag in the CDATA is the
same as a tag in the main envelope. See below for more comments.

To save the contents of a particular Tag or TagAttribute, use the 'fiphdr'
keyword :
    fiphdr:(FipHdr field)  (optional subkeywords)
        Either  tag:(name of tag)
                specify the tag name which contains the data required.
        Or  data:(FipSeq)
                for adding FipHdrs with standing data.
                fiphdr:TT   data:\$e\$y\$i\$d
                will create a FipHdr field DT with the current date in it
        Or  tag:(name of tag)@(name of attribute)
                specify the tag name and the attribute name which contains the data
        Or there can also be a 'key' parameter for selecting the data ONLY if there
is Key attribute with its data equal to a certain string:
            eg: if the tag is <meta name="category" content="f"/>
                fiphdr:NC   tag:meta@content key:meta@name=category
                Get the contents of the content attribute of 'meta' where another attribute
called 'name' has the value 'category'
            or  fiphdr:NC   tag:meta    key:meta@name=category
            or  fiphdr:NC   tag:meta@name=category
                Get the data for the 'meta' tag that has an att 'name' = 'category'
            Double quotes around the Key Data are optional unless there are embedded
spaces. The Key Data can be in FipSeq.

        For any of the tag options, use 'dup' to flag duplicated fields.
            dup:(optional separator)
                This field may be duplicated. Duplicate fields are separated
                with a space unless a separator chr is also specified.

        Where there might be embedded tags inside the main tag, use 'repxml' to
specify a replace string
            eg fiphdr:AL    tag:TD  repxml:+\s+
                and the data is <td>abc<br>efg<br>line3</td>
                will give   AL:abc+ +efg+ +line3

        As some FipHdr fields have distinct meanings - SN, DU, DP etc - please use
other 2 letter codes starting N or Q.
        In the current version of webwire, you CANNOT specify trees of tags ie
fiphdr:AA tag:entry/id.

    eg  fiphdr:NA   tag:itemid  dup:+
            get the data from each <ITEMID> field. If there is more than one,
            they are separated by a '+'.

    fiphdr-file:(Filename in /fip/fix/webwire/fiphdr)
        This allows data to be stored as FipHdrs at the end of the session - and read
at the begining of the next
        So items like Sequence numbers and time-of-access can be passed between
            ; default name
            combie:QA   WA,default
            ; save and possibly reuse the FipHdrs ....
            repeat:JQ   J1,+,1
            repeat:JD   J2,+,1
            fiphdr-save:BQ:\JQ\nBD:\JD\nXX:some comment\n
        ** This must be lower down the parameter file than any FipSeq if you are
using FipHdr fields as the example above !
        There can be multiple 'fiphdr-file' - all of which are read as the parameter
file is read.
        But if there is a fiphdr-save, ONLY the last 'fiphdr-file' is stored to.

        Add the FipHdr to each file on every level - default: no
    fiphdr-hash: (single chr in FipSeq)
        This will replace a Hash '#' in a FipHdr field (as Hashes are normally
end-of-fiphdr field)

    meta-save-file: (Filename)
    meta-save-on-tag: (tag name)
        This meta file is appended to on the End-of-tag specified (or end-of-file if
no tag specified)
            ; save these fields to the lookup file
    reset-fiphdr-on-tag: (tagName)
        Trim the FipHdr - and extra, added fields - on the end of this tag to the
same position when the tag started
        This can be used in meta-save to make sure that FipHdr fields from one group
of tags to not linger and are not used for the second or subsequent
        default: not used.
    grab-on-tag: (tagName)
    grab-on-endtag: (tagName)
        Any links should be grabbed at the start or end of this Tag
        default: all links are grabbed at the end of the page
        An extra parameter may be specified on the same line for level eg
            grab-on-endtag:VALUE    level:3
        NOTE that grab-on-endtag does not trim the FipHdr (as we might need the extra
meta for a fiphdr-save). So use reset-fiphdr-on-tag with the same tag to trim
(if there is NO fiphdr-save)

        Retry links which return a 404 Not Found error. Max is the number of retries
and Gap is the pause in seconds between the retries
        Use this for those sites which are a bit slow to add the data files the links
point to.
        If the files really are not there - and you do NOT want to abort the
transmission - use 'retry-404-error:ignore' to continue with the next grab
        OR you can use retry-404-error:move and retry-404-queue:(queue in spool) and
retry-404-fiphdr:(FipSeq) to send a item
        Retry links which return this system error - code can be any 3 digit number
above 400.
        Max is the number of retries and Gap is the pause in seconds between the
        Use this for those sites which are a bit slow to add the data files the links
point to.
        If the errors continue - and you do NOT want to abort the transmission - use
'retry-500-error:ignore' to continue with the next grab
        OR you can use retry-500-error:move and retry-500-queue:(queue in spool) and
retry-500-fiphdr:(FipSeq) to send a item
    save-data-path: (Fipseq pathname for data)
        This puts the data of the incoming file into this folder and creates a FipHdr
        file that contains 2 FipHdrs containing the full path/filename
            SX: and FTP_EXTERNAL_FILE:
        (ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE)
            eq  save-data-path:/fip/data/jpegs/\$e\$y\$i\$d/
        Use this for big files that you do not want to copy around the Fip Spool
    save-data-filename: (FipSeq name)
        Use this to specify exactly what the the 'save-data-path' name should be
        default is (incoming filename).(time).(seqno)
        eg  save-data-filename:\HR-\SU.raw
    save-data-balance-group: (Balance group)
        Balance all save-data files to the following group. default: do not balance
    save-data-balance-folder: (Balance folder)
        If balancing, put the token in this folder under spool. default: 2balance

    max-children:(number in Fipseq) - same as the -E switch. default: none
    forks-per-sec: (number in FipSeq)
        Throttle forks to this number per second    default: no throttle

-- More Complex sites ------

-------- Oauth2, Oauth digest and AWS notes --------

-- For accessing Oauth2 protected assets - eg GCP Cloud Storage or G-Drive or
Microsoft Azure features

    ; OAUTH2 authentication as per Google GCP or Microsoft Azure

        Use OAUTH2 to grab/use an access-token or Bearer token eg for Gmail access
        default is NO
    ; We need an access token

    ; which flavour of Oauth2 ? - only the first letter is meaningful
    ; oauth-flavour: Google (Gmail) or Microsoft (Office365)
    oauth-flavour:microsoft for office 365

    ; Current token file will be saved in /fip/fix/goauth2

    ; Credentials file in /fip/tables/cert

    ; sffoauth and imapwire

    ; Script to run when token expires - approximately every 12 hours
    oauth-refresh-script: (Script in FipSeq)        script to generate the access_token
using a refresh_token
    oauth-refresh-script:/fip/bin/sffoauth -z wire/IMAP.O365.OAUTH.SEA -c \OC -t
\OT -H '#WN:\WN' -a

    These 5 FipHdrs are use to generate, check, add/renew permissions to access
the remote data - normally Gmail or Office365

    oauth-client-fiphdr: (FipHdr)   default: IC
    oauth-secret-fiphdr: (FipHdr)   default: IS
    oauth-access-fiphdr: (FipHdr)   default: IA
    oauth-refresh-fiphdr: (FipHdr)  default: IR
    oauth-expiry-fiphdr: (FipHdr)   default: IX

-- For accessing other Oauth protected assets - like twitter
    OAUTH digest - such as twitter and dropbox use

There are three parameters to define to
    ; this is the salt key
    ; string to use
    ; fiphdr to add to
    ; type sha1

    oauth-signature-key: (FipSeq)   Key for Oauth; normally the ConsumerKey and the
    oauth-signature-data: (FipSeq)  Data string to encode - see your remote api doc
for what needs to be included and how it should be formatted
    oauth-signature-fiphdr: (2 letter FipHdr field) FipHdr which will hold the
    oauth-signature-type: (type)    Signature type
        valid types are md5, sha1, sha224 sha256 sha384 and sha512

-- For AWS grabs, use the same pararmeters ars oauth (almost!)

    aws-signature-key: (FipSeq) Key for AWS; use | as a sep :
    aws-signature-fiphdr: (2 letter FipHdr field) FipHdr which will hold the
    aws-signature-type: (type)  Signature type
        valid types are md5, sha1, sha224 sha256 sha384 and sha512
    aws-request: (FipSeq)   FipSeq string to hash using the signature key
    aws-data-fiphdr: (2 letter FipHdr field)    FipHdr field which will hold the
payload sha256 hash
        if POST, the data part which needs to be hashed; for GET, it is left blank
(obviously as there cannot be a payload)
            the hash is added to the last line of the 'aws-request' either as a fixed
string (for GET) or as a FipHdr field (aws-data-fiphdr)
            if GET, this can be ignored and (for SHA256), this is
            eg sffhmac -I "" -Z sha256 -S -D -H
    aws-data-md5-fiphdr: (2 letter FipHdr field)    FipHdr field which will hold the
payload md5 hash
        This is used for the S3 Content-MD5: mime header. as an alternative to the
SHA256 hash in x-amz-content-sha256: mime header which should be set to
UNSIGNED-PAYLOAD if you use md5

EG :
; if split into FipHdr fields for :
;   JA-accessKey, JB-secretKey, JH-sha of payload,JI-aws-id, JP-Url Params,
JQ-sqsque, JS-service, JR-region, JZ-utc datetime
combie:JA   AA,-noAccessKey
combie:JB   AB,-noSecretKey

combie:JI   AI,349445556777
combie:JM   AM,GET
combie:JQ   AQ,fiptest
combie:JS   AS,sqs
combie:JR   AR,us-east-99
newdate:JZ  gmt unixdate=\$p "\ZZ\ZM\ZGT\ZH\ZF\ZEZ"

; For GET - choose either
fixed:JH    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
; or
; aws-data:
; aws-data-fiphdr:JH


- In this case, the default request boils down to :


-- The links are not in the normal Anchor or Frame tags.
If the Site returns an XML feed rather than HTML, you can specify what the
contents of which tags you want to play with. There can be up to 20 tags
or  linktag:(tagname)@(attribute)   (for version 05u onwards)

or to imitate the defaults :

- sites which return other data which is not xml - such as CSVs
    data-type:CSV   (can be CSV for comma sep format, JSON, PSV for Pipe sep, TXT)
        define the column containing the link to the data
        define the column containing the headline
        define the column containing the skipdetails

-- RSS feeds
Sometimes a link can point to data which gets updated and there is a second tag
which gives either a unique-id or a date/time which you need to track for any
changes. Use the 'skip-details-tag' to specify the second tag - it is the
combination of the 'linktag' and 'skip-details-tag' which should be unique.
For general RSS 2.0 feeds, this can either be 'pubDate' or 'guid' :
In RSS feeds there is often a fake 'link' at the top which is the channel.
Usually you do not want this one - often it is a URN not a real URL, so use
'matchlinks' or 'ignorelinks' to bypass it.
if more than one skip details are needed, up to 9 skip-details-tag-X can be

-- If the data in the link is not complete..
Use templates to slot data from a link into another call. This is again used
extensively for XML work - like soap.

It uses either just a template (in FipSeq so you can add Header Fields etc) or
a template AND a template file if there is a lot of data.

There are 8 templates for levels 2 to 9. 'maxlevel:' and 'ignorelevel:' must
always be used with these to specify which one you need the data from.

A levelXtemplate on its own will generate a GET.
To POST something, you will also have to specify a 'levelXdata: (FipSeq)' eg
    ; level 3 - To get THAT file is always a POST of
    ; .. using the different EVTVAL and VWSTAT

will force as
    POST /proximity/Admin/FileManager.aspx
will data filled in for fiphdrs A2, G8 and G9 eg


Note there is no level1template as that is the same as the URL:.. BUT there is
a 'level1template-file' version. In this case the URL: should be just that.

There is a little used parameter levelXmime which can be used to change the
Content-type / Mime type just for that level.

The template-files are normally in /fip/tables/webwire. They are NOT force

The default Content-Type for POSTing data or forms,
'application/x-www-form-urlencoded', sometimes needs to be changed for
It can be changed with the 'levelXmime' parameter. For example, soap normally
likes a content-type of 'application/soap+xml':
    level1mime: application/soap+xml
unless you are Microsoft of course who usually/sometimes want
    level1mime: text/soap

The 'W$' in the example is because each link is put into a temporary FipHdr
field called W$ as it is being used. If the link data is too much or too
little, use FipSeq to chop/add/replace.
Eg  if the data in the link is "nine:/rt/newsart/Id="z1jit4":text"
    And you want a link like
    use repeat:R1   W$,:,1
        repeat:R2   W$,:,2
        repeat:R3   W$,:,3
        ; if there is no 3rd field, use 'xml' instead
        combie:W4   R3,xml

-- Values -------
Values can be -
    - EITHER a file containing lines of values to be used to repeatedly grab data
for a single file.
        using values-file:(filename in tables/webwire)
    - OR a sequential number
        using values-seqno:(min value):(max value):(incremental value)
        plus    values-seqno-fiphdr-from: (FipHdr field containing the From seqno - ie
start grabbing from the NEXT id after this)
            values-seqno-fiphdr-to: (FipHdr field containing the To seqno - ie each
seqno until and INCLUDING this one)
        Fipseq to POST a form or GET a link from a line in the
        values file. See below for a description.
    values-sep: Separator chr for splitting fields in the values file.
        default is a pipe - '|'
    values-leave-spaces: Normally leading spaces are trimmed from each
        field in the values file. Use this to preserve them.
    values-parallel: (Number of Simultaneous Hits)
        For 'values' the default is to run the hits serially, one after the other has
finished. Use this to send out a number of hits at the same time which should
reduce the total time by a large factor. However, you should check with the
remote and test what the number should be. For Apache sites for example, 8 is a
common default setting.
        eg  values-parallel: 10
    values-fiphdr: Normally fipHdr W1 will contain the first field of the values
file, W2 the second etc.
        So data can be specified by \W1
        Use this parameter to specify another field - ie if W1 is being used
        ** Note that if you are using iptimer to start webwire running a values file,
the Wx fields will be zapped in the output file.
        So in this case, always use 'value-fiphdr:' with a different FipHdr if you
want to use the Values in iproute or another downstream program.
        eg  values-fiphdr:R1
    values-pause: (secs)
        Gap/wait/pause between Grabs using the next value   default is 0 for none
    values-comment: (single FipSeq chr)
        comment - ignore any value line which has this chr as the first non-blank chr
default: ';' - semicolon
    values-allow: (single FipSeq chr)
        allow - only process values lines which have this (case-insensitive) chr as
the first non-blank chr default: all
    zap-values-file: (yes/no)
        Delete the values file after it has been used.  default no
        Only files in /fip/x/ starting TMP.. can be deleted.


Note that in the FipHdr - unless the 'nofiphdr' keyword has been requested, the
following fields will be filled in :
    Day and time in the normal HH,HD,HY etc fields
    ST  host
    SP  port
    SF  url - path/filename being grabbed
    SG  url - path/filename with is the link
Where webwire is sitting on a scrolled queue (using -i), the folder name is in
EQ and the filename EN (with all '#' replaced by the chr chged by

Extra FipHdr values are
    \@v is no of items in this file
    \@i is ths childId if spinning off children

Input Parameters (all optional) :
either  -i : scrolled queue             default: no default
        This checks the folder and for each file, checks the FipHdr for 'DF' which is
used for the name of the parameter file to run against
        This allow a variety of parameter files to be run
or  -1 : Run a single time and exit         default: spool
        The parameter is the name of the individual parameter file in tables/webwire
(ie NOT The top or main parameter file)
or  -T : Tuning mode                default: spool
        Display links and data for the page requested. Runs only that page and then
        The parameter is the name of the individual parameter file in tables/webwire
(ie NOT The top or main parameter file)
    -A : In Tuning mode, do NOT prompt before searching a link  default: prompt
    -a : log the actual link of each accesses in the FipLog     default: no
        This can be quite a lot of logging if you are grabbing lots of files !
        But is quite useful when starting/adding a new feed.
    -B : default balance group for skip files           default: none
        (see skip-balance-group parameter)
    -C : warm restart for cookies/api-keys      default: always ask for new
cookies/api-keys for logon
        ie do NOT re-logon if the previous session logged on and saved the cookie or
        if any apikey is missing or has timed out, all cookies and api-keys are wiped
and webwire needs to be re-run to logon and download.
        see note below
    -d : done folder for -i scrolled queue      default: none
        This can be overwritten by the 'doneque:' parameter
    -D : display the Request and Response       default: do not
    -e : exit with the Result Code of the last grab.    default: normal program exit
        The Normal exit is 0 if ok, negative number if not
        With -e this will be 0 for ok, and -1 (timeout) but 4XX or 5XX for page
    -E : maximum number of threads up to a max of 100 (not Win2k).  default: 1
        Note this is also a hardware limit in that small systems may not be able to
run as many.
    -f : path and filename of the output file if a non-200 HTTP code is returned;
default: fip standard
        use this to leave the file(s) in a non-std folder.
        ++ NOTE this was -E before version 6a47
    -F : do NOT add a FipHdr to the output file default: do
        this can be overridden by the 'nofiphdr:no' parameter
    -h : extra FipHdr information           default: none
        This is in FipSeq and should normally be quoted
        Note this is the means that 'iptimer' sends variable information to webwire
        eg : -h"SN:hello#TC:200401031"
    -H : display the Request and Response in fancy HTML default: do not
    -I : wire id                        default: 0
        used to track which instance of a multi-webwire system a file arrived/logged
    -k : ignore the Skip list (used mainly in tuning)   default: use skip-links:
    -K : Do NOT save or process any data, just build up a skip file.
        This can be used before putting sites into production so that all old links
are ignored and only new links will be tracked.
        ie run 'webwire -1 (name) -K' once beforehand.
    -l : no logging to the FipLog except for errors default: log all
    -L : log new files and errors to the FipLog default: log all
    -m : (FipSeq) no of items               default: grab ALL items
        eg -m 3 or -m \A1
        Generally used in testing to reduce the number of files grabbed
        This is overridden by 'max-items:...' parameter
    -N : path and filename of the output file   default: fip standard
        use this to leave the file(s) in a non-std folder.
    -o : output queue in 'spool'            default: spool/2go
        This can be overwritten by the 'outque' parameter
        This is ignoring in Tuning mode.
    -O : force ALL output to this queue in 'spool'  default: spool/2go
        This overwrites the 'outque' parameter
        This is ignoring in Tuning mode.
    -s : generate statistics for bandwidth usage    default: no
        using Hour_group files
    -S : generate statistics for bandwidth usage    default: no
        using name of group_client files
    -t : track status               default: no
        this can be overwridden by the parameter
    -V : if using spool-a-folder (-i) then stop when it is empty    default: keep
    -w : Wait in seconds between accessing links.   default: 5
    -x : Proxy server host or IP address        default: none
    -X : Proxy server port              default: 80
    -y : Proxy logon                default: none
    -Y : Proxy server is Squid          default: no
    -z : parameter file in 'tables/webwire'.    default: XWEB
    -v : Print the version number and exit

---- Other Notes ----

-- Netiquette --

Pls note if you are grabbing data off another site, then you should contact the
webmaster of the remote and let them know. Certainly if you are accessing every
few seconds, then there is a good chance they will put you on some refuse list.
So it pays to be nice !

-- How to find out the actual url....

Sometimes it is quite difficult to find out the real path to use for the url.

Especially so for script-driven gets and puts.

NetScape or Iexploiter is invaluable in this case..
 - use either 'View Source' or 'History' normally gives the game away!

Snooping using tcpdump or windump
    0. Open a Terminal/Cmd window and start you browser - without hitting the site
    1. Find out which interface
        tcpdump -D
    2. Leave tcpdump running in background
        On Mac OSX you will need to be sudo
        tcpdump -i1 -w remo.tdmp -X host www.remote.host

    3. On the browser, do the absolute minimum ..
        .. do a simple logon and grab ne file using Firefox, Mozilla, IExp, Safari
    4. CntrlC to stop tcpdump
    5. run tcpdump to show data
        rcpdump -r remp.tdmp > remo.fip
    6. call up remo.fip in an editor.

-- Cookie Cookie Cookie Cookie Cookie Cookie Cookie Cookie Cookie Cookie

Cookies are neat but nasty.
If you already know the cookie you need, just make a file in /fip/fix/webwire
with the name of the cookie (case is important on Unix boxes) and slap in the
whole of that cookie which has the syntax
ie  zumzum=hungryTummy

Before grabbing data pages we can attempt to logon to a box and get its cookies
This uses from 1 to 9 GETs or POSTs

    add-cookie:\C1; \C2 Add the Cookie on to the end of the HTTP headers in this
    get-cookie-1:   Command to send to get a cookie or to logon.
    get-cookie-data-1: Optional data usually required for a POST
    get-cookie-http-1: more HTTP headers used ONLY for this GET/POST
    cookie-fiphdr-1: name of the cookie to use as a FipHdr field C1 to C9
            ie if there are several cookies returned but only one
            is needed, put the key as the cookie-fiphdr
            ie Set-Cookie: ABC=12345
                add-cookie:\C1; perm=yes
            will result in a Cookie: ABC=12345; perm=yes
            If you  want all the cookies to be saved, use '*'
    follow-cookie-redirect: (yes/no)
        ie if you get a 302 Moved Temporarily status Plus a Location from a cookies
        use that rather than the 'url:..' specified.
            HTTP/1.1 302 Moved Temporarily$
            Date: Fri, 29 Oct 2010 00:17:19 GMT$
            Cache-Control: max-age=3$
    cookie-form-1: find and save an input tag in a form and put the data in a
FipHdr field starting F*
        use this to add hidden form zones into a reply for a logon for example.
        ; csrf_token will go into F1
        ; then send it back
        get-cookie-2:POST /login/

        Normally the access to the cookies do NOT give any data you need to save in
the FipHdr for use later on
        But there are times - eg when the cookie (or api-key) is a logon code - when
you DO want to save
        However if you do not want this (maybe there is some data which clashes) turn
this OFF by specifying NO

There can be up to 9 of these.
    eg  add-cookie:\C1
        get-cookie-1:GET /
        get-cookie-2:POST /logon.pl

Rarely are any  get-cookie-http-1 fields needed as
    Host, Content-type, and Content-length are added automatically
    Referer is added if you have specified a 'referer'
        which you should if running 'http-version:1.1'
    Keep-alive is added if you secify 'keep-alive:yes'
    Others 'httphdr' fields should be specified as normal..

As a general rule, some Microsoft IIS sites (who else!) have problems if you
HTTP headers are in the wrong order. Basically, make sure your CONTENT* lines
are last.

Example 1
; ------------------------------------------------------
; we need to go and get a cookie for this service
; we will call it C1 - so the httphdr will be 'Cookie: (contents of C1)'
; C1 will hold the contents of an incoming 'WebLogicSession=.....'
; this is the URL to hit (with parameters) to trigger the Cookie
get-cookie-1:GET /servlet/com.login.DispatchServlet?Login=&User=guest&Pwd=guest

Example 2
; ----------------------------------------------
; in this case we have 3 cookies C1, C2 and a fixed one 'b'
; C1 is SID=..
; C2 is ASP...=...
; add the fixed 'b=b' on the end
add-cookie:\C1 ;\C2 ;b=b
; just one grab at a cookie - and Logon and the same time
get-cookie-1:POST /login/Login.asp
; one logon string
; ignore the 302 return - it is only trying to send us to index.asp
; Save the two cookies as C1 and C2

This will POST - ie pretend to be a filled out html FORM - the logon back.

Note that the cookie-data is 'URI escaped' ie if it is a special chr - like
/?&+ - and is in the data bit, you must use the '%xx' notation (where xx is
the HEX value). But hopefully you would have seen that in your tcpdump/snoop

-- Proxies Proxies Proxies Proxies Proxies Proxies Proxies Proxies Proxies

When running through a proxy server, you will need :
    1. hostname of the proxy server
    2. port number on the proxy server if it is NOT port 80
    3. (optionally) a logon and password
    4. Is the proxy SQUID ?
        If so headers are slightly different.

If this information is NOT available, normally you can find it easily from any
PC or Mac on the internal network using a browser like Netscape or IExplorer.

Start a NEW copy of either of these.  - It must be a new copy to check on
logons etc.

Under 'Preferences' or 'Internet Options' there should be a 'Connections'
section and under that, the host name or ip address plus host name of any proxy

Note that often the main Fip server is NOT running DNS and will not be able to
resolve external hostnames, so the IP address must be used in this case.

Enter these values in the Fip parameter file as :
    proxy-server:   (no default)
    proxy-port:412          (this defaults to port 80)

Use the Browser to attempt to access a web site outside the firewall - like

If you are asked for a password to get through, you will probably need to add a
'proxy-logon' parameter too unless the keeper of the Firewall has made a hole
through just for you.

The data for 'proxy-logon' is in base64 in the format (logon) (colon)

Use 'sffb64' to generate this string :
    On a Sparc  echo -n "chris:magicman" | sffb64 -i
    On Linux    echo "chris:magicman" | sffb64 -i
    On Winnt    type "chris:magicman" | sffb64 -i


The actual 'You need to Logon, Pal' message is a '407 Authentication Required'

-- Repeat Offenders -----------------

Some sites add a session-id into each and every link. And this Id changes on
each access.

To 'webwire' this appears to be a new file and so it is grabbed every time -

There is an 'ignore-key' command to isolate and ignore the relavany parameter.
eg Take a site like :

which returns links like

If value of PHPSESSID changes each access, they you will get a copy of newsid
7866 every time.

Use :

Do NOT specify the '=' or '?' etc.

-- Others Others Others Others Others Others Others Others Others Others Others

--Where 'webwire' is used to drill down links, there is a wait of about 5
seconds between accesses which, hopefully, is enough time for other people to
use that server.

--Where a logon and password is requested as part of the Browser - ie a pop-up
from Netscape or IExplorer, NOT an HTML form - you will need to add a
'Authorization' line. This will be true if you get a message like :
    HTTP/1.0 999 Authorization failure
        ... etc etc etc ...
    Assuming you know your logon and password :
    1. Use uuencode or sffb64 to generate a Base64 string
        echo -n "logon:passwd" | sffb64 -i
    2. Add an extra line to the parameter file with the result of the sffb64 line
using 'httphdr'.
        Syntax: Authorization (colon) (spc) Basic (spc) (Base64 of logon:password)
(\n FipSeq for NL)
        Eg  httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n

-- Valid links are :
    - The HREF tag atttibute in A for Anchor    <a href="www.fingerpost.co.uk>
    - The SRC  tag attribute in FRAME       <frame src="ax1000.html">
    - The URL in a META/Refresh         <META HTTP-EQUIV="Refresh" CONTENT="0;

-- For 'matchlinks', the term LINK is the contents of the <a href="THISONE">,
NOT the associated text
    ie matchlinks:*boonies*
        will find   <a href="/rubbo/boonies/tunies.html">This is a Wonderful Page</a>
        BUT not     <a href="/tunies.html">This is the boonies Wonderful Page</a>

-- Note that 'ignorelinks' refers to both Links and Forms.

-- If you want to ignore all links and only get forms, use a weirdo name in

-- What are reasonable HTTP headers ?
1. If you are using HTTP Version 1.1, you MUST add a line in the headers which
specifies the actual host you are trying to access (ie the REMOTE hostname or
IP address):
    httphdr:Host: www.theirsite.com\n
or if DNS is a problem
    httphdr:Host: 123.456.789.012\n

2. Most servers would like to know what you are and what you can do - so lie !
    Try this for starters :
    httphdr:Accept: \052/\052\n
    httphdr:Accept-Language: en\n
    httphdr:User-Agent: Mozilla/4.0 (compatible; MSIE 4.01)\n
Note the syntax is httphdr:(Keyword) (colon) (space) (Parameter) (NL)
    Keyword is case-INsensitive
    There MUST a Colon-Space beteween the Keyword and Parameter.
    The line MUST finish with a single NL (which webwire will handle correctly)
    as Double NLs mean end of header.

3. If the data on a lower level is being served from a different host, if you
need authentication or some other httphdr, use the 'httphdr-on-all-grabs:yes'
parameter to add them for that server too.

-- ValuesFile ValuesFile ValuesFile ValuesFile ValuesFile ValuesFile --

Take the case where you need to get the 10 foreign exchange rates every 20
minutes from a site like Yahoo.

The normal way would be to test using one forex rate and, when ready, just
duplicate that parameter file another 9 times, just changing the forex
name/search string in the 'url' or 'post'.

The classy way is to pput all the search values (ie the bits that change) into
a single 'values-file' and reference them using FipHdr fields W1 to W9.

To Do this :
If the original url is :

1. Create a values-file in /fip/tables/webwire - lets call ir VALUES_4_FOREX
    This can have the normal Fip-style comments of ';' at the start of line
    ; Values file for Forex
    ; end of values file

2. In the WebWire parameter file - lets call it FOREX.
    ;   FoREX



... and let rip.....

Note that W1 is the first field, W2 the second etc. If you are already using W1
for something else, specify another FipHdr field to start on with the
'values-fiphdr' parameter.

Note that the FipHdr fields are useable for filename and other Fippy things.

will give filenames (and/or FipHdr SN) for our example of

-- Standard-FingerPost-Rant on bad HTML ----------------------
-- Using Webwire to pull off other file formats

Sometimes, 'webwire' seems to only grab part of a page and never returns
errors. Well, if you use a browser to look at the page and then 'View Source'
or 'View Frame Source', lo and behold there is probably a random </HTML> at
that point.

</HTML> is of course the End Tag of an HTML document. So we SHOULD stop there

But a lot of web sites do not care how awful their stuff is - or maybe a
conversion program has been set up wrongly (a well-known news agency in New
York uses </html> in place of </image> to end pictures for example)

So use the keyword 'end-of-document' to track either nothing - just timeout -
or the REAL end of document.

If the data is NOT html - some XML variant for example - use 'end-of-document'
to track that.

By the way, did you know you can immunise yourself from fingerpost-rants; pls
contact the sales dept.

-- Wrinkles with Ports and RSS

Some RSS servers like to service the initial list from one port - but you have
to grab the data from another

-- using warmrestarts and cookies to keep a logon current

1. Add '-C'

if using iptimer, add as ' switch:-C '
client:abc type:w template:abc.fip days:X  every:1s fiphdr:'#' switch:-C

2. run manually to see what the http response code is for a BAD ie logon again
eg good is normally a 200 code :    HTTP/1.1 200 OK
    bad is something like
HTTP/1.1 303 See Other$
Date: Fri, 24 Jul 2015 15:53:18 GMT$
Server: Apache$
Expires: Thu, 19 Nov 1981 08:52:00 GMT$
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0$
Pragma: no-cache$
Location: /login/$

2. to the parameter file
; YES or FIPHDR= logon returns data; COOKIE=logn returns a cookie in the
; look for this in the incoming data
get-cookie-1:POST /login/

when it goes bad, you will get this message in the item log
Fri Jul 24 16:58:22 webwiressl !z : Zap cookies for Logon NZXIS.APAC

-- API keys in the data

basic-authentication:JA whomee:never

apikey-url-1:POST /oauth/token
apikey-httphdr-1:Authorization: Basic \JA\n
apikey-save-1:\VZBZ:\JZ\nXX:as at \$h:\$n:\$b\nXX:expires \J1\nXX:scope
\J2\nXX:domain \J3\n\$o

; READ last token
combie:RZ   JZ|BZ

; ; ..
; if status code =401-Unauthorized .. redo the logon ..
fiphdr:J0   tag:status

; .. and if the test is invalid, zap the FipHdr file - use this very carefully
if you are using multiple FipHdr files !
; this will zap any cookies (default) matchlogon-invalid:yes

Version Control
;6a50d  26oct18 chgs for multiple aws grabs
    ;2-3 retry-404/500 parsed
    ;4-5 internal - trimming the FipHdr
    ;6-7 location can be same url but diff host/port and log-errors
    ;8-11 bugette with if-not-modified and method parsed, maxTree 100->300
    ;12-14 9jul19 added socks proxy too
    ;15 1nov19 cater for when there are > 9 values fields plus added values-allow
and allow-comment
    ;16 11nov19 added S2 S7 etc for system
    ;17 25nov19 redid lower levels and added fiphdr-ll ;18 12feb20 bug in Values
    ;19 4mar20 minor dontuse ;20-24 minor ; 25 sizeof url and relocate tuning
    ;26 linktags 10->20
    ;27-28 20jul20 redid apikey so params are parsed at runtime NOT on param file
    ;29-30 minor json [] issuette
    ;31-32 7feb21 redid AWS STS token and retry-404
    ;33-34 25feb21 woops - KAlive but diff host issue / FHlevel on last-TOPlevel
    ;35 17mar21 better zap of expired ApiKey
    ;36-37  5may21 preserve FH in copy_tmp as the new bits are needed for
outque/script etc
    ;38 6oct21 always pull_apart_json/xml for cookies to get FipHdrs
    ;39 29oct21 added -m maxItems
    ;40 12apr22 better handling of JSON Arrays
    ;41  8sep22 added RAW type for EFE api
    ;42-44 23sep22 better VIEWSTATE ;45 tuned VALUES
    ;46abc bugette in waitTimeout and (slightly) better messages
    ;47abcd  1aug23 FORK and children and fiphdr S5: lnko idx ;b
convert-CDATA-sections fipseq
    ;48a 7sep23 added oauth for gcp
    ;49 added poll-or-select and shorten-urn
    ;50a-d tuning FORK;  localSeqno better
;5z99   10may05 hourly bandwidth stats files rather than per client
    ;a 13may05 balance skiplists if changed
    ;b-c 25jun05 added -M and -K
    ;d-g 05aug05 added fiphdr:XX data:abc\A3 and wait-end-timeout
    ;h-k 04sep05 changed -x-X to force not default
    ;l-m 07nov05 added 24hour+ skip files
    ;n-p 25sep06 added ssl at last
    ;q-t 17oct06 added skip-details-tag
    ;u 29apr07 major change to linktag, added matchkeys and match-case-sensitive
    ;v-w14 21may07 add rest of path if 3rd+ level and no starting '/' (w14 -
tweaks to stuff_cookie)
    ;x1-6  8may08 added save-fiphdrs ;3 added -N newname ;6 bugette with VALUES
file and port != 80
        ;7 added -e and -E errname ;8 balance fiphdr fields ;9 meta-files ;10-12
        ;13-14 note_balance_action ;15-16 spc in url ;17 added pretend-301:200 ;19
allow feed:
        ;20-23 finally added basic-authentication: and redid ssl
        ;24 bugette/modette - allow multiple spaces in mime headers
        ;25 allow intergap of zero
        ;26 bugette - save_metadata missing if one and only one found
        ;27-29 25jun10 bugette when proxy is a Squid and host changes
    ;y1-9 26jul10 added grab-on-tag/endtag (major release) ;10-11 6sep10 bugette
with 302-move and http://...
        ;12-14 added matchlogon, bug (bg) with data-type:CSV, plus tom bug :
retry-404-max:3 retry-404-gap:1
        ;15-17 14oct10 added skip-save-data and days:Z for weekdays
        ;18 15nov10 added follow-cookie-redirect: ; 19 able to parse VALUES-FILE: ;20
added nofiphdr
        ;21-25 mess if too many 404 plus added -D and fiphdr-hash
        ;26-27 16mar11 added repxml for fiphdr: / include fiphdr-file in start of
        ;28-29 31mar11 added zap-values-file:yes
        ;30-32 poll.every secs bugette ;32 added need-proxy-cookie
        ;33 6jul11 better skips handling now allow 15000 skips and zap olds with
different skipdetails
        ;34 29jul11 added need-logon-token and cookie-host-X for rconnect
        ;35-36 added dbl-dblqtes in links plus Bugette in Chunks and redid outque for
        ;37-41 added CONNECT for proxy https plus started minitracking and sleep
between polls for XWEB
        ;42 allow multiple spaces in custom tag link and added filter ;43
null_next_link added
        ;43-45 added retry-404-error
    ;z1-8 15mar12 added eventvalaidation and viewstate and level5* and json
        ;9-10 allow multiple grabs, added level to grab-on-tag and matchlinks etc
        ;11-12 redid 302 moved to handle full paths better ;13 ;14 bugettes -
proxy/do NOT output file for cookies
        ;15 28feb13 tuning for level1template-file:
        ;16  4apr13 bug in skips if no headline
        ;17-23 11apr13 added trees, levels and keys to fiphdr:,  grab-on*tab: and
        ;24-28 17may13 added retry-500 kwds and better proxy handling ;27 added
level1mime and -I wireId
        ;29-31 17mar14 added 404/500action=move, que and FipHdr ;31 modette-repxml
for all tags
        ;32 14apr14 for custom logging ;33 4aug14 added -Z force DF ;34 bugette with
fiphdr.. key:
        ;35-36 12nov14 added httphdr-on-all-grabs
        ;37 17dec14 bugette WINNT Only, cookie=* ;38 ;39 fiphdr-hash for W$ too ;40
        ;41 28dec15 new apache does not like 443 on the end of Host:..
        ;42 13jan16 bug with https and proxy
        ;43-45 22mar16 allow 302/301 with Values and Bug with skipDetailsTag ;46
httphdrs on proxy
        ;47 10jun16 pullapartJson better
        ;48-52 14jun16 proxy and TLS1_2 and httphdr on proxy
        ;53  7sep16 allow same tag or tag@att to be in multiple fiphdrs
        ;54-55 19sep16 added SX/save-data-pathname
        ;56-57 cleanups bugettes - SU/DU if in extra, proxy and 302 handling
        ;58-59 16oct16 for ANP_FOTO fiphdrs and keydepth
        ;60-64 30dec16 added apikey stuff ; 65-68 maxLowerLevels 5->10
        ;69-72 bugette to newname/forcenewname and cookieForm
        ;73-76 28jul17 redid hmac and added recode ;77 JSON grab on endtag if in an
        ;78 3nov17 do not attempt to drill down during cookies
        ;79 15jan18 better JSON handling
        ;80-81 18feb18 updated ssl and added mime-type-fiphdr and level-fiphdr
        ;82 redid convertCDATA slightly ;83-4 cookie-host in FipSeq so we can vary it
;85-86 issuette with data-type and cookies
        ;87 matchlogon-invalid:zap added to zap the FIPHDR file as well as any
        ;88-90 added if-mod-suffix for filename if-modified
        ;91-92 better json support
        ;93 amz hmac support
        ;94-95 12sep18 added connection-retries and \@V
        ;96-98 6oct18 better handling of truncated data (aws-data added)
;004z   07jul04 tweaks...
    ;b 01aug04 added fiphdr:....
    ;c 10aug04 added levelXtemplate: where X is 2->4
    ;d-k 01sep04 -9 speedy and timing stats (f-maxlevel and values bugette)
    ;l-n 07oct04 added skps2, fixed one-file,
        fixed HTTP results with no messages
    ;o 28oct04 redid skps2
    ;p 01dec04 buglette with spaces in URLS - need to be stripped.
        plus lvl1file-lvl5file added
    ;s 31dec04 added -x proxy-host, -X proxy-port plus -y/-Y
    ;t-u 01feb05 added bandwidth-stats
    ;v-w 19feb05 added -u testPid and -U singlelevel only and split into files
        plus bugette with Chunking
    ;x-z 18apr05 added -O for rpt-offenders/small-diffs flag
;003z   15dec00 added one output file, tracking sents, only-get-if-modified
    ;a 20dec00 added watch on XWEB
    ;b/c 22jan01 allow hrefs to be NOT in dbl quotes plus added end-of-document
    ;d/e 19mar01 started proxies
    ;f 17sep01 proxies again
    ;g 29oct01 proxies again
    ;h 13dec01 minor mods - allow http:name:port in url and proxy
    ;i 08jan02 values-fiphdr and bugette with values
    ;j 08apr02 bug with one output file - core dump
    ;k 01jan03 MACOSX
    ;l-p 21jan04 added -h and allows secs for 'every'
        and 'no-data:'
    ;q-u 08jun04 added matchlinks/ignorelinks/url and now FipSeq
    ;u 27jun04 added -H html, -k ignore skipfile
    ;w-z 30jun04 proxy-is-squid added
;002b   24oct00 added values-file
    ; 06nov00 added 'every' and Chunks

(copyright) 2024 and previous years FingerPost Ltd.