This program is used to handle repeated Requests/Responses with a remote REST
or SOAP server - or anything that handles R-R.

It allows up to 10 Requests to be made, checked and processed in succession.

These could be, for example,
    - logon stage
    - verification stage
    - get 1st data stage
    - get last data stage

Each stage is split into 4 main sections
    - Beforehand
        - Add FipHdr information from a string or a standing file (or the input file
if there is one)
            the standing FipHDr file might be something saved from a previous run, such
as a cookie
        - Test to see if the stage is required - eg dont logon if there is alredy a
logon token from a previous run

    - Build and Send the Request
        - Complete the mime header
        - fill in the Envelope and Body using an existing template

    - Analyse the Response
        - Save the response and send it on to another Fip process
        - Extract FipHdr fields from tags/tagattibutes and save for future use
        - Test response for errors - commnucation, HTTP, data
            and stop, continue or jump to another loop.

    - Log it all and cleanup

The parameter file(s) are in tables/soap. The syntax of these are :

    ; comment start with a semi colon like this

    Before section
            Add/Parse extra filehdr fields from a standing file - such as Logon
            The format should be just a plain text file with FipHdr fields and NO
start/end FipHdr markers
            This adds a single line of FipHdrs (or a single FipHdr) to the FipHdr
            This is added AFTER any fiphdr-beforefile-X, so previously added FH can be
            Usually this is the Path of the Mimehdr
            test-before-2:  fiphdr:R1=""  action:goto1

    Build and send the Request
            Name of a MimeHeader template file in tables/soap.
            if the mimehdr is always the same, use 'default-mimehdr:'
            Name of template file in tables/soap
            This is usually needed for SOAP as it will contian the main Soap Xml Request
            For REST, a blank envelope file containing NO lines should be created and
            Name (and method) of remote host for this step.
            eg  host-5:https://rmd.wiki.com
            This is ONLY needed if the host is different to the default input switch

    Analyse the Response
            queue:(output folder)
            fiphdr:(optional extraFipHdr to add )
            before:(optional data to add before)
            after:(optional data to add after)
            tag:(name of tag if NOT all the response is to be saved)
            data:xml or txt or raw or NO for none
                data:(dont send any data from the response - this assumes there is fiphdr
and/or before and/or after)

        test-after-X:   [result or fiphdr or tag] [= or != or #] (some test)
                action: [stop or continue or gotoX or error]
                    action:stop stop all processing
                    action:error    stop all processing and flag an error
                    action:continue continue with the next Loop (default action)
                    action:goto9    continue at loop 9 (or whichever number you have specified)
                log: (Fipseq - extra information to add the the log)
                script: (Script or program to run in FipSeq)
            tag:(tag name)  save:(name of fiphdr file)

            test-after-1:   result=404  action:error

; ---------------------------------------------

; -------- loop 0 ------------
; logon and get a token


fiphdr-after-0:R1   tag:Expiration          save:exp.file
fiphdr-after-0:R2   tag:global:Token        save:token.file

    port:   Port number of the Remore Server.       default: 80
        This forces the port to be this if none is specified.
    donque: done queue for the input file.
    nofiphdr: Do NOT add a Fip Hdr to the file.     default: yes pls

    httphdr: Extra lines of HTTP header you may need.   default: none
        Remember to add a NL at the end of each line.
        There can be multiple httphdr lines but pls remember to add a '\n' at the
        end of each one. (or you can try to force all on one httphdr line!)
        eg  httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n
            httphdr:User-Agent: Mozilla/4.0\n
            httphdr:Host: wibble.wobble.com\n
        see below for 'useful, common header lines'

    filename: Filename for the output file in FipSeq.   default: WEB\$Z
        If this does NOT start with a '/' it is left under the
        Output Queue as specified on startup (default spool/2go)
        eg  filename:AFP\$d.\$z

    log:    FipSeq custom logging for the item log.  default:\SN \SU \EF : \ST,\SP
        This logs each Page grabbed
        Note that
            ST  remote site host
            SP  remote site port
            SF  remote site url

    extra-pre: Extra FipHdr fields to be added to the output file.  default: none
        To separate FipHdr fields, pls use a '#'.
        extra-pre is added as soon as the file is read - so may be used for
information in the URL
        extra is only used for any output file and is not used at all for any other
        eg  extra:ZH:NYNZ#DI:Headline News#QZ:333
    connection-timeout: (secs)
    wait-end-timeout: (secs)
        For slow, busy sites, data - especially big files - may take a lot longer
than normal to be
        retreived. Use this to expand that time. Default is 120 (it should be
divisible by 5 for some arcane reason)
    no-data: (FipSeq string in place of data)
        Do not get/send the data - just this string
    proxy-server: If using a proxy, these are the name and port to aim at.
    proxy-logon: This is the logon and password to get thru the firewall
        if required. The format is (logon) (colon) (password) and is
        converted to base 64.

        To generate :
            echo -n "logon:password" | sffb64 -i
        eg  echo -n "chris:sleekpanther" | sffb64 -i
        gives   Y2hyaXM6c2xlZWtwYW50aGVy
    proxy-is-squid:yes/no   Is the proxy a Squid ?  default: no
    proxy-handshake:yes/no  Does the proxy need to say hello first ?    default: no

    logeachfile:(dest) Send a Success/failed msg to this destination
            for each file. There is no default. This log file is
            just a FipHdr with the following extra fields :
                DR-File Sent OK  DR:ok or DR:error
                DG-Will Retry later DG:retrying, DG:stopped
                DT-Some message text    DT:No connection
            default: no log created.
        The text for the DR and DG can be in FipSeq and so can contain
        FipHdr and other variables. As they are FipHdr fields, please
        do NOT put NL, CR etc in the fields.
        do NOT put NL, CR etc in the fields.
        Note that System Variable \$q holds the time taken for transmission.
    DRgood:(text)   Message for the FipHdr field DR on a   successful tx
            default: ok
    DRbad: (text)   Message for the FipHdr field DR on a unsuccessful tx
            default: error
    DGcont:(text)   Message for the FipHdr field DG if, after an
            unsuccessful tz, another attempt will be made.
            default: retrying
    DGstop:(text)   Message for the FipHdr field DG if no further
            attempts will be made as the file was sent successfully
            or the maximum no of attempts has been tried.
            default: stopped
    fiphdr-for-logeachfile: (FipSeq) or
    msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
            'logeachfile' or 'loglasterrfile' msg. This should be in FipHdr
            format and be in FipSeq. It can be used to pass FipHdr fields
            in the outgoing file into the log file.
            eg  msgeachfile:    DF:logdial\nSS:\SS\n            default: nothing added

    check-primary-server: pseudo-host name that is specified in
        Use this when an ipspool on 2 systems is accessing the same remote folder -
in a redundant way.
        that is used whether the current host should be getting the files or not.           
ie in the parameter file is
        and in the DEST_REDUN is
            ; psuedohost    primary secondary
            timersvr    fip1    fip2
        and in the SYSTEM file for both fip1 AND fip2 there is  the same 'ipsoap'
line with the same parameter file
        If fip1 is up,  'ipsoap' on fip1 will always get while on fip2 it will just
        If fip1 is down 'ipsoap' on fip2 will start processing the files.

    max-loop-retry  default 20

    ssl-method: (1,2,3,23,999)
        Version number to use for TLS/SSL       default: 999 for current default (2 or 3)
    ssl-password: (password)
    ssl-passwd: (password)                default: none
        Optional password if the handshake requires a shared secret
    ssl-cert: (name of a PEM certificate file)      default: none
    ssl-root-cert: (name of a root PEM certificate file)    defaunt: none
        Optional certificates - held in tables/ssl
    ssl-verify: yes/no  verify certificates     default: yes
    ssl-ciphers: (list) acceptable ciphers
        (use 'openssl ciphers' to list)
        default:  "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS"

Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).

    They are normally specified :
        fixed:QZ    1234543
        partial:QT  ST,3,2,U,<,>
        combie:QY   ep|na,(0000000)a
        option:QE   ep,11,7,s
        repeat:QK   XK,-,3
    or  repeat:QP   PK,,4,#X
        style:QS    XN,%.03d

Input Parameters are (Mandatory) :
    -s : hostname of the remote server      default: none
    -z : default parameter file in tables/soap  default: SOAP
    -1 : path/filename for single shot      default: none
        The input file is NOT deleted
        If this does NOT start with a '/', it is assumed relative to the current
    -i or -q : input queue              default: none
        If this does NOT start with a '/', it is assumed under spool.
    -d : display the transactions           default: no
    -D : display the transactions           default: no
        break out the xml of the response
    -F : use FipHdr field DF for parameter file default: always use -z parm file
    -G : no of secs between reruns          default: run once
    -h : extraFipHdr information            default: none
        to be used during the requests and for any output files
        this is separate and additional to the 'extra-fiphdr' keyword in the
parameter file
    -H : alias hostname (or cluster hostname)   default: none
        use this for checking primary/secondary where the actual hostname might be
different from the cluster
    -l : log level                  default: log errors
    -o : default output queue for any grabbed files     default: spool/2go
        If this does NOT start with a '/', it is assumed under spool.
        This is overridden by any path/folder in the save-response parameter
    -p : port number of the remote soap server  default: 80
    -u : dump the transactions in /fip/dump     default: no
    -U : Break the Dump file (-d) into Hourly files.    default: no - daily
    -w : file wait for files arriving across a network. default: none
    -v : print version number and exit


Version Control
;0f19   20nov01 original version
    ;f10-12 better ssl
    ;f13-14 20mar11 added host-X
    ;16  9may11 added script to test-after
    ;17-19 19apr16 added proxyHandshake etc and bio/ssl

(copyright) 2017 and previous years FingerPost Ltd.