ipbdcast

ipbdcast

    Variable-headered broadcast text-out program

IPBDCAST is used to send text down a leased line. It uses a Format table in
tables/bdcast to define the message format for that file.

In order to present a standard feed to clients, different types of messages
from different sources use different format files which reflect the different
information held in the header fields.

Features include :
    - Check message generation or suppression.
    - Conversion to non-ascii character sets on the fly.
    - Creation of log copies of transmitted files
    - Exact dumps of transmitted data.
    - Writer flag for Writer PCs to review traffic.
    - High Priority BreakIn and Interrupt.
    - Ignore messages which are older than a 'sell-by' date/time.
    - Send messages either in Unix queue order or priority/time.
    - Wordwrap at defined line lengths so that the Editorial system does not need
to HNJ the messages before transmission.
    - Split into takes where the Receiver cannot accept large files
        or split on a string in the data
    - Send/Post to an HTTP server like Apache, NGinx, even old ones like Domino,
NetScape/iPlanet or IIS
    - Failover to a second port/server
    OR Failover to another technique such as dialout or ftp

IPBDCAST reads the file to send, searches for the DB FIP header field which
says what format file to use (if no DB, the default file is used). It then
checks  whether the format file is already in core - normally 10 tables are
held. If no chk_msg, queue, format and archive are specified then they are set
to the same as 'name'.

It can also be used to send SOAP or other http POST - see later for notes on
HTTP POST/SOAP.

Essentially the parameter file is used to describe the strings to be output for
the :
    - PRE HEADER    ie optional strings to start Satellite selectors etc
    - HEADER    ie strings to be sent BEFORE any DATA
    - TRAILER   ie strings to be sent AFTER all the DATA

Example of a parameter file
    ; setup keywords- ascii like chrset, max 80 columns and insert CR NL
    chrset:ascii
    ww:80
    eol:\r\n
    delay:\177

    ; send the preheader
    pre:\026\026--ab\r\177
    ; send the header
    hdr:\001\XS\XN\r\r\002
    ; the text is sent now
    ; send the trailer
    tlr:\003\$d-\$m-\$y\004

The syntax of the Format file is :
    ; comment line
    (keyword:) (subparameters) (eoln)

In more detail :

    pre:(FipSeq)    preheader string to be output
    hdr:(FipSeq)    header string to be output BEFORE any data
    tlr:(FipSeq)    trailer string to be output AFTER any data

These can be preset files of  FipSeq which are specified as
    http-header-file: (Filename)    (same as http-header-file-0)
    http-header-file-X: (Filename)  (where X is 0-9)
        Note there MUST be an http-header-file: if there are any http-header-file-X
        Note that Takesize chopping is disabled for multiple http headers
        Note that FipHdr fields are filled in
            D0 (dee zero) holds the loop/HTTP hdr number - D0:999 is for auth, D0:0 for
the first loop .. D0:9 for the last
        D0 may be changed to another FipHdr field using 'newD0:'
            eg  newD0:P0
    pre-file: (Filename)
    hdr-file: (Filename)
    tlr-file: (Filename)

The Http Header is used only for transferring data to a RestFull API or HTTP/S
server like Apache, Domino or IIS, It is output BEFORE any of the other
files/templates. The only diference between this and any other is that each
line is ended with one and only one CR NL and the header is terminated by a
blank line even if you do not specify one.

The PRE, HDR and TLR keywords plus default fields can contain any FipSeq
including :
    - printable ascii text (not NL, CR or FF as they end the string)
    - unix escape chrs : \r, \n, \s, \\, \t etc
        such as \r is a carriage return
            \n is a line feed or new line
    - numbers : \001, \377
    - FIP header fields : \SN, \DQ
        See the Chapter on FipHdr fields for more information
        Note that Source Header fields (SH) can be accessed by \X? ie \XP for
priority, \XC for category.
    - system variables : all the normal FipSeq ones plus
        Field   Remarks      Example
        \$A J11 style NAME;dd/mm    JAMES;03/12
        \$S 3 figure sequence number 123
        \$Z 4 figure sequence number 0037
        \$Y year  (99)      97
        \$E century  (19 or 20) 19
        \$I month (mm)      11
        \$M month (Mmm)     Mar
        \$D day   (99)      23
        \$H hour  (99)      10
        \$N min   (99)      54
        \$B sec   (99)      45
        \$J julian day of the year   111
        \$R random alpha numeric character between a-z, 0-9
        \$X strip trailing spaces from this point to the last non-space
        \$O if Optional field failed, strip to this point
    Plus
        \$C Total No of Chrs
        \$W Total No of Words
        \$T 1st 64 chrs of text
        \$U 1st line of text - up to 64 chrs
        \$1 1st line of text
            ..
        \$9 9th line of text

Note the normal FipSeq fields are available (see the System Admin Guide)
    fixed:QT    2233344
    partial:QU  ST,3,2,u
    combie:QX   ep|na,(000000)a
    option:QO   ep,11,7
    repeat:Q5   XK,+,1
    style:Q6    HJ,%.03d
    replace:Q7  XC  abc=def spo=dog
    unique:Q8   XC,=
The old style notation for these fields are also allowed :
        Fixed text fields   ie %QT:2233344
        Partial hdr fields  ie %QT\ST,3,2,u
        Combination fields  ie %QT|ep|na,(000000)a
        Optional fields     ie %QT?ep,11,7
    or  Repeating fields    ie %Q5*xk,+
    Note that the punctuation following the header field defines the type of
setup.
        eg For partial fields
        %QT\ST,3,2,U

Other (optional) keywords are :

    chrset: ascii, baudot or none       default: none
    eol:    string for end of line      default: none
    spceol: string for end of line on non CRLF default:none
    ww:
or  wordwrap:   max line length or 0 if none    default: 0
    The above 4 keywords are used to automatically wordwrap text so it
    does not need to be hnj-ed by the editorial system. In all 3/4 cases
    the parameter can be in FipSeq.
        eol:\f
        spceol:\r\n
        wordwrap:\QW
        combie:AW,80
        ; a para is FormFeed, an inserted wordwrapped line is CRNL and
        ; the actual line length is whatever is in the AW FipHdr field or 80
    max-buffer-size: (700 - 128000)     default: 128000
        Use this setting to minimise pkt fragmentation and/or pace a link
        Old versions of ipbdcast sent in chunks of 1450.
    speed: (bits per second)        default: 0 for no throttle
        Throttle a serial line
    throttle-gap: (FipSeq to a number 0 or positive number)
        pause in milleseconds (6 digits) between files for throttling

    dump-data: (FipSeq) yes/no
        write Data sent (and recevied for REST/API calls) to a file in /fip/dump
        overrides the -d input switch   default: no, or yes if '-d'
    dump-suffix: (FipSeq)
        (optional) extra suffix for the file name in /fip/dump  default: same as '-n'
input switch
    log-line: Replace line for the item log if sent successfully in FipSeq
        eg  log-line:\DB \SU \SN \ER
    log: destination of a log copy of file as it was sent. default: none
        ie log:sgwcpy
        where sgwcpy is a valid destination (DU) in tables sys/USERS.
    verbose-logging: yes/no                 default: no
        add more detailed logging in the item log ALL
    log-status: (FipSeq)
    extra-fiphdr: (FipSeq)
        This is for extra FipHdr information to add for all files. (default:  none)
    fiphdr-for-log:(FipSeq) Additional information to add to the FipHdr of the
'log' file. This should be in FipHdr format and be in
        FipSeq. It can be used to pass FipHdr fields in the outgoing file into the
log file.
            eg  fiphdr-for-log: DF:status\nSS:\SS\n
            default: nothing added
    size:   min, max or fixed length of hdr fields. default: size of data
        ie size: ep 1,5 , xk 24
        ep field is min 1, max 5; xk is fixed at 24
        (Note that no FIP header field starts with a number).
    delay:  The delay-for-one-second character      default: none
        If specified, when the chr is encountered in the pre:, hdr:,
        or tlr: the program waits a second before continuing
        This an be a printable character (like punctuation), Unix escape
        or an octal number AS LONG AS it is not needed for any field.
    inter-file-delay: Delay in milliseconds between each file - default is 0 ms
    pkt-delay: Delay in milliseconds after each 1000 chrs sent - default is 0 ms
    wrdlen: Length of an average word for the FipSeq variable \$W
        This defaults to 6. To estimate the number of lines in the
        file, this can be set to say 48 or 49 if the line length is 64.
        This MUST be greater than 4.
    stopww: A single chr used to flag that the following text should NOT be
        word wrapped. For example a table embedded in text.
    startww: A single chr used to flag that the following text should be
        word wrapped.
    wrap-header: wrap the header too
        if the parameter is '0' then the same as text
        if a positive value, wrap at this.
    take-splitter-string:  Split the outgoing file into takes at this point in the
text
        default is Do NOT split.
    split-delay: Delay in milliseconds between each take of a split - default is 0
ms
    max-take-size:  Split the outgoing file into takes with this maximum.
        default is Do NOT split.
    take-start: Text (in FipSeq) to add at the begining of a take
        but NOT the first file.  default: none
    take-end:   Text (in FipSeq) to add at the end of a take but NOT the end of
file.   default: none
        Note that FipHdr fields are filled in
            D1 holds the take number.
            D2 holds the last take number
            D3 holds the next take number
            D4 holds 'YES' to mean this is part of a multi-take
                or NO if the filesize is smaller so no splitting is needed.
            D9 holds 'YES' for the first take overall.
        D1 may be changed to another FipHdr field using 'newD1:'
            eg  newD1:PD
        This also changes D2, D3 and D4 to follow sequentially so in
            this case these would be PE,PF and PG
    extra-fiphdr-for-takes: (FipSeq)
or  take-extra-fiphdr: (FipSeq)
        This is for extra FipHdr information in addition to the D1-D4 above.
    major-take-start: (FipSeq)
    major-take-end; (FipSeq)
    major-take-every: (number)
        Use these for a second level of takes where every X takes results in a
slightly different start and end.
            D5 holds the major take number.
            D6 holds the last major take number
            D7 holds the next take major number
            D8 holds 'YES' to mean this is a major part of a multi-take
                or NO if it is one of the intermediate ones

    send-no-data: send NO data from the file at all.
        Probably this would be used ONLY to send a header/trailer.

    newDdollar:FipHdr field for actual loop     default: D$
        (not often needed/changed unless you are using ipsplit to split files)
    newD0:FipHdr field for the HTTP loop number default: D0
    newD1:FipHdr field for the Take Number      default: D1
        D1 may be changed to another FipHdr field using 'newD1:'
            eg  newD1:D6
    fiphdr-for-hostname: (2 letters)
    fiphdr-for-ip-address: (2 letters)
    fiphdr-for-remote-host: (2 letters)
    fiphdr-for-remote-port: (2 letters)
    fiphdr-for-service: (2 letters)
        Two character FipHdr field that will contain the 'x' so it can be used in the
Hdr/Tlr of the message or the log.
            eg  fiphdr-for-hostname:JH and the hostname is 'wobbly.wibble.com'
            will give   JH:wobbly.wibble.com
        Note the service will be the same as the -n input switch
    fiphdr-for-uuid: (2 letters)
        Two character FipHdr field that will contain Fip generated UUID (RFC4412
style);
        There can be multiple fipHdrs specified separated by commas/spaces
        each one will be given a difference UUID - probably for a different use - for
each file.
        eg fiphdr-for-uuid:F1,F2,F3

    wait-for-http-ack:
        When sending to an HTTP server like Apache, use this to make sure the file is
Posted correctly.
    http-ack-fiphdr: (2letter)
        FipHdr to save the http response - eg 201 File Created
        default : none
    ack-wait-timeout: (no of seconds)
        Number of seconds to wait for an ACK from the remote server
        default is 20 seconds.
    http-500-folder: (Path in FipSeq)
        if the remote server returns a 50x message (indicating they have something
wrong internally), move files here.
        This might have a delay using ipdelay before putting back for retry - or may
be a separate method like ftp.
        This is often used when ipbdcast gets a 503-SlowDown message from Amazon S3.
THe folder should be one polled by ipdelay to wait 20 seconds and them move the
file back into the ipbdcast queue to retry
    max-500-retries: (number in FipSeq)
        Number of times to retry on 500 before sent to the error folder or
spool/woops
        Note the FipHdr field DO (dee oh NOT zero) is used to carry the number of
retry attempt. (D0 zero is the loops)
        default is 10
    http-500-delay: ( number of MILLEsecs in FipSeq)
        Throttle/wait/delay AFTER a 500 error
        NOTE this is microsecs (1 sec = 1000000 msecs) / max is 1000000
        Requires an 'http-500-folder' to be set
    http-error-folder: (Path in FipSeq)
        When sending to an HTTP server like Apache, this folder will hold any files
which have NOT been posted correctly.
            (ie response code from 300->599)
        Otherwise the files are retried - possibly forever !
        Files are ONLY put in this folder if the return code is a valid one (and if
http-500-folder has NOT been specified for 50x errors)
        NOTE files which timeout - ie NO HTTP response code as the remote is down -
are retried.
    http-keep-alive: yes/no
        Keep the HTTP connection alive or not.  default for http is YES
        so the connection is dropped only when there are no more files to send.
    save-ack-response: (FipDU)
        Where the HTTP server may be sending a long xml response which will need to
be converted/processed/displayed somewhere.

    locale: use a different 'locale' ( look at the man pages for locale)
        Most computers are set to US English and never changed.
        Use this parameter to customise any date/time or Chr translations
        The parameter MUST be a valid locale on your system!
        To find out what valid locale exist, type 'locale -a' on unix.
        Eg  ; Set for for Brazil, portugese
            locale:pt_BR
    balance-seqno:  Send the Sequence number to this Balance Group (see
'ipbalance')
    balance-delete: Delete the Data on remote systems once sent
            using this Balance Group (see 'ipbalance')
    no-archive: Do NOT archive the incoming files in /fip/log/data
            default: archive
    connect-timeout: Max time in seconds for the connection
            The call is aborted if it takes longer than this.
            If the remote is across the computer room, this time can be
            reduced to 5 or 10 secs; if over the internet, 120 secs is the norm.
            default: 30 for 30 seconds
    connect-wait: Wait in secs after the connecting before sending any data.
default: none
    disconnect-wait: Wait in secs before dropping the connection    default: 3 secs
            Use this when the '-D' flag is set on those platforms like RS6000
            which have nasty socket-closes. For HTTP connections this is set to 0.
    timeout:    Wait in secs for each packet before it is assumed lost  default: 30
    logeachfile:(dest) Send a Success/failed msg to this destination for each
file.
            There is no default.
            This log file is just a FipHdr with the following extra fields :
                DR-File Sent OK  DR:ok or DR:error
                DG-Will Retry later DG:retrying or DG:stopped
                DT-Some message text    DT:No connection
                DO-Number of last attempt   DO:5
                HT-Date and Time    HT:25 Nov 98 15:35:25
            default: no log created.
            The DR and DG messages can be changed by using :
                log-dr-ok:(FipSeq)
                log-dr-error:(FipSeq)
                log-dg-stop:(FipSeq)
                log-dg-retry:(FipSeq)
            eg: log-dr-ok:File \VN sent to Remote Host (\$h:\$n)
    msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
'logeachfile'
            msg. This should be in FipHdr format and be in FipSeq. It can be
            used to pass FipHdr fields in the outgoing file into the log file.
            eg  msgeachfile:    DF:logdial\nSS:\SS\n
            default: nothing added
    remote-host:
    remote-port:
            Hostname and port of the MAIN remote connection in FipSeq
            This option is only applicable for connections to a remote socket and not
for x25, passive port, tty etc.
            This overrides the -s and -p input parameters.
            As these are in FipSeq, they can use variables from the data file being sent
            eg. remote-host:\JQ.\JS.\JR.amazonaws.com
                premote-port:443
    failover-host:
    failover-other-host:
    failover-port:
            Hostname and port of a second remote connection to be used if the main link
is down.
            The default port is the same as the existing port.
            This option is only applicable for connections to a remote socket and not
for x25, passive port, tty etc.
            'failover-host' is a single entry and will always be used.
            'failover-other-host' is where there are two redundant systems and
            we want the OTHER system that is NOT specified as the '-s'
            input parameter to be the failover.
                failover-other-host:fip1
                failover-other-host:fip2
            If NOT using failover-other-host, there can be up to 4 hosts and ports -
failover-host3 etc.
    failover-check-time: (seconds) Time in seconds before trying the primary
            if we have failed over to the secondary.
            default is  failover-check-time:300 for after 5 minutes
    create-ZQ:
        Use this to create a fake output sequence number in the FipHdr
        field ZQ. If there is a number to this parameter it will be used
        as the amount to increment - default is 2.
        eg default create ZQ:3-4, the next ZQ:5-6
        if create-ZQ:12, file will have ZQ:22-33 then ZQ:34-45 etc.
        if create-ZQ:1,  file will have ZQ:2-2 then ZQ:3-3 etc.
    grab-first-file: Always process the first file in the queue first.
        This is for WinNT/2000 which sorts alphabetically/timewise.
    hdr-hash: (FipSeq for a single chr) This chr is replaced with a '#'
        eg hdr-hash:\200
    data-must-end-with: (FipSeq string)
        If the data part of a take/file does NOT end with this, it is added.
    data-is-binary: (FipSeq resolve to N or Y)
        Set the default type to binary or no (overrides -B and is overridden by
FipHdr FBIN:n/y)
    max-dollar-t: size of data for \$t  default is 68 chrs
        This can be up to 2048 chrs
    ignore-chkmsg-log: yes/no
        If YES, do not log Check Messages       default: log
    bw-binary-hdr: (yes/no)
        Send 4 digit filesize before hdr        default: no

    use-sx:
or  use-external-file:
        if there is an SX FipHdr field with a path to the data file, use that rather
than the data in the input file.

    send-multiple-file:yes
        Sending multiple files that are related (eg XML companion file to a JPEG)
        These files are located in a separate folder and MUST exist before sending
        NOTE - the best way is to zip all the files up and send the single zip !
Otherwise you can get all sorts of problems when the transmission breaks (which
it will do for big files on occasion) half way thru sending.
    send-multiple-que: (name of folder)
        specify a folder where the files are - this is a must
    send-multiple-key: (FipSeq)
        As the folder MAY have lots of files, specify a KEY or STUBname which can be
used to find all related files
        (ie if you do an ls for this key in this folder you ONLY get the files you
want)
        Often the key is a part of the incoming filename, or the contents of a FipHdr
field
    You must also specify ONE of these
        send-multiple-all:yes
            send everything matching the key
        send-multiple-ext:(fileextension in fipseq)
            only send files with this extension - this can be a FipHdr or FipSeq field
    And the remaining parameters are all optional :
        send-multiple-fiphdr: yes/no
            depending on whether you need to sent the FipHdr too
        send-multiple-zap: yes/no
            zap after successful send ?
        send-multiple-remote-name:\$e\$y\$i\$d_\$h\$n\$b_\E1.\$z
            name for remote if different - might want to add a unique string - date/time
for example

    use-ssl: yes/no
    ssl-method: (1,2,3,23,999)
        Version number to use for TLS/SSL       default: 999 for current default (2 or 3)
    ssl-password: (password)
    ssl-passwd: (password)                default: none
        Optional password if the handshake requires a shared secret
    ssl-pem: (name of a PEM certificate file)       default: none
    ssl-cert: (name of a PEM certificate file)      default: none
    ssl-root-cert: (name of a root PEM certificate file)    defaunt: none
        Optional certificates - held in tables/ssl
    ssl-verify: yes/no  verify certificates     default: yes
    ssl-ciphers: (list) acceptable ciphers
        (use 'openssl ciphers' to list)
        default:  "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS" (from 59m35)
    ssl-display: no/yes/(999)           default:no
        Display SSL items - checking certs, method used etc
        If a number is specified, display up to this no of chrs for each read or
write

For HTTP proxies :
    proxy-connect: (FipSeq)
        Proxy Connect (or Wakeup) string for network device default: none
        This wakes up the pad on startup or Connects via a proxy
    proxy-host: (hostname)
    proxy-port: (port)
        These are ONLY used for the TLS hostname checking as the name may be
different to the connection IPaddress

For Socks 4/5 proxies :
    socks-host: (hostname of the socks proxy)   no default
    socks-port: (port number of the socks proxy)    default: 1080
    socks-user: (user name for the socks proxy) no default
        if nothing specified, assumed that there is none
    socks-pwd: (password for the socks proxy)   no default

-------- SOAP and HTTP/POST notes ---------

In this case the data MUST be 100% unchanged by ipbdcast - any header and
trailers should have already been applied by a previous program sucg as ipxchg
or ipedsys.

And there MUST be no parameter which change the data - so no ww:, wws: etc

And it is rare for check_messages to be needed.

++ Entry in tables/sys/SYSTEM would be something like
    ipbdcastssl -s demo.stonewing.net -p 443 -q 2ww -D -m -n stonewing -d

++ In the Parameter file, some of the following parameters are recommended -

For tuning/debugging, use save-ack-reponse to check what is being sent back
; Save what is sent and is replied and send it to DU:saveack
save-ack-response:saveack

; Wait for an HTTP acknowledgement
wait-for-http-ack:

; set to 60 secs / default is 20
ack-wait-timeout:60

; Do not keep the HTTP connection alive
http-keep-alive:no

; Send a MIME header at the top of each file
http-header-file:/fip/tables/bdcast/POST2FLY.MIMEHDR

; HTTP error directory
http-error-folder:woops

A simple MIMEHDR file (POST2FLY.MIMEHDR in the above setting) for an HTTP Push
could look like :

Host: financial_deformer.bampot.uk\r
Content-type: text/xml\r
Content-length: \$c\r
Accept: * / *\r
Accept-Language: en\r
User-Agent: FippoWizzo 1.1\r
\r

- note the \r at the end of each line and the extra blank line at the end
- note \$c for the Content-length which will take the exact number of bytes of
the data part of the file to send
- replace the Host name with the name of the remote host (and port numberi if
it is not 443 or 80)
- replace the Content-type with the proper mime type if not xml text/plain,
text/html application/msword
- replace the path after the POST with the proper path
    POST /ad_hoc/xxx_push HTTP/1.0\r

- Note a Minimal Mimehdr for most remote servers, require Content-type, Host
and Content-length. All others are optional.

For SOAP you can logon, download a response (in XML), extract information from
the response and use that ads more FipHdr fields for filling templates
outbound.

-- Expect: 100-continue

Sometimes the remote HTTP server required a pre-send check where the Mimehdr is
first checked and validated before you can upload data - perhaps checking for a
small range of valid content-type or max size of upload in content-size.
The mimhdr 'Expect: 100-continue' triggers the remote to check and
confirm/reject.
The remote server then responses with a '100' if ok and ipbdcast will then
upload the data with the same mimehdr WITHOUT the Expect ..
(Or the servers replies with a 499/599 if it is NOT valid and the file upload
is aborted

To set this up, in mimehdr file (NOT on the first OR last lines !) add :
    Expect: 100-continue

This will kick out the 'Expect ...', wait for the 100 response, and then resend
the mimehdr WITHOUT the Expect when uploading data

You may also need to set 'keep-alive:yes'.

-------- BAUDOT notes --------

If chrset:BAUDOT, all strings including octal or unix is translated as per
standard tables. NOTE that LET and NUM can be sent by sending { (octal 123 -
LET) or } (octal 125 - NUM). Nuls are normally stripped unless expressly sent
with '~' (octal 176). Chrs NOT In the Baudot chrset are sent as NULs.

-------- Oauth2, Oauth digest and AWS notes --------

-- For accessing Oauth2 protected assets - eg GCP Cloud Storage or G-Drive or
Microsoft Azure features

    ; OAUTH2 authentication as per Google GCP or Microsoft Azure

    use-oauth:yes/no
        Use OAUTH to grab/use an access-token or Bearer token eg for Gmail access
        default is NO
    ; We need an access token
    use-oauth:yes

    ; which flavour of Oauth2 ? - only the first letter is meaningful
    ; oauth-flavour: Google (Gmail) or Microsoft (Office365)
    oauth-flavour:microsoft for office 365

    ; Current token file will be saved in /fip/fix/goauth2
    oauth-token-file:\OT

    ; Credentials file in /fip/tables/cert
    oauth-credentials-file:\OC

    ; sffoauth and imapwire
    oauth-scope:https://outlook.office365.com/.default

    ; Script to run when token expires - approximately every 12 hours
    oauth-refresh-script: (Script in FipSeq)        script to generate the access_token
using a refresh_token
    oauth-refresh-script:/fip/bin/sffoauth -z wire/IMAP.O365.OAUTH.SEA -c \OC -t
\OT -H '#WN:\WN' -a

    These 5 FipHdrs are use to generate, check, add/renew permissions to access
the remote data - normally Gmail or Office365

    oauth-client-fiphdr: (FipHdr)   default: IC
    oauth-secret-fiphdr: (FipHdr)   default: IS
    oauth-access-fiphdr: (FipHdr)   default: IA
    oauth-refresh-fiphdr: (FipHdr)  default: IR
    oauth-expiry-fiphdr: (FipHdr)   default: IX

-- For accessing other Oauth protected assets - like twitter
    OAUTH digest - such as twitter and dropbox use

There are three parameters to define to
    ; this is the salt key
    oauth-signature-key:\JC&\JA
    ; string to use
    oauth-signature-data:\R5
    ; fiphdr to add to
    oauth-signature-fiphdr:RS
    ; type sha1
    oauth-signature-type:sha1

    oauth-signature-key: (FipSeq)   Key for Oauth; normally the ConsumerKey and
the AccessToken
    oauth-signature-data: (FipSeq)  Data string to encode - see your remote api
doc for what needs to be included and how it sho
uld be formatted
    oauth-signature-fiphdr: (2 letter FipHdr field) FipHdr which will hold the
signature
    oauth-signature-type: (type)    Signature type
        valid types are md5, sha1, sha224 sha256 sha384 and sha512

-- For Twitter Posts ..
 .. for media uploads ..
    twitter-media-seqno-fiphdr:JG
    twitter-media-id-fiphdr:JI

    twitter-media-test:media
    twitter-media-test-fiphdr:JK

    max-chunk-size:5 MB (5 * 1024 * 1024)
    max-chunk-max: (video = 150 MB; pix 5MB)

-- For AWS grabs, use the same pararmeters ars oauth (almost!)
    aws-signature-key: (FipSeq) Key for AWS; use | as a sep :
secretkey|dateStamp|regionName|serviceName
    aws-signature-fiphdr: (2 letter FipHdr field) FipHdr which will hold the
signature
    aws-signature-type: (type)  Signature type
        valid types are md5, sha1, sha224 sha256 sha384 and sha512
    aws-request: (FipSeq)   FipSeq string to hash using the signature key
            eg
aws-request:GET\n/349445556777/fiptest1\nAction=ReceiveMessage&AttributeName=All&MaxNumberOfMessages=10&MessageAttributeName=All&Version=2012-11-05&VisibilityTimeout=1&WaitTimeSeconds=20\nhost:sqs.us-east-99.amazonaws.com\nx-amz-date:20180907T112658Z\n\nhost;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    aws-data-fiphdr: (2 letter FipHdr field)    FipHdr field which will hold the
payload sha256 hash
    aws-data-md5-fiphdr: (2 letter FipHdr field)    FipHdr field which will hold the
payload md5 hash
        This is used for the S3 Content-MD5: mime header. as an alternative to the
SHA256 hash in x-amz-content-sha256: mime header which should be set to
UNSIGNED-PAYLOAD if you use md5

AWS example
; if split into FipHdr fields for :
;   JA-accessKey, JB-secretKey, JH-sha of payload,JI-aws-id, JP-Url Params,
JQ-sqsque, JS-service, JR-region, JZ-utc datetime
combie:JA   AA,-noAccessKey
combie:JB   AB,-noSecretKey

combie:JI   AI,349445556777
combie:JM   AM,GET
combie:JP
AP,Action=ReceiveMessage&AttributeName=All&MaxNumberOfMessages=10&MessageAttributeName=All&Version=2012-11-05&VisibilityTimeout=1&WaitTimeSeconds=20
combie:JQ   AQ,fiptest
combie:JS   AS,sqs
combie:JR   AR,us-east-99
newdate:JZ  gmt unixdate=\$p "\ZZ\ZM\ZGT\ZH\ZF\ZEZ"

; For GET - choose either
fixed:JH    e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
; For POST/PUT
aws-data-fiphdr:JH

aws-signature-key:AWS4\JB|\JD|\JR|\JS|aws4_request
aws-signature-type:sha256
aws-signature-fiphdr:RX
aws-request:\JM\n/\JI/\JQ\n\JP\nhost:\JS.\JR.amazonaws.com\nx-amz-date:\JZ\n\nhost;x-amz-date\n\JH

- In this case, the default request boils down to :
GET
/349714556777/fiptest
Action=ReceiveMessage&AttributeName=All&MaxNumberOfMessages=10&MessageAttributeName=All&Version=2012-11-05&VisibilityTimeout=1&WaitTimeSeconds=20
host:sqs.us-east-99.amazonaws.com
x-amz-date:20180907T112658Z

host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

-------- Input Parameters are :

Mandatory :
    -n : name of service                    default: none
Either  -s : Terminal server (spider name) or TTY device name   default: none
and -p : port number if spider              default: none
or  -P : Passive port number                default: none

Optional :
    -1 : (filename) send a single file and stop     default: spooled
        see also -V and -G for extra display info for -1 single
    -a : archive file                   default: name
    -B : All data should be sent binary / no change     default: check for word wrap
etc
    -c : Time check message file in tables/bdcast
        This MUST be in the correct chrset for the service.
        Note that the format should cater for time/date in the trailer.
        This is NOT required if '-m' no check messages is set.
    -d : take exact copy of outgoing text in ~/dump/SERDES  default: no
        used for debugging and testing.
    -D : disconnect from the port after each transmission   default: no
        This will connect to the remote host/port ONLY when a file is ready for
sending and will disconnect after all files in the input folder have been sent.
        This allows more than one bdcast (on one or more systems)
        to send to the same port.
    -E : If the program cannot send the file in this number of seconds
        Move the file to spool/2dial for 'ipdial' to attempt.   def: no
    -f : default parameter (or format) file         default: same as '-n'
        in tables/bdcast
    -g : disable any SSL parameters  default: use-ssl if present
    -G : if using -1 (single), display any SSL debugging infomation. default: none
    -h : set High priority                  default: 1
        Any file with a higher priority is set to the default (-R)
        Used for the Priority Send option only.
    -H : Fip Hdr field for Priority             default: PR
        Only the first numeric part of the field is used.
        ie in PR:abc146-153, only 146 is significant
        Used for the Priority Send option only.
    -i : interval in secs between chk_msgs      default: 600 secs/10 min
    -I : force chk_msgs out                 default: no
        Use this to always transmit chk msgs every 10 mins or so.
        This will automatically adjust to though it is starting on the hour
        ie '-I -i 600' for 10 mins, chk msgs will be sent at
            0, 10, 20, 30, 40, 50 mins past the hour.
        An extra chkmsg will also be sent out immediately the program is restarted.
    -k : Delay all files by this number of seconds before sending   def: no delay
    -l : do NOT log items                   default: log
    -m : no chk_msgs pls                default: chk msgs pls
    -M : extra dump id number               default: none
         when using multiple bdcasts
    -o : send files in priority/time order  default: send in unix order
        Files are normally sent in the order they are in the unix queue.
        This switch is for low speed lines where we want to send in priority/time
order.
    -O : done queue in spool for raw files          default: none
    -q : queue to scan in spool             default: same as '-n'
    -Q : keep quiet and do NOT message if the terminal  default: log
        server port is busy
    -r : set Lowest priority.               default: 9
        If the Priority is less than this, the default is used.
        Used for the Priority Send option only.
    -R : set the default priority for files with no     default: 3
        priority or priority out of range.
        Used for the Priority Send option only.
    -S : Pause before continuing to scroll.         default: no
        Time in seconds to wait before checking if any other
        file with higher priority is waiting. eg -S 5 for 5 secs
        Used for the Priority Send option only.
    -t : scan interval of the queue             default: 1 sec
    -T : make a log trace of each transaction       default: do not
        This creates a one line log of each file sent is stored in
        log/remote_trace with a name of 'date_service'.
    -u : default delay in milliseconds between takes.   default: 0 for no delay
        This can be overridden by the 'split-delay' parameter in the param file.
    -U : Break the Dump file (-d) into Hourly files.    default: no - daily
    -w : this is a writer double queue where a second queue
        (spool/pc/'name') is used for access by Writer  default: no
    -x : client specific for SGW : at end of each file/group of files
         send 1024 spaces to flush their buffer     default: no
    -X : If remote device is a Host or Terminal Server, default: 60 secs
        and it goes offline, attempt reconnect and message every -X seconds.
    -W : Wrap the data in Format header and trailer at this
        line length - same as 'wrap-header'     default: no
    -V : display progress and info (for -1 single shot only) dafault: no
    -z : allow high-priority breakin            default: no
    -4 : make \$W Word Count 4 digits with leading zeros    default: no
        no longer used - pls use 'style'
    -5 : make \$W Word Count 5 digits with leading zeros    default: no
        no longer used - pls use 'style'

    -v : print version no and exit

This program uses 2 environment variables :
    FIP_USA_DATE y/n for dates
    FIP_BDCAST_WRITER for Writer style Pounds/Hashs

Note that in this version 'chrset' does not refer to a file in tables/chrset
but to the basic Ascii or Baudot.

Version Control
; 59m76 07oct04 minors..
    ;b-d 21jan05 added -M, bugette with speedy
    ;e-h 28jul06 added remote_trace style stats to send msg as fiphdr H4
    ;i-l 05apr07 better soap control
    ;m1-3 18jan08 added soap-meta-script ;2 bugette when connection dropped during
drain
    ;m4 added interDelay/pktDelay
    ;m5-6 added use-ssl and connect-wait
    ;m7-14 note_balance_action added and -B default is binary and wireId for
seqno; pktdelay
    ;m15 added use-sx
    ;m16-22 minor mod to failover - altport defaults to the main port if NOT
specified
         added alt234 and ignore-chkmsg-log ;22 bugette with long filenames
SN->STDBUF
    ;m23 26feb13 added bw-binary-hdr:yes ;24 wireId in timings ;25-26 added
httpAckFH and content-length tracking
    ;m27 30apr14 added file_trace
    ;m28-29 28may14 bug-if speedy, no chkmsg! ;30 logging ;31 12jun16 added
proxy-connect
    ;m32-34  6sep16 allow buflen -> BIGBUF -> willy-200 added max-buffer-size
    ;35 14aug17 added ssl-pem, ssl-ciphers, ssl-verify
    ;36 25feb18 added ssl-privateKey
    ;37 thru_socks_proxy
    ;38-39 9oct18 aws support plus -1 single plus -F extra fiphdr
    ;40-41 11feb19 speedy on log output ;42 fipSSLremhost missing ;43-44 make
spider parseable
    ;45-48 1apr19 added pause after expect: 100-continue better and -G SSL
display/ssl-display:
    ;49-50 http-error-queue was moving EVERYTHING
    ;51 using fipauth for generic aws/oauth/dropbox
    ;52-55 allow up to 9 http-header-file and amz STS
    ;56 19sep19 redid twitter for MediaUpload
    ;57-58 31mar20 added http-500-queue and max-500-retries and fh and PIPE
signals
    ;59 31mar21 added status* parameters and use the 10th/last loop as a means of
polling for a status
    ;60 17may21 added BDCAST_ZAP_SX fiphdr and zap-sx:no/yes ;61 throttle-gap
    ;62 10nov21 big files for aws (> 1gb) ;63-64 bug in log-each-msg
    ;65 10nov22 timeout and alarms better ;66 issuette in defPort ;67a minor plus
timeout for thru_socks_proxy
    ;68 added verbose-logging
    ;69 31aug23 added MULTIPLES plus BIO_close plus sigaction
    ;70-74  7sep23 oauth for gcp and -V now displays mimeheader/setup data too; 74
doneque renames duplicate files
    ;75 8mar24 MULTIPLES cleanup
    ;76 fipSSLtimeout added
;058z   18apr02 added major-takes*
    ;a 23apr02 added wordwrap as well as ww and made it FipSeq
    ;b/c 28may02 added grab-first-file: and SerDes is now FipHdr SF:
    ;d 19jun02 added chgchr to text as well
    ;e/f 17oct02 added split-text-string
    ;g 19dec02 added 'data-must-end-with:\n' and buglette with 1st splitter string
    ;h/i 01jan03 MACOSX
    ;j-n 12mar03 added split-delay/bugette in splitter/hdr-hash in splitters
    ;o 17sep03 added -U : hourlyDmpFiles = yes
    ;p 28oct03 added timing-stats:
    ;q-w 08mar04 better handling of http error q and connect-timeout
    ;x-y 12aug04 added save-http-response:
    ;z 23sep04 speedy
;057z   26apr00 added -k for delaying ALL traffic x seconds
    ;a 07jul00 cleanup
    ;b/c 29nov00 improved logeachmsg
    ;d 02mar01 cleanup after network error better
    ;e/f/g/h 13mar01 added Takes plus tweaked Check Messages
    ;j 04sep01 added newD1 - FipHdr D1 is ONLY output if there is a take-end/start
    ;k/m 10sep01 added fiphdr-for-hostname/ip-address and log-line
    ;o 21sep01 added failover-* and http-header
    ;p/q/r 24sep01 added take-extra-fiphdr and added the date to the dump file
    ;s 12nov01 if fixed interval chkmsgs, start with a chkmsg
    ;t 08jan02 added failover-check-time
    ;u/v 07mar02 added send-no-data
    ;w 18mar02 added remote-trace
    ;x/y 03apr02 create-ZQ added
    ;z 16apr02 added (optionally) D2, D3 for previous/last
        plus D4 is filled in at first go if filesize > than takesize

(copyright) 2024 and previous years FingerPost Ltd.