ipsoap
ipsoap
This program is used to handle repeated Requests/Responses with a remote REST
or SOAP server - or anything that handles R-R.
It allows up to 10 Requests to be made, checked and processed in succession.
These could be, for example,
- logon stage
- verification stage
- get 1st data stage
- get last data stage
Each stage is split into 4 main sections
- Beforehand
- Add FipHdr information from a string or a standing file (or the input file
if there is one)
the standing FipHDr file might be something saved from a previous run, such
as a cookie
- Test to see if the stage is required - eg dont logon if there is alredy a
logon token from a previous run
- Build and Send the Request
- Complete the mime header
- fill in the Envelope and Body using an existing template
- Analyse the Response
- Save the response and send it on to another Fip process
- Extract FipHdr fields from tags/tagattibutes and save for future use
- Test response for errors - commnucation, HTTP, data
and stop, continue or jump to another loop.
- Log it all and cleanup
The parameter file(s) are in tables/soap. The syntax of these are :
; comment start with a semi colon like this
Before section
fiphdr-beforefile-X:
Add/Parse extra filehdr fields from a standing file - such as Logon
Information
The format should be just a plain text file with FipHdr fields and NO
start/end FipHdr markers
eg
RL:fippo
RP:hoho23
fiphdr-before-X:
This adds a single line of FipHdrs (or a single FipHdr) to the FipHdr
This is added AFTER any fiphdr-beforefile-X, so previously added FH can be
used
Usually this is the Path of the Mimehdr
fiphdr-before-3:RX:/rmd/rest/xml/login?username=\RL&password=\RP
test-before-X:
test-before-2: fiphdr:R1="" action:goto1
Build and send the Request
mimehdr-X
Name of a MimeHeader template file in tables/soap.
if the mimehdr is always the same, use 'default-mimehdr:'
envelope-X
Name of template file in tables/soap
This is usually needed for SOAP as it will contian the main Soap Xml Request
For REST, a blank envelope file containing NO lines should be created and
specified
host-x
Name (and method) of remote host for this step.
eg host-5:https://rmd.wiki.com
This is ONLY needed if the host is different to the default input switch
'-s'
Analyse the Response
save-response-9:(filename)
queue:(output folder)
fiphdr:(optional extraFipHdr to add )
before:(optional data to add before)
after:(optional data to add after)
tag:(name of tag if NOT all the response is to be saved)
data:xml or txt or raw or NO for none
data:(dont send any data from the response - this assumes there is fiphdr
and/or before and/or after)
fiphdr-after-X:
test-after-X: [result or fiphdr or tag] [= or != or #] (some test)
action: [stop or continue or gotoX or error]
action:stop stop all processing
action:error stop all processing and flag an error
action:continue continue with the next Loop (default action)
action:goto9 continue at loop 9 (or whichever number you have specified)
log: (Fipseq - extra information to add the the log)
script: (Script or program to run in FipSeq)
result=200
result=timeout
tag:(tag name) save:(name of fiphdr file)
test-after-1: result=404 action:error
; ---------------------------------------------
default-mimehdr:SOAP.MIMEHDR
; -------- loop 0 ------------
; logon and get a token
fiphdr-before-0:RX:/api/2006/05/01/TokenManagement_1.svc/Anonymous
envelope-0:rkd.create.service.token.xml
fiphdr-after-0:R1 tag:Expiration save:exp.file
fiphdr-after-0:R2 tag:global:Token save:token.file
OPTIONAL:
port: Port number of the Remore Server. default: 80
This forces the port to be this if none is specified.
donque: done queue for the input file.
nofiphdr: Do NOT add a Fip Hdr to the file. default: yes pls
httphdr: Extra lines of HTTP header you may need. default: none
Remember to add a NL at the end of each line.
There can be multiple httphdr lines but pls remember to add a '\n' at the
end of each one. (or you can try to force all on one httphdr line!)
eg httphdr:Authorization: Basic AbGtGgbhpdOkOTE=\n
httphdr:User-Agent: Mozilla/4.0\n
httphdr:Host: wibble.wobble.com\n
see below for 'useful, common header lines'
filename: Filename for the output file in FipSeq. default: WEB\$Z
If this does NOT start with a '/' it is left under the
Output Queue as specified on startup (default spool/2go)
eg filename:AFP\$d.\$z
log: FipSeq custom logging for the item log. default:\SN \SU \EF : \ST,\SP
This logs each Page grabbed
Note that
ST remote site host
SP remote site port
SF remote site url
extra:
extra-pre: Extra FipHdr fields to be added to the output file. default: none
To separate FipHdr fields, pls use a '#'.
extra-pre is added as soon as the file is read - so may be used for
information in the URL
extra is only used for any output file and is not used at all for any other
purpose.
eg extra:ZH:NYNZ#DI:Headline News#QZ:333
connection-timeout: (secs)
wait-end-timeout: (secs)
For slow, busy sites, data - especially big files - may take a lot longer
than normal to be
retreived. Use this to expand that time. Default is 120 (it should be
divisible by 5 for some arcane reason)
no-data: (FipSeq string in place of data)
Do not get/send the data - just this string
proxy-server: If using a proxy, these are the name and port to aim at.
proxy-port:
proxy-logon: This is the logon and password to get thru the firewall
if required. The format is (logon) (colon) (password) and is
converted to base 64.
proxy-logon:Y2hyaXMuaHVnaGpvbmVzOnBhbnRoZXIK=
To generate :
echo -n "logon:password" | sffb64 -i
eg echo -n "chris:sleekpanther" | sffb64 -i
gives Y2hyaXM6c2xlZWtwYW50aGVy
proxy-logon:Y2hyaXM6c2xlZWtwYW50aGVy=
proxy-is-squid:yes/no Is the proxy a Squid ? default: no
proxy-handshake:yes/no Does the proxy need to say hello first ? default: no
logeachfile:(dest) Send a Success/failed msg to this destination
for each file. There is no default. This log file is
just a FipHdr with the following extra fields :
DR-File Sent OK DR:ok or DR:error
DG-Will Retry later DG:retrying, DG:stopped
DT-Some message text DT:No connection
default: no log created.
The text for the DR and DG can be in FipSeq and so can contain
FipHdr and other variables. As they are FipHdr fields, please
do NOT put NL, CR etc in the fields.
do NOT put NL, CR etc in the fields.
Note that System Variable \$q holds the time taken for transmission.
DRgood:(text) Message for the FipHdr field DR on a successful tx
default: ok
DRbad: (text) Message for the FipHdr field DR on a unsuccessful tx
default: error
DGcont:(text) Message for the FipHdr field DG if, after an
unsuccessful tz, another attempt will be made.
default: retrying
DGstop:(text) Message for the FipHdr field DG if no further
attempts will be made as the file was sent successfully
or the maximum no of attempts has been tried.
default: stopped
fiphdr-for-logeachfile: (FipSeq) or
msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
'logeachfile' or 'loglasterrfile' msg. This should be in FipHdr
format and be in FipSeq. It can be used to pass FipHdr fields
in the outgoing file into the log file.
eg msgeachfile: DF:logdial\nSS:\SS\n default: nothing added
check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
Use this when an ipspool on 2 systems is accessing the same remote folder -
in a redundant way.
that is used whether the current host should be getting the files or not.
ie in the parameter file is
check-primary-server:timersvr
and in the DEST_REDUN is
; psuedohost primary secondary
timersvr fip1 fip2
and in the SYSTEM file for both fip1 AND fip2 there is the same 'ipsoap'
line with the same parameter file
If fip1 is up, 'ipsoap' on fip1 will always get while on fip2 it will just
check/loop.
If fip1 is down 'ipsoap' on fip2 will start processing the files.
max-loop-retry default 20
use-ssl: no/yes
Use Secure Sockets Layer (TLS/SSL) default: no
ssl-method: tls tls1 tls1.1 tls1.2 sslv2 sslv3 sslv2and3
Version number to use for TLS/SSL default: 999 for current default (2 or 3)
(only the digits are significant, so add other text to make it readable)
For 'modern' connection, pls do NOT use sslv2 ! as it is deemed insecure
If default it will check the available list and pick the highest.
The default is currently 23 which on a modern server is sslv3 and tls1_2 !)
ssl-password: (password)
ssl-passwd: (password) default: none
Optional password if the handshake requires a shared secret
ssl-key: (name of a certiticate key file) default: none
ssl-cert: (name of a certificate file) default: none
ssl-root-cert: (name of a root PEM certificate file) defaunt: none
Optional certificates are in tables/ssl unless name starts with '/'
ssl-verify: yes/no verify server certificates default: yes
ssl-ciphers: (list) acceptable ciphers
(use 'openssl ciphers' to list)
default: "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS"
ssl-display: yes/no display SSL connection details default: no
Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).
They are normally specified :
fixed:QZ 1234543
partial:QT ST,3,2,U,<,>
combie:QY ep|na,(0000000)a
option:QE ep,11,7,s
repeat:QK XK,-,3
or repeat:QP PK,,4,#X
style:QS XN,%.03d
Input Parameters are (Mandatory) :
-s : hostname of the remote server default: none
-z : default parameter file in tables/soap default: SOAP
Either
-1 : path/filename for single shot default: none
The input file is NOT deleted
If this does NOT start with a '/', it is assumed relative to the current
path.
Or
-i or -q : input queue default: none
If this does NOT start with a '/', it is assumed under spool.
(Optional)
-d : display the transactions default: no
-D : display the transactions default: no
break out the xml of the response
-F : use FipHdr field DF for parameter file default: always use -z parm file
-G : no of secs between reruns default: run once
-h : extraFipHdr information default: none
to be used during the requests and for any output files
this is separate and additional to the 'extra-fiphdr' keyword in the
parameter file
-H : alias hostname (or cluster hostname) default: none
use this for checking primary/secondary where the actual hostname might be
different from the cluster
-l : log level default: log errors
-o : default output queue for any grabbed files default: spool/2go
If this does NOT start with a '/', it is assumed under spool.
This is overridden by any path/folder in the save-response parameter
-p : port number of the remote soap server default: 80
-u : dump the transactions in /fip/dump default: no
-U : Break the Dump file (-d) into Hourly files. default: no - daily
-w : file wait for files arriving across a network. default: none
-v : print version number and exit
---NOTES---
Version Control
;0f22 20nov01 original version
;f10-12 better ssl
;f13-14 20mar11 added host-X
;16 9may11 added script to test-after
;17-19 19apr16 added proxyHandshake etc and bio/ssl ;20-20-22 1may18
up-to-date SSL
(copyright) 2024 and previous years FingerPost Ltd.