ipsoap This program is used to handle repeated Requests/Responses with a remote REST or SOAP server - or anything that handles R-R. It allows up to 10 Requests to be made, checked and processed in succession. These could be, for example, - logon stage - verification stage - get 1st data stage - get last data stage Each stage is split into 4 main sections - Beforehand - Add FipHdr information from a string or a standing file (or the input file if there is one) the standing FipHDr file might be something saved from a previous run, such as a cookie - Test to see if the stage is required - eg dont logon if there is alredy a logon token from a previous run - Build and Send the Request - Complete the mime header - fill in the Envelope and Body using an existing template - Analyse the Response - Save the response and send it on to another Fip process - Extract FipHdr fields from tags/tagattibutes and save for future use - Test response for errors - commnucation, HTTP, data and stop, continue or jump to another loop. - Log it all and cleanup The parameter file(s) are in tables/soap. The syntax of these are : ; comment start with a semi colon like this Before section fiphdr-beforefile-X: Add/Parse extra filehdr fields from a standing file - such as Logon Information The format should be just a plain text file with FipHdr fields and NO start/end FipHdr markers eg RL:fippo RP:hoho23 fiphdr-before-X: This adds a single line of FipHdrs (or a single FipHdr) to the FipHdr This is added AFTER any fiphdr-beforefile-X, so previously added FH can be used Usually this is the Path of the Mimehdr fiphdr-before-3:RX:/rmd/rest/xml/login?username=RL&password=RP test-before-X: test-before-2: fiphdr:R1="" action:goto1 Build and send the Request mimehdr-X Name of a MimeHeader template file in tables/soap. if the mimehdr is always the same, use 'default-mimehdr:' envelope-X Name of template file in tables/soap This is usually needed for SOAP as it will contian the main Soap Xml Request For REST, a blank envelope file containing NO lines should be created and specified host-x Name (and method) of remote host for this step. eg host-5:https://rmd.wiki.com This is ONLY needed if the host is different to the default input switch '-s' Analyse the Response save-response-9:(filename) queue:(output folder) fiphdr:(optional extraFipHdr to add ) before:(optional data to add before) after:(optional data to add after) tag:(name of tag if NOT all the response is to be saved) data:xml or txt or raw or NO for none data:(dont send any data from the response - this assumes there is fiphdr and/or before and/or after) fiphdr-after-X: test-after-X: [result or fiphdr or tag] [= or != or #] (some test) action: [stop or continue or gotoX or error] action:stop stop all processing action:error stop all processing and flag an error action:continue continue with the next Loop (default action) action:goto9 continue at loop 9 (or whichever number you have specified) log: (Fipseq - extra information to add the the log) script: (Script or program to run in FipSeq) result=200 result=timeout tag:(tag name) save:(name of fiphdr file) test-after-1: result=404 action:error ; --------------------------------------------- default-mimehdr:SOAP.MIMEHDR ; -------- loop 0 ------------ ; logon and get a token fiphdr-before-0:RX:/api/2006/05/01/TokenManagement_1.svc/Anonymous envelope-0:rkd.create.service.token.xml fiphdr-after-0:R1 tag:Expiration save:exp.file fiphdr-after-0:R2 tag:global:Token save:token.file OPTIONAL: port: Port number of the Remore Server. default: 80 This forces the port to be this if none is specified. donque: done queue for the input file. nofiphdr: Do NOT add a Fip Hdr to the file. default: yes pls httphdr: Extra lines of HTTP header you may need. default: none Remember to add a NL at the end of each line. There can be multiple httphdr lines but pls remember to add a 'n' at the end of each one. (or you can try to force all on one httphdr line!) eg httphdr:Authorization: Basic AbGtGgbhpdOkOTE=n httphdr:User-Agent: Mozilla/4.0n httphdr:Host: wibble.wobble.comn see below for 'useful, common header lines' filename: Filename for the output file in FipSeq. default: WEB$Z If this does NOT start with a '/' it is left under the Output Queue as specified on startup (default spool/2go) eg filename:AFP$d.$z log: FipSeq custom logging for the item log. default:SN SU EF : ST,SP This logs each Page grabbed Note that ST remote site host SP remote site port SF remote site url extra: extra-pre: Extra FipHdr fields to be added to the output file. default: none To separate FipHdr fields, pls use a '#'. extra-pre is added as soon as the file is read - so may be used for information in the URL extra is only used for any output file and is not used at all for any other purpose. eg extra:ZH:NYNZ#DI:Headline News#QZ:333 connection-timeout: (secs) wait-end-timeout: (secs) For slow, busy sites, data - especially big files - may take a lot longer than normal to be retreived. Use this to expand that time. Default is 120 (it should be divisible by 5 for some arcane reason) no-data: (FipSeq string in place of data) Do not get/send the data - just this string proxy-server: If using a proxy, these are the name and port to aim at. proxy-port: proxy-logon: This is the logon and password to get thru the firewall if required. The format is (logon) (colon) (password) and is converted to base 64. proxy-logon:Y2hyaXMuaHVnaGpvbmVzOnBhbnRoZXIK= To generate : echo -n "logon:password" | sffb64 -i eg echo -n "chris:sleekpanther" | sffb64 -i gives Y2hyaXM6c2xlZWtwYW50aGVy proxy-logon:Y2hyaXM6c2xlZWtwYW50aGVy= proxy-is-squid:yes/no Is the proxy a Squid ? default: no logeachfile:(dest) Send a Success/failed msg to this destination for each file. There is no default. This log file is just a FipHdr with the following extra fields : DR-File Sent OK DR:ok or DR:error DG-Will Retry later DG:retrying, DG:stopped DT-Some message text DT:No connection default: no log created. The text for the DR and DG can be in FipSeq and so can contain FipHdr and other variables. As they are FipHdr fields, please do NOT put NL, CR etc in the fields. do NOT put NL, CR etc in the fields. Note that System Variable $q holds the time taken for transmission. DRgood:(text) Message for the FipHdr field DR on a successful tx default: ok DRbad: (text) Message for the FipHdr field DR on a unsuccessful tx default: error DGcont:(text) Message for the FipHdr field DG if, after an unsuccessful tz, another attempt will be made. default: retrying DGstop:(text) Message for the FipHdr field DG if no further attempts will be made as the file was sent successfully or the maximum no of attempts has been tried. default: stopped fiphdr-for-logeachfile: (FipSeq) or msgeachfile:(FipSeq) Additional information to add to the FipHdr of the 'logeachfile' or 'loglasterrfile' msg. This should be in FipHdr format and be in FipSeq. It can be used to pass FipHdr fields in the outgoing file into the log file. eg msgeachfile: DF:logdialnSS:SSn default: nothing added check-primary-server: pseudo-host name that is specified in tables/sys/DEST_REDUN Use this when an ipspool on 2 systems is accessing the same remote folder - in a redundant way. that is used whether the current host should be getting the files or not. ie in the parameter file is check-primary-server:timersvr and in the DEST_REDUN is ; psuedohost primary secondary timersvr fip1 fip2 and in the SYSTEM file for both fip1 AND fip2 there is the same 'ipsoap' line with the same parameter file If fip1 is up, 'ipsoap' on fip1 will always get while on fip2 it will just check/loop. If fip1 is down 'ipsoap' on fip2 will start processing the files. max-loop-retry default 20 ssl-method: (1,2,3,23,999) Version number to use for TLS/SSL default: 999 for current default (2 or 3) ssl-password: (password) ssl-passwd: (password) default: none Optional password if the handshake requires a shared secret ssl-cert: (name of a PEM certificate file) default: none ssl-root-cert: (name of a root PEM certificate file) defaunt: none Optional certificates - held in tables/ssl Where sections of FipHdr fields are required or changes to the output style, use keywords : fixed, partial, combie, optional, repeat, newdate and/or style. (see The SysAdmin manual for more information). They are normally specified : fixed:QZ 1234543 partial:QT ST,3,2,U,<,> combie:QY ep|na,(0000000)a option:QE ep,11,7,s repeat:QK XK,-,3 or repeat:QP PK,,4,#X style:QS XN,%.03d Input Parameters are (Mandatory) : -s : hostname of the remote server default: none -z : default parameter file in tables/soap default: SOAP Either -1 : path/filename for single shot default: none The input file is NOT deleted If this does NOT start with a '/', it is assumed relative to the current path. Or -i or -q : input queue default: none If this does NOT start with a '/', it is assumed under spool. (Optional) -d : display the transactions default: no -D : display the transactions default: no break out the xml of the response -F : use FipHdr field DF for parameter file default: always use -z parm file -G : no of secs between reruns default: run once -h : extraFipHdr information default: none to be used during the requests and for any output files this is separate and additional to the 'extra-fiphdr' keyword in the parameter file -H : alias hostname (or cluster hostname) default: none use this for checking primary/secondary where the actual hostname might be different from the cluster -l : log level default: log errors -o : default output queue for any grabbed files default: spool/2go If this does NOT start with a '/', it is assumed under spool. This is overridden by any path/folder in the save-response parameter -p : port number of the remote soap server default: 80 -u : dump the transactions in /fip/dump default: no -U : Break the Dump file (-d) into Hourly files. default: no - daily -w : file wait for files arriving across a network. default: none -v : print version number and exit ---NOTES--- Version Control ;00f16 20nov01 original version ;f10-12 better ssl ;f13-14 20mar11 added host-X ;16 9may11 added script to test-after (copyright) 2014 and previous years FingerPost Ltd.