ipbalan'ce This program is used to balance data files on Fip systems. It sends files so marked to IPSVRD or equivalent on other systems It can either check the FipHdr for information where to send or can be forced to send all data to a group/single box. If for some reason the remote box is down, 'ipbalan' will move the files to an offline queue and resend when that host is restored. The FipHdr of the file is checked and the following headers are used : DG - name of the group - mandatory if not running 'force-group' SX - full pathname of data file on both source and remote systems If there is NO SX field, the data is assumed to follow the FipHdr in the file or there is NO data. DN - full pathname of data file on the remote system This overrides the SX FipHdr field for the REMOTE only. DK - action on the remote - optional, defaults to STORE 'store' - Store this file 'append' - Append data to this file 'link' - Link this file - unix only 'delete' - Delete this file 'script' - run this script 'ignore' - just this one 'notes-upload' - upload to notes These are optional and only available on Unix DD - if 'store' or 'link', link the new file TO a this name DV - if 'link', link the file FROM this file. If the file is NOT found and there is data attached to the incoming XML, it is used instead. As the '#' is an end-of-fip-hdr marker, SX and DN are parsed so that # should be encoded as the 4 printable chrs '43' (otherwise SX and DN will be truncated to end on the first '#'). eg DN:/fip/spool/testdata/SN or DN:/fip/spool/DQ/43SN:SN43DQ:endfolder If DQ is 'pdf' and SN is 'abc123' this gives DN:/fip/spool/pdf/#SN:abc123#DQ:endfolder if the FipHdr field ZBALANCE-DATA exists, the SX field is used for the file name on the remote and the Data following the FipHdr is used. if the FipHdr field ZBALANCE-STUB exists as well as an SX FipHdr field (with an external filename), then ONLY the external file is sent. Normally the two are merged unless the 'nofiphdr:' subkeyword is specified for that 'group:(server)' If the file cannot be sent, it is moved to the offline queue and the FipHdr field 'CY' used to denote the hostname of the failed system. The offline folder is scanned once a minute to see what needs to be retried. If the FipHdr field ZBALANCE-ID exists and the remote system is offline, then only a single copy of the file to be sent will be saved - normally this mechanism is used for balancing skipfiles or other standing files. When the remote system reappears, only a single copy of the file will be enough to bring it up-to-date. If the FipHdr field ZBALANCE-ZAP-INPUT:(filename) exists, the file pointed to will be zapped after the data has been sent (successfully). The file MUST be in /fip/data/.. /fip/spool/... or in /fip/x starting 'TMP...' The parameter file in tables/sys, normally called BALANCE has the following parameters : ; comment group:(name) host:(host name) port:1234 fippath:/data4/fip notjustfip: name is the name of this group - there can be one or many lines all with the same group name. hostname is a valid IP address or name port if NOT the default which is 9140 for 'ipsvrd' fippath default is /fip only need if NOT /fip notjustfip: default is just under /fip nolog: do NOT log the item nofiphdr: Do NOT send the FipHdr Normally we send the FipHdr as well as data. This really refers to the incoming file in spool/2balance and NOT any external file. So if the data follows the FipHdr (ie there is NO external file - SX:), this flag will send the data only and NO FipHdr. if there is an external file and it has a FipHdr, this is considered part of the data anyway and is NEVER stripped off for transmission. So the complete external file is ALWAYS sent. So if you have an external file which has a FipHdr, always specify this option - eg for FipRemotes noxmlhdr: The file already contains FipAgent XML so we do NOT need to send one. notfipagent: NO= none, YES=fipagent, MAYBE= just text nowait Do NOT wait for the response from the other end There may be someone who needs this but we have not worked out why ! round-robin-mode: failover-mode: Normally 'ipbalan' sends to all servers in the Group. These flags denote only one copy needs to be sent to only one in the group. So instead of one file sent to ALL systems in the group, a file to ONE of the group : - for failover, the first one specified if it is live, second if not, etc. - for round-robin, tripping from one server to the next In both cases, you will need to track that each remote server is currently live (or not) using tables/sys/REDUN and ipredun/ipxnet to monitor them ignore-localhost: For this group, do NOT send if the remote host is actually the local machine. force-queue: (queue or folder on remote system) For STORE, all files are to be sent to this folder on the remote system The default is the contents of the DN FipHdr field (or SN/SX) newname: (replacement name on the remote system if there is no DN FipHdr field) Otherwise The name on the remote system is (in sequence) either contents of the DN FipHdr field (if there) or contents of SX or contents of SN force-newname: (replacement name on the remote system which overrides DN, SX, SN) use-permissions:no/yes Use same permissions on the remote as on the source (unix only) So if the source file has permissions of RWXRWXRWX (777) - it will be made so on the remote. The default is NO (or YES if -P input switch is set) meaning the remote permissions will be the same as the logon which started 'ipsvrd' overwrite: yes/no Overwrite the file at the remote host - default yes unless -U unique-name: SYSTEM name of a file which will be different on the remote systems hostname is automatically suffixed. ie SYSTEM -> SYSTEM.fip1 default-group: (name of a group) use this Group if none is specified (same as -d Input switch) retry-timeouts:(yes/no) if a message to 'ipsvrd' timeout or the link is broken, the default is to retry ad-infinitum. Use this parameter 'retry-timeouts:no' to move the file to 'woops' allow-data-with-script: (yes/no) normally SCRIPT does not have any data sent; use this to do so. The data is put into a Temporary file which the script can use as FipHdr SX max-open-retries: (number) Number of retries to open a connection - default is 5 open-timeout: (seconds) default: 120 Wait time in seconds before the remote host should allow a connection Reduce this to 10 or so if the remote is just across the computer room ! timeout: (seconds) default: 120 Wait time in seconds before the remote host should ack a file once sent If sending large files over very noisy or busy or slow connections, try increasing this if you get lots of errors. The following are only for use with the XML FipAgent and rarely need changing : wrapper: xml wrapper string eg : wrapper:DEVICE_MESSAGE default is FIPAGENT data-length: tag containing the length of data following the wrapper default is DATALENGTH action: Action of the message default is REPLY message: FipAgent Error Message default is MSG result: FipAgent result code default is RESULT device-result: result code from the host process default is DEVICE-RESULT poll-timeout: no of seconds which a reply should arrive default is 20 (serial running only) timeout-message: no of seconds before logging a slow transmission default is 20 (parallel running only) log-line: Extra logging in FipSeq default is none ; ; BALANCE ; unique-name:SYSTEM ; ; group:tables host:fip1 port:9002 group:tables host:fip2 port:9002 fippath:/data3/fip group:w4 host:fip1 notjustfip: group:w4 host:fip2 notjustfip: Input Parameters are (all optional): -1 : (filename) send a single file and stop default: spool -a : default action if none specified default: STORE -d : default group if none specified default: check each file -E : ignore any external file (SX: FipHdr field). default: send contents of SX if present -f : force all data to this group or client default: check each file -I : (eye) increment the Port number default: always send to the 9140 (or the port) ipbalan check the IPaddress of its own system and takes the LAST digit and adds that to 9140 ie if the IPaddress is 195.13.83.27 then the last digit is 7 and the remote port -> 9147 there should be multiple ipsvrd -P914X (where X is 0->n) on each system -J : max increment for -I default: 10 -l : (el) log each file through default: no -L : log each file in /fip/log/BALAN default: no -o : done queue for raw files after processing default: deleted -O : Overwrite files in Offline with the same name. default: all files are unique -q : queue to poll default: spool/2balance -Q : quiet mode - do not flag any missing files.default: do -R : run Remotebalance in parallel default: no The parameter is the group in BALANCE ** This option is ONLY for balancing ipremdata files -U : make files on the remote server unique and not overwrite -w : file wait for files arriving via NFS etc default: 0 secs -W : 10 ths of a second - MUST wait if nothing processed default: 2 for 2/10ths -x : do NOT use the offline folder as ALL default: do traffic must go to this client. Use this ONLY when sending to a single remote system. -z : Parameter file default: tables/sys/BALANCE -v : print version no and exit Current maximums Groups and Hosts 400 Unique Filenames 100 Balance can be used to get 'ipsvrd' to run a script on the remote systems : Create a file with the following FipHdr fields - remember the tilde NL at the top: ~ DK:SCRIPT SX:/fip/local/fff UP DG:scriptgroup where DK is fixed as 'SCRIPT' SX is the full path and filename of the script with any parameters; it can be a FipSeq string. You must be able to run the script or program from the logon which 'ipsvrd' has been started. DG is the normal group which determines which systems to run the script. Example BALANCE group:fiptest host:fip1 ignore-localhost nofiphdr: group:fiptest host:fip2 ignore-localhost nofiphdr: group:fiptest host:fip3 ignore-localhost nofiphdr: group:fiptest host:fip4 ignore-localhost nofiphdr: .. and the stub file ~ DG:fiptest DK:STORE XX:time.1180132986.pid.11281.seq.1 XX:/fip/bnew/ipvi XX:at.Fri May 25 23:43:06 2007 SX:TTTT DN:/fip/spool/xo Version Control ;002w10 08dec05 allow zero length files to be copied over ;b-f 24feb06 cater for Resource problems on Sparc (f for 64bit wait3) ;g-h 13sep06 added force-queue: ;i 23sep06 added max-open-retries and bug in offline sometimes sending to more than one system = DONT USE ;j-l 03oct06 new style DR with up to 10 secondaries PLUS bug in 02i PLUS bug in single ;m-q 07nov06 added action notes-upload and -a (; p speedy) ;r-u 31mar07 added preserve-filename: and log-line: and timing-stats ;v9 31may07 redid modes/permissions and added overwrite and -U ;w4 31jul08 added -E to ignore-external-data ;1 19may09 added ZBALANCE-ID ; 2 8jun09 added ZBALANCE_STUB ;4 parse doneque ;5-6 added ZBALANCE-ZAP-INPUT ;7 round-robin-mode added ;9 added force-name ;10 added file_trace ;001z5 19may00 redid offline a bit ;b 05jun00 added LINK and DV ;c 14nov00 added DN for name on remote ;d/e 25apr01 allow data-in-file as a default if no SX -this is a change! ;f/g/h 02jan02 added failover: and bugette for FipHdr on File and no SX ;h/i 22mar02 SX and DN are now parsed so that # should be encoded as 43 ;j 25jul02 redid Timeout routine ;k 28jul02 buggette in WINNT offline ;l/m/n 09aug02 added open-timeout ;o 28aug02 better handling of 4xx errors - send to woops ;p 28aug02 added retry-timeouts. ;q-s 14jul03 added APPEND as an Action ;t 17sep04 speedy ;u-y 25feb05 remote parallel sends ;z5 09aug05 added -I increment Ports and -O overwrite offline files (copyright) 2014 and previous years FingerPost Ltd.