ipbalan
ipbalan'ce
This program is used to balance data files on Fip systems. It sends files so
marked to IPSVRD or equivalent on other systems
It can either check the FipHdr for information where to send or can be forced
to send all data to a group/single box.
If for some reason the remote box is down, 'ipbalan' will move the files to an
offline queue and resend when that host is restored.
The FipHdr of the file is checked and the following headers are used :
DG - name of the group - mandatory if not running 'force-group'
SX - full pathname of data file on both source and remote systems
If there is NO SX field, the data is assumed to follow
the FipHdr in the file or there is NO data.
DN - full pathname of data file on the remote system
This overrides the SX FipHdr field for the REMOTE only.
DK - action on the remote - optional, defaults to STORE
'store' - Store this file
'append' - Append data to this file
'copy' - Copy this file
'move' - Move this file
'link' - Link this file - unix only
'delete' - Delete this file
'script' - run this script
'ignore' - just this one
'notes-upload' - upload to notes
These are optional and only available on *nix
DD - if 'store' or 'link', link the new file TO a this name
DV - if 'link', link the file FROM this file. If the file is NOT found
and there is data attached to the incoming XML, it is used instead.
As the '#' is an end-of-fip-hdr marker, SX and DN are parsed so that # should
be encoded as the 4 printable chrs '\043' (otherwise SX and DN will be
truncated to end on the first '#').
eg DN:/fip/spool/testdata/\SN
or DN:/fip/spool/\DQ/\043SN:\SN\043DQ:endfolder
If DQ is 'pdf' and SN is 'abc123' this gives
DN:/fip/spool/pdf/#SN:abc123#DQ:endfolder
if the FipHdr field ZBALANCE-DATA exists, the SX field is used for the file
name on the remote and the Data following the FipHdr is used.
(ZBALANCE-DATA:no also negates this; ZBALANCE-DATA:(anything else) or
ZBALANCE-DATA: is valid)
if the FipHdr field ZBALANCE-STUB exists as well as an SX FipHdr field (with an
external filename), then ONLY the external file is sent. Normally the two are
senr individually unless the 'nofiphdr:' subkeyword is specified for that
'group:(server)'
(ZBALANCE-STUB:no is the same as ZBALANCE-STUB: while ZBALANCE-STUB:yes will
Also balance the Stub FipHdr as well as the External file)
If the file cannot be sent, it is moved to the offline queue and the FipHdr
field 'CY' used to denote the hostname of the failed system.
The offline folder is scanned once a minute to see what needs to be retried.
If the FipHdr field ZBALANCE-ID exists and the remote system is offline, then
only a single copy of the file to be sent will be saved - normally this
mechanism is used for balancing skipfiles or other standing files. When the
remote system reappears, only a single copy of the file will be enough to bring
it up-to-date.
If the FipHdr field ZBALANCE-ZAP-INPUT:(filename) exists, the file pointed to
will be zapped after the data has been sent (successfully). The file MUST be in
/fip/data/.. /fip/spool/... or in /fip/x starting 'TMP...'
The parameter file in tables/sys, normally called BALANCE has the following
parameters :
; comment
group:(name) host:(host name) port:1234 fippath:/data4/fip notjustfip:
name is the name of this group - there can be one or many lines
all with the same group name.
hostname is a valid IP address or name
port if NOT the default which is 9140 for 'ipsvrd'
fippath default is /fip
only need if NOT /fip
notjustfip:
default is just under /fip
nolog: do NOT log the item
nofiphdr: Do NOT send the FipHdr
Normally we send the FipHdr as well as data.
This really refers to the incoming file in
spool/2balance and NOT any external file.
So if the data follows the FipHdr (ie there is NO
external file - SX:), this flag will send the data only
and NO FipHdr.
if there is an external file and it has a FipHdr, this
is considered part of the data anyway and is NEVER
stripped off for transmission. So the complete external
file is ALWAYS sent.
So if you have an external file which has a FipHdr,
always specify this option - eg for FipRemotes
noxmlhdr: The file already contains FipAgent XML so we
do NOT need to send one.
notfipagent:
NO= none, YES=fipagent, MAYBE= just text
nowait
Do NOT wait for the response from the other end
There may be someone who needs this but we have not
worked out why !
round-robin-mode:
failover-mode:
Normally 'ipbalan' sends to all servers in the Group. These flags denote
only one copy needs to be sent to only one in the group.
So instead of one file sent to ALL systems in the group, a file to ONE of
the group :
- for failover, the first one specified if it is live, second if not, etc.
- for round-robin, tripping from one server to the next
In both cases, you will need to track that each remote server is currently
live (or not) using tables/sys/REDUN and ipredun/ipxnet to monitor them
ignore-localhost:
For this group, do NOT send if the remote host
is actually the local machine.
force-queue: (queue or folder on remote system)
For STORE, all files are to be sent to this folder on the remote system
The default is the contents of the DN FipHdr field (or SN/SX)
newname: (replacement name on the remote system if there is no DN FipHdr
field)
Otherwise The name on the remote system is (in sequence)
either contents of the DN FipHdr field (if there)
or contents of SX
or contents of SN
force-newname: (replacement name on the remote system which overrides DN, SX,
SN)
preserve-newname: yes/no (keep exactly the same name on the remote)
use-permissions:no/yes
Use same permissions on the remote as on the source (unix only)
So if the source file has permissions of RWXRWXRWX (777) - it will be made
so on the remote.
The default is NO (or YES if -P input switch is set) meaning the remote
permissions will be the same as the logon which started 'ipsvrd'
overwrite: yes/no
Overwrite the file at the remote host - default yes unless -U
ignore-links: yes/no
ignore DD and DV default NO
unique-name: SYSTEM
name of a file which will be different on the remote systems
hostname is automatically suffixed.
ie SYSTEM -> SYSTEM.fip1
default-group: (name of a group)
use this Group if none is specified (same as -d Input switch)
retry-timeouts:(yes/no)
If a message to 'ipsvrd' timeout or the link is broken, the default is to
retry ad-infinitum.
Use this parameter 'retry-timeouts:no' to move the file to 'woops'
allow-data-with-script: (yes/no)
normally SCRIPT does not have any data sent; use this to do so.
The data is put into a Temporary file which the script can use as FipHdr SX
max-open-retries: (number)
Number of retries to open a connection - default is 3 (was 5)
open-timeout: (seconds) default: 10 (was 120)
Wait time in seconds before the remote host should allow a connection
Reduce this to 10 or so if the remote is just across the computer room !
timeout: (seconds) default: 20 (was 120)
Wait time in seconds before the remote host should ack a file once sent
If sending large files over very noisy or busy or slow connections, try
increasing this if you get lots of errors.
use-ssl:yes/no
The commends are for a ftp running over SSL/TLS on the remote server
default is NO
no - normal, standard plain TCP
yes - connect (normally) and immediately start SSL ontop
ssl-method: (1,2,3,23,999)
Version number to use for TLS/SSL default: 999 for current default (2 or 3)
ssl-password: (password)
ssl-passwd: (password) default: none
Optional password if the handshake requires a shared secret
ssl-key: (name of a certiticate key file) default: none
ssl-cert: (name of a certificate file) default: none
ssl-root-cert: (name of a root PEM certificate file) defaunt: none
Optional certificates are in tables/ssl unless name starts with '/'
ssl-verify: yes/no verify server certificates default: yes
ssl-ciphers: (list) acceptable ciphers
(use 'openssl ciphers' to list)
default: "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS"
ssl-display: yes/no display SSL connection details default: no
The following are only for use with the XML FipAgent and rarely need changing :
wrapper: xml wrapper string
eg : wrapper:DEVICE_MESSAGE
default is FIPAGENT
data-length: tag containing the length of data following the wrapper
default is DATALENGTH
action: Action of the message
default is REPLY
message: FipAgent Error Message
default is MSG
result: FipAgent result code
default is RESULT
device-result: result code from the host process
default is DEVICE-RESULT
poll-timeout: no of seconds which a reply should arrive
default is 20 (serial running only)
timeout-message: no of seconds before logging a slow transmission
default is 20 (parallel running only)
log-line: Extra logging in FipSeq
default is none
;
; BALANCE
;
unique-name:SYSTEM
;
;
group:tables host:fip1 port:9002
group:tables host:fip2 port:9002 fippath:/data3/fip
group:w4 host:fip1 notjustfip:
group:w4 host:fip2 notjustfip:
; using Balance to replace 2net
; 2net local ipbalan -q 2net -f route2net
combie:QP CP|DP,localhost
group:route2net host:\QP ignore-localhost: force-queue:/fip/spool/2go
preserve-name:yes
Input Parameters are (all optional):
-9 : on a FipSpeedy system, ignore speedyness
-1 : (filename) send a single file and stop default: spool
-a : default action if none specified default: STORE
-d : default group if none specified default: check each file
-E : ignore any external file (SX: FipHdr field). default: send contents of SX
if present
-f : force all data to this group or client default: check each file
-I : (eye) increment the Port number default: always send to the 9140 (or the
port)
ipbalan check the IPaddress of its own system and takes the LAST digit and
adds that to 9140
ie if the IPaddress is 195.13.83.27 then the last digit is 7 and the remote
port -> 9147
there should be multiple ipsvrd -P914X (where X is 0->n) on each system
-J : max increment for -I default: 10
-l : (el) log each file through default: no
-L : log each file in /fip/log/BALAN default: no
-o : done queue for raw files after processing default: deleted
-O : Overwrite files in Offline with the same name. default: all files are
unique
-p : preserve file permissions on remote(s) default: default will be rw-rw----
(660)
-P : do NOT preserve file permissions on remote
-q : queue to poll default: spool/2balance
-Q : quiet mode - do not flag any missing files.default: do
-R : run Remotebalance in parallel default: no
The parameter is the group in BALANCE
** This option is ONLY for balancing ipremdata files
-U : make files on the remote server unique and not overwrite
-w : file wait for files arriving via NFS etc default: 0 secs
-W : 10 ths of a second - MUST wait if nothing processed default: 2 for
2/10ths
-x : do NOT use the offline folder as ALL default: do
traffic must go to this client.
Use this ONLY when sending to a single remote system.
-z : Parameter file default: tables/sys/BALANCE
-v : print version no and exit
Current maximums
Groups and Hosts 400
Unique Filenames 100
Balance can be used to get 'ipsvrd' to run a script on the remote systems :
Create a file with the following FipHdr fields - remember the tilde NL at the
top:
~
DK:SCRIPT
SX:/fip/local/fff UP
DG:scriptgroup
where
DK is fixed as 'SCRIPT'
SX is the full path and filename of the script with any parameters;
it can be a FipSeq string. You must be able to run the script
or program from the logon which 'ipsvrd' has been started.
DG is the normal group which determines which systems to run the script.
Example
BALANCE
group:fiptest host:fip1 ignore-localhost nofiphdr:
group:fiptest host:fip2 ignore-localhost nofiphdr:
group:fiptest host:fip3 ignore-localhost nofiphdr:
group:fiptest host:fip4 ignore-localhost nofiphdr:
.. and the stub file
~
DG:fiptest
DK:STORE
XX:time.1180132986.pid.11281.seq.1
XX:/fip/bnew/ipvi
XX:at.Fri May 25 23:43:06 2007
SX:TTTT
DN:/fip/spool/xo/TTTT.3
Version Control
;2w38 08dec05 allow zero length files to be copied over
;b-f 24feb06 cater for Resource problems on Sparc (f for 64bit wait3)
;g-h 13sep06 added force-queue:
;i 23sep06 added max-open-retries and bug in offline sometimes sending to more
than one system = DONT USE
;j-l 03oct06 new style DR with up to 10 secondaries PLUS bug in 02i PLUS bug
in single
;m-q 07nov06 added action notes-upload and -a (; p speedy)
;r-u 31mar07 added preserve-filename: and log-line: and timing-stats
;v9 31may07 redid modes/permissions and added overwrite and -U
;w4 31jul08 added -E to ignore-external-data ;1 19may09 added ZBALANCE-ID ; 2
8jun09 added ZBALANCE_STUB
;4 parse doneque ;5-6 added ZBALANCE-ZAP-INPUT ;7 round-robin-mode added ;9
added force-name
;10-12 added file_trace and STATTIME
;13-16 4oct17 added parallel run -N and COPY and MOVE ;17 bugette with
-R=//run4remote
;18 added DM/message on remote
;19-20 12oct18 bugette with APPEND
;21-24 31aug19 added SSL ;25-27 minor
;28 5feb21 tuning sr_single ;29 redid SendBal - BAD !! ;30-32 fix for 29
;33 28jul23 sigaction NOT signal ;ab x/balance not getting zapped
;34 22jun24 added ignore-links (ignore DV and DD)
;35 3jul24 more definsive code to close socket which might have been left
dangling after error
;36-37 14aug24 bugette: chk_one, rets blatted if after error; leading to too
many open files plus check sock >= 0
;38 4nov24 abort if SX file does NOT exist and send stub file to woops
;001z5 19may00 redid offline a bit
;b 05jun00 added LINK and DV
;c 14nov00 added DN for name on remote
;d/e 25apr01 allow data-in-file as a default if no SX -this is a change!
;f/g/h 02jan02 added failover: and bugette for FipHdr on File and no SX
;h/i 22mar02 SX and DN are now parsed so that # should be encoded as \043
;j 25jul02 redid Timeout routine
;k 28jul02 buggette in WINNT offline
;l/m/n 09aug02 added open-timeout
;o 28aug02 better handling of 4xx errors - send to woops
;p 28aug02 added retry-timeouts.
;q-s 14jul03 added APPEND as an Action
;t 17sep04 speedy
;u-y 25feb05 remote parallel sends
;z5 09aug05 added -I increment Ports and -O overwrite offline files
(copyright) 2024 and previous years FingerPost Ltd.