ipbalan

ipbalan'ce

This program is used to balance data files on Fip systems. It sends files so
marked to IPSVRD or equivalent on other systems

It can either check the FipHdr for information where to send or can be forced
to send all data to a group/single box.

If for some reason the remote box is down, 'ipbalan' will move the files to an
offline queue and resend when that host is restored.

The FipHdr of the file is checked and the following headers are used :
    DG - name of the group      - mandatory if not running 'force-group'
    SX - full pathname of data file on both source and remote systems
            If there is NO SX field, the data is assumed to follow
            the FipHdr in the file or there is NO data.
    DN - full pathname of data file on the remote system
            This overrides the SX FipHdr field for the REMOTE only.
    DK - action on the remote   - optional, defaults to STORE
            'store'     - Store this file
            'append'    - Append data to this file
            'link'      - Link this file - unix only
            'delete'    - Delete this file
            'script'    - run this script
            'ignore'    - just this one
            'notes-upload'   - upload to notes
These  are optional and only available on Unix
    DD - if 'store' or 'link', link the new file TO a this name
    DV - if 'link', link the file FROM this file. If the file is NOT found
        and there is data attached to the incoming XML, it is used instead.

As the '#' is an end-of-fip-hdr marker, SX and DN are parsed so that # should
be encoded as the 4 printable chrs '\043' (otherwise SX and DN will be
truncated to end on the first '#').
    eg  DN:/fip/spool/testdata/\SN
    or  DN:/fip/spool/\DQ/\043SN:\SN\043DQ:endfolder
        If DQ is 'pdf' and SN is 'abc123' this gives
            DN:/fip/spool/pdf/#SN:abc123#DQ:endfolder

if the FipHdr field ZBALANCE-DATA exists, the SX field is used for the file
name on the remote and the Data following the FipHdr is used.

if the FipHdr field ZBALANCE-STUB exists as well as an SX FipHdr field (with an
external filename), then ONLY the external file is sent. Normally the two are
merged unless the 'nofiphdr:' subkeyword is specified for that 'group:(server)'

If the file cannot be sent, it is moved to the offline queue and the FipHdr
field 'CY' used to denote the hostname of the failed system.

The offline folder is scanned once a minute to see what needs to be retried.

If the FipHdr field ZBALANCE-ID exists and the remote system is offline, then
only a single copy of the file to be sent will be saved - normally this
mechanism is used for balancing skipfiles or other standing files. When the
remote system reappears, only a single copy of the file will be enough to bring
it up-to-date.

If the FipHdr field ZBALANCE-ZAP-INPUT:(filename) exists, the file pointed to
will be zapped after the data has been sent (successfully). The file MUST be in
/fip/data/.. /fip/spool/... or in /fip/x starting 'TMP...'

The parameter file in tables/sys, normally called BALANCE has the following
parameters :
    ; comment

    group:(name)    host:(host name) port:1234  fippath:/data4/fip notjustfip:
        name is the name of this group - there can be one or many lines
            all with the same group name.
        hostname is a valid IP address or name
        port if NOT the default which is 9140 for 'ipsvrd'
        fippath default is /fip
            only need if NOT /fip
        notjustfip:
            default is just under /fip
        nolog:      do NOT log the item
        nofiphdr:   Do NOT send the FipHdr
            Normally we send the FipHdr as well as data.
            This really refers to the incoming file in
            spool/2balance and NOT any external file.
            So if the data follows the FipHdr (ie there is NO
            external file - SX:), this flag will send the data only
            and NO FipHdr.
            if there is an external file and it has a FipHdr, this
            is considered part of the data anyway and is NEVER
            stripped off for transmission. So the complete external
            file is ALWAYS sent.
            So if you have an external file which has a FipHdr,
            always specify this option - eg for FipRemotes
        noxmlhdr:   The file already contains FipAgent XML so we
                do NOT need to send one.
        notfipagent:
            NO= none, YES=fipagent, MAYBE= just text
        nowait
            Do NOT wait for the response from the other end
            There may be someone who needs this but we have not
            worked out why !
        round-robin-mode:
        failover-mode:
            Normally 'ipbalan' sends to all servers in the Group. These flags denote
only one copy needs to be sent to only one in the group.
            So instead of one file sent to ALL systems in the group, a file to ONE of
the group  :
                - for failover, the first one specified if it is live, second if not, etc.
                - for round-robin, tripping from one server to the next
            In both cases, you will need to track that each remote server is currently
live (or not) using tables/sys/REDUN and ipredun/ipxnet to monitor them

        ignore-localhost:
            For this group, do NOT send if the remote host
            is actually the local machine.
        force-queue: (queue or folder on remote system)
            For STORE, all files are to be sent to this folder on the remote system
            The default is the contents of the DN FipHdr field (or SN/SX)
        newname: (replacement name on the remote system if there is no DN FipHdr
field)
            Otherwise The name on the remote system is (in sequence)
                either  contents of the DN FipHdr field (if there)
                or  contents of SX
                or  contents of SN
        force-newname: (replacement name on the remote system which overrides DN, SX,
SN)

        use-permissions:no/yes
            Use same permissions on the remote as on the source (unix only)
            So if the source file has permissions of RWXRWXRWX (777) - it will be made
so on the remote.
            The default is NO (or YES if -P input switch is set) meaning the remote
permissions will be the same as the logon which started 'ipsvrd'

        overwrite: yes/no
            Overwrite the file at the remote host - default yes unless -U

    unique-name: SYSTEM
        name of a file which will be different on the remote systems
        hostname is automatically suffixed.
            ie SYSTEM -> SYSTEM.fip1

    default-group: (name of a group)
        use this Group if none is specified (same as -d Input switch)

    retry-timeouts:(yes/no)
        if a message to 'ipsvrd' timeout or the link is broken,
        the default is to retry ad-infinitum.
        Use this parameter 'retry-timeouts:no' to move the file to 'woops'

    allow-data-with-script: (yes/no)
        normally SCRIPT does not have any data sent; use this to do so.
        The data is put into a Temporary file which the script can use as FipHdr SX

    max-open-retries: (number)
        Number of retries to open a connection  - default is 5
    open-timeout:   (seconds)   default: 120
        Wait time in seconds before the remote host should allow a connection
        Reduce this to 10 or so if the remote is just across the computer room !
    timeout: (seconds)      default: 120
        Wait time in seconds before the remote host should ack a file once sent
        If sending large files over very noisy or busy or slow connections, try
        increasing this if you get lots of errors.

The following are only for use with the XML FipAgent and rarely need changing :
    wrapper: xml wrapper string
        eg : wrapper:DEVICE_MESSAGE
        default is FIPAGENT
    data-length: tag containing the length of data following the wrapper
        default is DATALENGTH
    action: Action of the message
        default is REPLY
    message: FipAgent Error Message
        default is MSG
    result: FipAgent result code
        default is RESULT
    device-result: result code from the host process
        default is DEVICE-RESULT
    poll-timeout: no of seconds which a reply should arrive
        default is 20   (serial running only)
    timeout-message: no of seconds before logging a slow transmission
        default is 20   (parallel running only)
    log-line: Extra logging in FipSeq
        default is none

;
;   BALANCE
;
unique-name:SYSTEM
;
;

group:tables    host:fip1   port:9002
group:tables    host:fip2   port:9002   fippath:/data3/fip

group:w4    host:fip1   notjustfip:
group:w4    host:fip2   notjustfip:

Input Parameters are (all optional):
    -1 : (filename) send a single file and stop default: spool
    -a : default action if none specified       default: STORE
    -d : default group if none specified        default: check each file
    -E : ignore any external file (SX: FipHdr field). default: send contents of SX
if present
    -f : force all data to this group or client default: check each file
    -I : (eye) increment the Port number        default: always send to the 9140 (or the
port)
        ipbalan check the IPaddress of its own system and takes the LAST digit and
adds that to 9140
        ie if the IPaddress is 195.13.83.27 then the last digit is 7 and the remote
port -> 9147
        there should be multiple ipsvrd -P914X (where X is 0->n) on each system
    -J : max increment for -I           default: 10
    -l : (el) log each file through         default: no
    -L : log each file in /fip/log/BALAN        default: no
    -o : done queue for raw files after processing  default: deleted
    -O : Overwrite files in Offline with the same name. default: all files are
unique
    -q : queue to poll              default: spool/2balance
    -Q : quiet mode - do not flag any missing files.default: do
    -R : run Remotebalance in parallel      default: no
        The parameter is the group in BALANCE
        ** This option is ONLY for balancing ipremdata files
    -U : make files on the remote server unique and not overwrite
    -w : file wait for files arriving via NFS etc   default: 0 secs
    -W : 10 ths of a second - MUST wait if nothing processed    default: 2 for
2/10ths
    -x : do NOT use the offline folder as ALL   default: do
        traffic must go to this client.
        Use this ONLY when sending to a single remote system.
    -z : Parameter file         default: tables/sys/BALANCE
    -v : print version no and exit

Current maximums
    Groups and Hosts    400
    Unique Filenames    100

Balance can be used to get 'ipsvrd' to run a script on the remote systems :
Create a file with the following FipHdr fields - remember the tilde NL at the
top:
    ~
    DK:SCRIPT
    SX:/fip/local/fff   UP
    DG:scriptgroup
where
    DK is fixed as 'SCRIPT'
    SX is the full path and filename of the script with any parameters;
        it can be a FipSeq string. You must be able to run the script
        or program from the logon which 'ipsvrd' has been started.
    DG is the normal group which determines which systems to run the script.

Example
    BALANCE
        group:fiptest   host:fip1   ignore-localhost nofiphdr:
        group:fiptest   host:fip2   ignore-localhost nofiphdr:
        group:fiptest   host:fip3   ignore-localhost nofiphdr:
        group:fiptest   host:fip4   ignore-localhost nofiphdr:

.. and the stub file
~
DG:fiptest
DK:STORE
XX:time.1180132986.pid.11281.seq.1
XX:/fip/bnew/ipvi
XX:at.Fri May 25 23:43:06 2007

SX:TTTT
DN:/fip/spool/xo

Version Control
;2w12   08dec05 allow zero length files to be copied over
    ;b-f 24feb06 cater for Resource problems on Sparc (f for 64bit wait3)
    ;g-h 13sep06 added force-queue:
    ;i 23sep06 added max-open-retries and bug in offline sometimes sending to more
than one system = DONT USE
    ;j-l 03oct06 new style DR with up to 10 secondaries PLUS bug in 02i PLUS bug
in single
    ;m-q 07nov06 added action notes-upload and -a (; p speedy)
    ;r-u 31mar07 added preserve-filename: and log-line: and timing-stats
    ;v9 31may07 redid modes/permissions and added overwrite and -U
    ;w4 31jul08 added -E to ignore-external-data ;1 19may09 added ZBALANCE-ID ; 2
8jun09 added ZBALANCE_STUB
        ;4 parse doneque ;5-6 added ZBALANCE-ZAP-INPUT ;7 round-robin-mode added ;9
added force-name
        ;10-12 added file_trace and STATTIME
;001z5  19may00 redid offline a bit
    ;b 05jun00 added LINK and DV
    ;c 14nov00 added DN for name on remote
    ;d/e 25apr01 allow data-in-file as a default if no SX -this is a change!
    ;f/g/h 02jan02 added failover: and bugette for FipHdr on File and no SX
    ;h/i 22mar02 SX and DN are now parsed so that # should be encoded as \043
    ;j 25jul02 redid Timeout routine
    ;k 28jul02 buggette in WINNT offline
    ;l/m/n 09aug02 added open-timeout
    ;o 28aug02 better handling of 4xx errors - send to woops
    ;p 28aug02 added retry-timeouts.
    ;q-s 14jul03 added APPEND as an Action
    ;t 17sep04 speedy
    ;u-y 25feb05 remote parallel sends
    ;z5 09aug05 added -I increment Ports and -O overwrite offline files

(copyright) 2017 and previous years FingerPost Ltd.