ipremdata

ipremdata

This program builds the remote database by reading files stuffed into its input
folder.

Each incoming file is compared against the parameter file to see which services
the file should be sent on.

A single file can be in none, one or many services. If none, why are you
sending this data in anyway ? So 'none's are logged. If a 'default-service' is
specified, 'none's are stuffed in that.

A single copy of the file is maintained for all the services and it is this
that is copied/exported.

The Shared Memory structure is also maintained.

The Type of File is checked - but not altered as it assumes 'ipxchg' or
someother program has already cleaned it up.

Services are tested using the 'testforservice' parameters.

The Parameter file is in tables/remote and, by default, is called SERVICES.
The syntax is the normal Fip style :
    ; comment
    service: Define the name of each service.
        service: (name of service)
        There can be up to 1000 services.
        Case is ignored for the names of the 'service'.
        eg  service:MOTOR

        There can be none of several of the following sub-parameters on the service
line:
        max-seqno:999
            max-seqno is maximum allowable      default is 9999
        reset-seqno-at-midnight
            seqno normally goes to max and resets - this forces first file after
midnight to be 0001

    dest: define which services a particular Dest (FipHdr DU) is for.

    test-for-service: or
    testforservice: define which services to put a file by testing FipHdr fields.
        testforservice:(service1,service2,..) (FipHdr)=(test) (FipHdr)#(test)
        There can be one or more services separated by a comma (no spaces).
        There can be one or more tests with can be either equal '=' or not equal '#'.
        For the test a single wildcard '*' can be added at the end.
        To test for a blank field (or a field which does not exist), use double
quotes : XY="" ZZ#""
        eg: testforservice:AFX_SPORTS   SU=afx XC=s* XC#sdd

        Both the FipHdr and the Test fields can be FipSeq .. eg
            ; Check if the source is 'epd'
            ; should be the XA field XA:epd
            ; BUT also XA:/AFP-SX77, so repeat on punctuation
            ; if XA does NOT exist or there is no data, chk SU
            repeat:Q1   XA,,1,#x
            repeat:Q2   XA,,2,#x
            combie:QA   Q1|Q2|SU
            testforservice:epd      QA=epd\$d

        Combination tests can also be made using & for AND and | for OR
        Groups can be used by grouping tests in ()
            testforservice:SPORTS   (SU=AP | SU=REU) & (XC=SP* | XC=RRR)
        Add to the SPORTS service if the source is AP or REU and the Category starts
SP or equals RRR
        Note that to test for a trailing Bracket, use the FipSeq number \051

    Making copies for other feeds
    copy-1-folder:(folder name)
        if the folder name does NOT start with a '/', it is assumed to be under
'/fip/spool'
    copy-1-fiphdr:(Extra FipSeq for new/additional FipHdr fields)
    copy-1-service:(name)   Only copy if file is for this service
    copy-1-external-file:(yes/no)   Use the DataFile in data/remote and do NOT copy
it to the copy folder
                    default is NO (ie the output file is FipHdr AND data)
    There can be up to 199 copy-X-folder/fiphdrs/service/external-file

Balancing other remote servers.
    Usually 'ipremdata' is running on more than one system. This makes it
transparent for clients which system they are connected to. So a single system
failure does not seriously affect delivery, Use 'ipbalan' and 'ipsvrd' to keep
these systems in sync.

    To make a copy of all new files, add the parameter:
        balance-folder:2balance
    where the folder name resides under /fip/spool.

    If the group name for 'ipbalan' is NOT 'fipremote', force the name using
        balance-group:remote2

    Normally only new files are balanced - not any file which has been sent from
an existing remote. (This uses FipHdr field 'R5' to note the System Id of the
first system the file hits)
        balance-all-files:

    An example tables/sys/BALANCE parameter file for remote data is :
        ;
        ;   BALANCE for fipremote files
        ; This will balance files on 3 servers.
        ;
        group:fipremote host:fiprem1    ignore-localhost:
        group:fipremote host:fiprem2    ignore-localhost:
        group:fipremote host:fiprem3    ignore-localhost:
    The SYSTEM file should have two more entries
        bal local   ipbalan
        svrd    local   ipsvrd -l

Other less often used parameters :
    extra-fiphdr: Add more FipHdr fields to all the data files
        You can add any FipSeq for this
        eg  extra-fiphdr:\nFG:small
    add-fiphdr-for-source: Create or Remake FipHdr fields
        add-fiphdr-for-source: (source) (spc) FipSeq strings
        where Source is the contents of the FipHdr field SU
        and FipSeq is any selection of FipHdrs and data.
        eg add-fiphdr-for-source:Aff    HK:\$1\nSH:S\XS:N:\XN:C\XC:T:\QR:J\XK
    add-seqno-for-dest:
        add-seqno-for-dest: (DU) : (fiphdr) : (inc) : (min) : (max)
        dest or DU - a valid entry in the tables/sys/USERS file
        fiphdr  - a two letter FipHdr where the seqno is added
        inc - optional increment value (default is 1)
        min - optional minimum value (default is 1)
        max - optional maximum value (default is 9999)

        There can be up to 20 of these
        The FipHdr is ONLY added to the file that are for that service.

    default-service: (name of service)
        Default in case an incoming file does not match ANY service.
        This must be defined as a 'service' too.
        If you do not use 'default-service', any non-matching file is ignored.
    number: default number system - octal,decimal or hex.
    output-filename: change the output filename
    owner:  Unix only, logon of the owner of the files if not yours
    archive:    Archive the file in log/data
    NewSU:      FipHdr field for source if NOT 'SU'
    wild:       wild string chr for matching if not the default '*'
    singlewild: wild single chr for matching if not the default '?'
    done-queue: Folder to hold the raw input files once processed
                    default is to delete the incoming file.
    before: text to add at the top of the data file.
    after:  text to add at the bottom of the data file.
    filebefore: file to add at the top of the data file.
    fileafter:  file to add at the bottom of the data file.
    script: (Path/Script Parameters all in FipSeq)
        Run a script after the file has been written.
    log: (FipSeq)
        Item log entry if not default.
    max-round-robin: (max number)
    round-robin-fiphdr: (2 letter FipHdr field)
        if this fiphdr is part of a FipSeq string - such as copy-9-folder or -fiphdr
    use-sx:
or  use-external-file:
        if there is an SX FipHdr field with a path to the data file, use that rather
than the data in the input file.
    track-high-priority: (Priority number - eg 1)
        Create a list of High Priority items for ipremsvr to use.
    fiphdr-high-priority: (FipHdr field - eg PR)
        FipHdr to check for Priority.
        fiphdr-high-priority:QR
        combie:QR   PR|XP|WR|W0,3
    fiphdr-services:R9
        add all the services as this fiphdr field (for logging etc)
        default: none
    fiphdr-service-override:RS
        use this FipHdr field to force data to this service
        ie if the file has 'RS:SPORT' - it would be sent to the SPORT service only
and ignore any 'dest' and 'testforfiphdr'
        default: none

    max-single-fiphdr-size: size that a single fiphdr field can be.
        default is 8196-sep2012 (was 2024)
        the minimum is 1024 and maximum is 31000

    add-search: (yes/no)    default:no
    search-metadata: (FipSeq)
    search-binary-data: (FipSeq)

If 'ipremdata' is being used against a database, the following parameters are :

Keywords for the parameter file are :
    ; comment line
    dbuser: database user name
        user:sa
    dbpassword: database passwd
        passwd:sports
    dbserver: database server name
        server:JERRYORA
    dbdatabase: name of database    (NOT Oracle)
        database:fino

Plus the usual suspects for FipSeq
    fixed:
    partial:
    combie:
    option:
    repeat:
    style:
    replace:
    newdate:
    unique:
    valid:
    sun:
    filter:
    perl:

    \$1 is the first line of text
    \$2 is the second line of text
        ..
    \$9 is the ninth line of text

Input switches (all optional) :
    -l : do not log files       default: log all
    -k : Shared Memory key      default: 9009
    -o : Output path name       default: /fip/data/remote
        If this does not start with a '/', it is assumed
        to be under /fip/spool.
    -q : queue to scan      default: 2remote
    -t : sleep time betwix scans    default: 1 sec
    -z : default parameter file default: tables/remote/SERVER
    -v : print version number and exit

...................................
FipHdrs used

    R0 - one client
    R1 - date YYYYMMDD
    R2 - hour HH
    R3 - seqno
    R4 - uid YYYMMDDHHseqno
    R5 - sysid (if there - check if it is THIS host)
    R6 - TEXT or mimetype
    RX - not really used- holds SX in fip/data/..
    RS - service FipHdr field fiphdr-service-code:RS

...................................
Note : Shared Memory Key by default is 9009 (unix) and FipRemoteServer (winnt)

On Unix, use ipcs and ipcrm to wipe if you REALLY need

[fip@rock src]$ ipcs
------ Shared Memory Segments --------
key shmid   owner   perms   bytes   nattch  status
0x0000f1ca 0      fip   666 1536    0
0x0000eed6 32769    fip 666 1536    0
0x00002331 65538    fip 666 12040   0

------ Semaphore Arrays --------
key semid   owner   perms   nsems

------ Message Queues --------
key msqid   owner   perms   used-bytes  messages

[fip@rock src]$ ipcrm -m 65538
or
ipcrm -M 9009
(which is 0x2331)

-- Maintenance

Nightly maintenance script, zapfiplog or zapfiplog.cmd, should cleanup the
following folders after an appropiate time (change the 30 to the number of days
required)
    /fip/bin/ipdelque -q/fip/fix/remote/clients -i1 -m30
    /fip/bin/ipdelque -q/fip/fix/remote/services -i1 -m30
    /fip/bin/ipdelque -q/fip/fix/remote/search -i1 -m30
    /fip/bin/ipdelque -q/fip/data/remote -i1 -Q -A 30

Version Control
;2a14   26apr10 added search and maxCopies->200 and RS
    ;4 24jun10 added copy-X-external-file
    ;5 30apr14 added file-trace
    ;6-7 added endofday to top to stop remsvr whinging
    ;8 16feb15 added use-sx
    ;9 14apr15 added -C case SENSitive compares and copy-X-xchg
    ;10-11 26may15 added test-for-multiple-services
    ;12-13 9oct17 added max-single-fiphdr-size
    ;14 27feb19 added round-robin
;001w3  12mar02 database version finalised
    ;a WINNT flush SHM
    ;b-e 22nov03 redid Services lookup
    ;f 29jul04 added 'track-high-priority:'
    ;g 17sep04 speedy
    ;h 09oct04 timing-stats
    ;j-m 30oct04 added CLIENT_FEEDS file and testforclient
    ;n-o 17jan05 tuning end of day
    ;p 12dec06 added add-seqno-for-dest:
    ;q-s 01feb07 added copy-X-service and able to run 2 remdatas
    ;t-w3 01mar07 balance is now speedy
;000q   25jan00 oryy vershum from ipw4 04g
    ;c 30jun00 added addfiphdr-for-source
    ;h-19jul00 added repeats
    ;i-09aug00 added logging and bugette in hdr
    ;j/k-24apr01 make sure there are no duplicates in the SERVICES file
    ;m-26apr01 added balance better
    ;n/o/p/q 27jun01 added WINNT

(copyright) 2024 and previous years FingerPost Ltd.