ipremdata
ipremdata
This program builds the remote database by reading files stuffed into its input
folder.
Each incoming file is compared against the parameter file to see which services
the file should be sent on.
A single file can be in none, one or many services. If none, why are you
sending this data in anyway ? So 'none's are logged. If a 'default-service' is
specified, 'none's are stuffed in that.
A single copy of the file is maintained for all the services and it is this
that is copied/exported.
The Shared Memory structure is also maintained.
The Type of File is checked - but not altered as it assumes 'ipxchg' or
someother program has already cleaned it up.
Services are tested using the 'testforservice' parameters.
The Parameter file is in tables/remote and, by default, is called SERVICES.
The syntax is the normal Fip style :
; comment
service: Define the name of each service.
service: (name of service)
There can be up to 1000 services.
Case is ignored for the names of the 'service'.
eg service:MOTOR
There can be none of several of the following sub-parameters on the service
line:
max-seqno:999
max-seqno is maximum allowable default is 9999
reset-seqno-at-midnight
seqno normally goes to max and resets - this forces first file after
midnight to be 0001
dest: define which services a particular Dest (FipHdr DU) is for.
test-for-service: or
testforservice: define which services to put a file by testing FipHdr fields.
testforservice:(service1,service2,..) (FipHdr)=(test) (FipHdr)#(test)
There can be one or more services separated by a comma (no spaces).
There can be one or more tests with can be either equal '=' or not equal '#'.
For the test a single wildcard '*' can be added at the end.
To test for a blank field (or a field which does not exist), use double
quotes : XY="" ZZ#""
eg: testforservice:AFX_SPORTS SU=afx XC=s* XC#sdd
Both the FipHdr and the Test fields can be FipSeq .. eg
; Check if the source is 'epd'
; should be the XA field XA:epd
; BUT also XA:/AFP-SX77, so repeat on punctuation
; if XA does NOT exist or there is no data, chk SU
repeat:Q1 XA,,1,#x
repeat:Q2 XA,,2,#x
combie:QA Q1|Q2|SU
testforservice:epd QA=epd\$d
Combination tests can also be made using & for AND and | for OR
Groups can be used by grouping tests in ()
testforservice:SPORTS (SU=AP | SU=REU) & (XC=SP* | XC=RRR)
Add to the SPORTS service if the source is AP or REU and the Category starts
SP or equals RRR
Note that to test for a trailing Bracket, use the FipSeq number \051
Making copies for other feeds
copy-1-folder:(folder name)
if the folder name does NOT start with a '/', it is assumed to be under
'/fip/spool'
copy-1-fiphdr:(Extra FipSeq for new/additional FipHdr fields)
copy-1-service:(name) Only copy if file is for this service
copy-1-external-file:(yes/no) Use the DataFile in data/remote and do NOT copy
it to the copy folder
default is NO (ie the output file is FipHdr AND data)
There can be up to 199 copy-X-folder/fiphdrs/service/external-file
Balancing other remote servers.
Usually 'ipremdata' is running on more than one system. This makes it
transparent for clients which system they are connected to. So a single system
failure does not seriously affect delivery, Use 'ipbalan' and 'ipsvrd' to keep
these systems in sync.
To make a copy of all new files, add the parameter:
balance-folder:2balance
where the folder name resides under /fip/spool.
If the group name for 'ipbalan' is NOT 'fipremote', force the name using
balance-group:remote2
Normally only new files are balanced - not any file which has been sent from
an existing remote. (This uses FipHdr field 'R5' to note the System Id of the
first system the file hits)
balance-all-files:
An example tables/sys/BALANCE parameter file for remote data is :
;
; BALANCE for fipremote files
; This will balance files on 3 servers.
;
group:fipremote host:fiprem1 ignore-localhost:
group:fipremote host:fiprem2 ignore-localhost:
group:fipremote host:fiprem3 ignore-localhost:
The SYSTEM file should have two more entries
bal local ipbalan
svrd local ipsvrd -l
Other less often used parameters :
extra-fiphdr: Add more FipHdr fields to all the data files
You can add any FipSeq for this
eg extra-fiphdr:\nFG:small
add-fiphdr-for-source: Create or Remake FipHdr fields
add-fiphdr-for-source: (source) (spc) FipSeq strings
where Source is the contents of the FipHdr field SU
and FipSeq is any selection of FipHdrs and data.
eg add-fiphdr-for-source:Aff HK:\$1\nSH:S\XS:N:\XN:C\XC:T:\QR:J\XK
add-seqno-for-dest:
add-seqno-for-dest: (DU) : (fiphdr) : (inc) : (min) : (max)
dest or DU - a valid entry in the tables/sys/USERS file
fiphdr - a two letter FipHdr where the seqno is added
inc - optional increment value (default is 1)
min - optional minimum value (default is 1)
max - optional maximum value (default is 9999)
There can be up to 20 of these
The FipHdr is ONLY added to the file that are for that service.
default-service: (name of service)
Default in case an incoming file does not match ANY service.
This must be defined as a 'service' too.
If you do not use 'default-service', any non-matching file is ignored.
number: default number system - octal,decimal or hex.
output-filename: change the output filename
owner: Unix only, logon of the owner of the files if not yours
archive: Archive the file in log/data
NewSU: FipHdr field for source if NOT 'SU'
wild: wild string chr for matching if not the default '*'
singlewild: wild single chr for matching if not the default '?'
done-queue: Folder to hold the raw input files once processed
default is to delete the incoming file.
before: text to add at the top of the data file.
after: text to add at the bottom of the data file.
filebefore: file to add at the top of the data file.
fileafter: file to add at the bottom of the data file.
script: (Path/Script Parameters all in FipSeq)
Run a script after the file has been written.
log: (FipSeq)
Item log entry if not default.
max-round-robin: (max number)
round-robin-fiphdr: (2 letter FipHdr field)
if this fiphdr is part of a FipSeq string - such as copy-9-folder or -fiphdr
use-sx:
or use-external-file:
if there is an SX FipHdr field with a path to the data file, use that rather
than the data in the input file.
track-high-priority: (Priority number - eg 1)
Create a list of High Priority items for ipremsvr to use.
fiphdr-high-priority: (FipHdr field - eg PR)
FipHdr to check for Priority.
fiphdr-high-priority:QR
combie:QR PR|XP|WR|W0,3
fiphdr-services:R9
add all the services as this fiphdr field (for logging etc)
default: none
fiphdr-service-override:RS
use this FipHdr field to force data to this service
ie if the file has 'RS:SPORT' - it would be sent to the SPORT service only
and ignore any 'dest' and 'testforfiphdr'
default: none
max-single-fiphdr-size: size that a single fiphdr field can be.
default is 8196-sep2012 (was 2024)
the minimum is 1024 and maximum is 31000
add-search: (yes/no) default:no
search-metadata: (FipSeq)
search-binary-data: (FipSeq)
If 'ipremdata' is being used against a database, the following parameters are :
Keywords for the parameter file are :
; comment line
dbuser: database user name
user:sa
dbpassword: database passwd
passwd:sports
dbserver: database server name
server:JERRYORA
dbdatabase: name of database (NOT Oracle)
database:fino
Plus the usual suspects for FipSeq
fixed:
partial:
combie:
option:
repeat:
style:
replace:
newdate:
unique:
valid:
sun:
filter:
perl:
\$1 is the first line of text
\$2 is the second line of text
..
\$9 is the ninth line of text
Input switches (all optional) :
-l : do not log files default: log all
-k : Shared Memory key default: 9009
-o : Output path name default: /fip/data/remote
If this does not start with a '/', it is assumed
to be under /fip/spool.
-q : queue to scan default: 2remote
-t : sleep time betwix scans default: 1 sec
-z : default parameter file default: tables/remote/SERVER
-v : print version number and exit
...................................
FipHdrs used
R0 - one client
R1 - date YYYYMMDD
R2 - hour HH
R3 - seqno
R4 - uid YYYMMDDHHseqno
R5 - sysid (if there - check if it is THIS host)
R6 - TEXT or mimetype
RX - not really used- holds SX in fip/data/..
RS - service FipHdr field fiphdr-service-code:RS
...................................
Note : Shared Memory Key by default is 9009 (unix) and FipRemoteServer (winnt)
On Unix, use ipcs and ipcrm to wipe if you REALLY need
[fip@rock src]$ ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x0000f1ca 0 fip 666 1536 0
0x0000eed6 32769 fip 666 1536 0
0x00002331 65538 fip 666 12040 0
------ Semaphore Arrays --------
key semid owner perms nsems
------ Message Queues --------
key msqid owner perms used-bytes messages
[fip@rock src]$ ipcrm -m 65538
or
ipcrm -M 9009
(which is 0x2331)
-- Maintenance
Nightly maintenance script, zapfiplog or zapfiplog.cmd, should cleanup the
following folders after an appropiate time (change the 30 to the number of days
required)
/fip/bin/ipdelque -q/fip/fix/remote/clients -i1 -m30
/fip/bin/ipdelque -q/fip/fix/remote/services -i1 -m30
/fip/bin/ipdelque -q/fip/fix/remote/search -i1 -m30
/fip/bin/ipdelque -q/fip/data/remote -i1 -Q -A 30
Version Control
;2a14 26apr10 added search and maxCopies->200 and RS
;4 24jun10 added copy-X-external-file
;5 30apr14 added file-trace
;6-7 added endofday to top to stop remsvr whinging
;8 16feb15 added use-sx
;9 14apr15 added -C case SENSitive compares and copy-X-xchg
;10-11 26may15 added test-for-multiple-services
;12-13 9oct17 added max-single-fiphdr-size
;14 27feb19 added round-robin
;001w3 12mar02 database version finalised
;a WINNT flush SHM
;b-e 22nov03 redid Services lookup
;f 29jul04 added 'track-high-priority:'
;g 17sep04 speedy
;h 09oct04 timing-stats
;j-m 30oct04 added CLIENT_FEEDS file and testforclient
;n-o 17jan05 tuning end of day
;p 12dec06 added add-seqno-for-dest:
;q-s 01feb07 added copy-X-service and able to run 2 remdatas
;t-w3 01mar07 balance is now speedy
;000q 25jan00 oryy vershum from ipw4 04g
;c 30jun00 added addfiphdr-for-source
;h-19jul00 added repeats
;i-09aug00 added logging and bugette in hdr
;j/k-24apr01 make sure there are no duplicates in the SERVICES file
;m-26apr01 added balance better
;n/o/p/q 27jun01 added WINNT
(copyright) 2025 and previous years FingerPost Ltd.