ipremdata This program builds the remote database by reading files stuffed into its input folder. Each incoming file is compared against the parameter file to see which services the file should be sent on. A single file can be in none, one or many services. If none, why are you sending this data in anyway ? So 'none's are logged. If a 'default-service' is specified, 'none's are stuffed in that. A single copy of the file is maintained for all the services and it is this that is copied/exported. The Shared Memory structure is also maintained. The Type of File is checked - but not altered as it assumes 'ipxchg' or someother program has already cleaned it up. Services are tested using the 'testforservice' parameters. The Parameter file is in tables/remote and, by default, is called SERVICES. The syntax is the normal Fip style : ; comment service: Define the name of each service. service: (name of service) There can be up to 1000 services. Case is ignored for the names of the 'service'. eg service:MOTOR There can be none of several of the following sub-parameters on the service line: max-seqno:999 max-seqno is maximum allowable default is 9999 reset-seqno-at-midnight seqno normally goes to max and resets - this forces first file after midnight to be 0001 dest: define which services a particular Dest (FipHdr DU) is for. testforservice: define which services to put a file by testing FipHdr fields. testforservice:(service1,service2,..) (FipHdr)=(test) (FipHdr)#(test) There can be one or more services separated by a comma (no spaces). There can be one or more tests with can be either equal '=' or not equal '#'. For the test a single wildcard '*' can be added at the end. To test for a blank field (or a field which does not exist), use double quotes : XY="" ZZ#"" eg: testforservice:AFX_SPORTS SU=afx XC=s* XC#sdd Both the FipHdr and the Test fields can be FipSeq .. eg ; Check if the source is 'epd' ; should be the XA field XA:epd ; BUT also XA:/AFP-SX77, so repeat on punctuation ; if XA does NOT exist or there is no data, chk SU repeat:Q1 XA,,1,#x repeat:Q2 XA,,2,#x combie:QA Q1|Q2|SU testforservice:epd QA=epd$d Combination tests can also be made using & for AND and | for OR Groups can be used by grouping tests in () testforservice:SPORTS (SU=AP | SU=REU) & (XC=SP* | XC=RRR) Add to the SPORTS service if the source is AP or REU and the Category starts SP or equals RRR Note that to test for a trailing Bracket, use the FipSeq number 51 Making copies for other feeds copy-1-folder:(folder name) if the folder name does NOT start with a '/', it is assumed to be under '/fip/spool' copy-1-fiphdr:(Extra FipSeq for new/additional FipHdr fields) copy-1-service:(name) Only copy if file is for this service copy-1-external-file:(yes/no) Use the DataFlle in data/remote and do NOT copy it to the copy folder default is NO (ie the output file is FipHdr AND data) There can be up to 199 copy-X-folder/fiphdrs/service/external-file Balancing other remote servers. Usually 'ipremdata' is running on more than one system. This makes it transparent for clients which system they are connected to. So a single system failure does not seriously affect delivery, Use 'ipbalan' and 'ipsvrd' to keep these systems in sync. To make a copy of all new files, add the parameter: balance-folder:2balance where the folder name resides under /fip/spool. If the group name for 'ipbalan' is NOT 'fipremote', force the name using balance-group:remote2 Normally only new files are balanced - not any file which has been sent from an existing remote. (This uses FipHdr field 'R5' to note the System Id of the first system the file hits) balance-all-files: An example tables/sys/BALANCE parameter file for remote data is : ; ; BALANCE for fipremote files ; This will balance files on 3 servers. ; group:fipremote host:fiprem1 ignore-localhost: group:fipremote host:fiprem2 ignore-localhost: group:fipremote host:fiprem3 ignore-localhost: The SYSTEM file should have two more entries bal local ipbalan svrd local ipsvrd -l Other less often used parameters : extra-fiphdr: Add more FipHdr fields to all the data files You can add any FipSeq for this eg extra-fiphdr:nFG:small add-fiphdr-for-source: Create or Remake FipHdr fields add-fiphdr-for-source: (source) (spc) FipSeq strings where Source is the contents of the FipHdr field SU and FipSeq is any selection of FipHdrs and data. eg add-fiphdr-for-source:Aff HK:$1nSH:SXS:N:XN:CXC:T:QR:JXK add-seqno-for-dest: add-seqno-for-dest: (DU) : (fiphdr) : (inc) : (min) : (max) dest or DU - a valid entry in the tables/sys/USERS file fiphdr - a two letter FipHdr where the seqno is added inc - optional increment value (default is 1) min - optional minimum value (default is 1) max - optional maximum value (default is 9999) There can be up to 20 of these The FipHdr is ONLY added to the file that are for that service. default-service: (name of service) Default in case an incoming file does not match ANY service. This must be defined as a 'service' too. If you do not use 'default-service', any non-matching file is ignored. number: default number system - octal,decimal or hex. output-filename: change the output filename owner: Unix only, logon of the owner of the files if not yours archive: Archive the file in log/data NewSU: FipHdr field for source if NOT 'SU' wild: wild string chr for matching if not the default '*' singlewild: wild single chr for matching if not the default '?' done-queue: Folder to hold the raw input files once processed default is to delte the incoming file. before: text to add at the top of the data file. after: text to add at the bottom of the data file. filebefore: file to add at the top of the data file. fileafter: file to add at the bottom of the data file. script: (Path/Script Parameters all in FipSeq) Run a script after the file has been written. log: (FipSeq) Item log entry if not default. track-high-priority: (Priority number - eg 1) Create a list of High Priority items for ipremsvr to use. fiphdr-high-priority: (FipHdr field - eg PR) FipHdr to check for Priority. fiphdr-high-priority:QR combie:QR PR|XP|WR|W0,3 fiphdr-services:R9 add all the services as this fiphdr field (for logging etc) default: none fiphdr-service-override:RS use this FipHdr field to force data to this service ie if the file has 'RS:SPORT' - it would be sent to the SPORT service only and ignore any 'dest' and 'testforfiphdr' default: none add-search: (yes/no) default:no search-metadata: (FipSeq) search-binary-data: (FipSeq) If 'ipremdata' is being used against a database, the following parameters are : Keywords for the parameter file are : ; comment line dbuser: database user name user:sa dbpassword: database passwd passwd:sports dbserver: database server name server:JERRYORA dbdatabase: name of database (NOT Oracle) database:fino Plus the usual suspects for FipSeq fixed: partial: combie: option: repeat: style: replace: newdate: $1 is the first line of text $2 is the second line of text .. $9 is the ninth line of text Input switches (all optional) : -l : do not log files default: log all -k : Shared Memory key default: 9009 -o : Output path name default: /fip/data/remote If this does not start with a '/', it is assumed to be under /fip/spool. -q : queue to scan default: 2remote -t : sleep time betwix scans default: 1 sec -z : default parameter file default: tables/remote/SERVER -v : print version number and exit Note : Shared Memory Key by default is 9009 (unix) and FipRemoteServer (winnt) On Unix, use ipcs and ipcrm to wipe if you REALLY need [fip@rock src]$ ipcs ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x0000f1ca 0 fip 666 1536 0 0x0000eed6 32769 fip 666 1536 0 0x00002331 65538 fip 666 12040 0 ------ Semaphore Arrays -------- key semid owner perms nsems ------ Message Queues -------- key msqid owner perms used-bytes messages [fip@rock src]$ ipcrm -m 65538 or ipcrm -M 9009 (which is 0x2331) -- Maintenance Nightly maintenance script, zapfiplog or zapfiplog.cmd, should cleanup the following folders after an appropiate time (change the 30 to the number of days required) /fip/bin/ipdelque -q/fip/fix/remote/clients -i1 -m30 /fip/bin/ipdelque -q/fip/fix/remote/services -i1 -m30 /fip/bin/ipdelque -q/fip/fix/remote/search -i1 -m30 /fip/bin/ipdelque -q/fip/data/remote -i1 -Q -A 30 Version Control ;002a5 26apr10 added search and maxCopies->200 and RS ;a4 24jun10 added copy-X-external-file ;a5 30apr14 added file-trace ;001w3 12mar02 database version finalised ;a WINNT flush SHM ;b-e 22nov03 redid Services lookup ;f 29jul04 added 'track-high-priority:' ;g 17sep04 speedy ;h 09oct04 timing-stats ;j-m 30oct04 added CLIENT_FEEDS file and testforclient ;n-o 17jan05 tuning end of day ;p 12dec06 added add-seqno-for-dest: ;q-s 01feb07 added copy-X-service and able to run 2 remdatas ;t-w3 01mar07 balance is now speedy ;000q 25jan00 oryy vershum from ipw4 04g ;c 30jun00 added addfiphdr-for-source ;h-19jul00 added repeats ;i-09aug00 added logging and bugette in hdr ;j/k-24apr01 make sure there are no duplicates in the SERVICES file ;m-26apr01 added balance better ;n/o/p/q 27jun01 added WINNT (copyright) 2014 and previous years FingerPost Ltd.