ipedsys
ipedsys
This program adds headers and trailers to files so they can be used by
A.N.other editorial system - such as a PC or MAC based box.
It first goes and gets its parameter file : tables/edsys/FORMAT or the
parameter file stated in the DF header field. The latter is usually WIRE for
wire traffic, VWIRE, ZWIRE, BOX for mailbox, FORM for data formats traffic
etc..
IPEDSYS then scans its input directory and mangles the files. The destination
queue is the contents of the EQ header field.
The EQ field is normally added to the file at the ipwheel-tables/sys/USERS
stage. Sometimes funny chrs such as Space or Slash are required which will of
course give weirdo results to other programs downstream from ipwheel and
upstream from ipedsys. So a dot, '.', is used as an escape chr. Dot is followed
by an UPPERCASE chr or a '@' then octal 040 is subtracted from it. Hence :
.@ = SPC
.N or .. = .
.O = /
.J = " etc
THIS can be ignored if the 'ignore-escapes:' keyword is in the parameter file.
If the queue name is NOT in the EQ field, it can be replaced with any other
using the 'outque' parameter. eg:
outque:TOP/\SU/nextLevel/\EQ
For a file with an SU of 'AP' and an EQ of 'BIZ' will be put in the queue :
/fip/spool/top/ap/nextlevel/biz
or with a -o /wires input switch (see below)
/wires/top/ap/nextlevel/biz
Note that the whole queuename is forced lowercase.
As the text has already been cleaned up by IPXCHG, IPEDSYS is normally used to
add Fixed and header information at the top and bottom of the file using the
'before:' and 'after' parameters.
Incoming files are checked for FIP header fields :
DF: Format table file default: FORMAT
DS: Supercede this file if it already exists default: no
DM: send a unix mail msg to these Unix logons default: none
Files of the same name in a different queue are normally NOT overwritten - the
filename is altered depending on the 'dup:' keyword.
It uses environment variables :
FIP_EDSYS_defEQ default queue default: junk
FIP_EDSYS_defSN default filename default: new.000
FIP_EDSYS_LINE default line length for \$L def: 80
this can also be changed with the 'line-length' parameter below
FIP_EDSYS_WORD default word length for \$W def: 6
this can also be changed with the 'word-length' parameter below
The Translation parameter file has the following syntax :
; comment line
name: (fipseq for filename)
newname: same as 'name:'
format: (pc, mac, unix, nt, raw, alnum)
ext: (extention to the filename)
number: (dec, oct or hex)
before: (fipseq)
after: (fipseq)
filebefore: (filename)
fileafter: (filename)
binary-filebefore: (filename)
binary-fileafter: (filename)
hdrchgchr: (chr to replace) (replacement chr/string)
hdrzap: zap all chrs between FROM to TO in the "dirline" data
hdrzap:<:> Delete all <P>, <\HTML> etc
Only one hdrzap may be specified.
chgchr: (chr to replace) (replacement chr/string)
dup: ($ or inc or zero) handling duplicates
append:(fipseq) Append the new data to the end of the output file
(use FipSeq to make conditional - only if not 'N' will it APpend)
prepend:(fipseq) Prepend the new data to the end of the output file
(use FipSeq to make conditional - only if not 'N' will it PREpend)
supercede: handling duplicates
dup-rename: handling duplicates
takes: (Field Name:NewName)
size: (size of filename)
abstract: (size of abstract)
outque: (Fip Sequence to replace the Contents of EQ)
outque-failover: (Fip Sequence to replace the Contents of EQ) - see CHKEXISTS
comments below
newEN: (New Hdr field to replace EN)
newEP: (New Hdr field to replace EP)
newEQ: (New Hdr field to replace EQ)
owner: (new owner/logon for these files - generally must be root/sudo to do
this)
group: (change just the group for these files - normally the fip logon will
be a memeber of this group)
script: (name of script plus any optional fipseq)
Run script After processing
There can be up to 35 scripts script1: to script9: and script-a to script-z
respath:(path for resource fork)
restype:(type of resource fork)
resorig:(original template res fork)
resfile: (type and creator of the Mac file)
archive: (name of archive in log/data)
forcename: (upper/lower/nochg)
forceque: (upper/lower/nochg)
log: (log message)
always-log:
never-log:
ignore-escapes:
chkexists: (Full Path/file or queuename)
chkexists-timeout: (time between trying/logging if path is not there)
chkexists-failover: (Full Path/file or queuename of failover system)
createque:
outputdrive: (single WINNT drive letter)
done-queue: (Folder name in FipSeq)
Put the input file into this folder after processing
This overrides the -f (defaultDoneQueue) input switch.
balance-done-queue: (Balance group) Balance the done queue items
nodata:
or no-data:
Do not write any data from the incoming file
(but DO add any before, after, filebefore, fileafter etc)
This parameter has no value - if the parameter exists, then NO data is output
(see 'data: below)
default: add the data from the incoming file.
output-data:(fipSeq resolving to YES or NO)
default: YES - add the data from the incoming file.
eg combie:QA TA,no
output-data:\QA
ignore-zero-length-files: y/n
if the output file is zero length, do NOT create it
default: always create an output file, even if it is empty
use-sx:
or use-external-file:
if there is an SX FipHdr field with a path to the data file, use that rather
than the data in the input file.
preserve-fiphdr:
Do not zap the FipHdr from the out file
extra-fiphdr: (FipSeq)
Add to the FipHdr ONLY if 'preserve-fiphdr:' has been specified too.
create-metadata-file: (filename)
create-metadata-file1: (filename) .. create-metadata-file9: (filename)
plus either
metadata-template: (filename)
metadata-template1: (filename) .. metadata-template9: (filename)
or
metadata-binary: (filename)
metadata-binary1: (filename) .. metadata-binary9: (filename)
In addition to the main output file, create an independent file using
metadata (or binary)from this input file
2 options are a FipSeq template OR a binary file:
The template is a file of FipSeq which is parsed and the output named as in
the 'create-metadata-file'
The binary file is just data and is copied directly with no change
There can be up to 36 metadata files and templates or binary; each with
different templates
For example, a single output can be used to trigger the 2 related outputs 1.
binary of a JPEG and 2. a Companion JSON description file from a template.
metadata-fiphdr:(2 letter fiphdr) can be used to hold the current metadata
file in use
ie metadata-fiphdr:E0 will hold '4' for create-metadata-file4:
Note there is only 1 metadata-fiphdr for all possible create-metafile1-9
preserve-metadata-fiphdr: (yes/no)
preserve-metadata-fiphdr1 .. 9: (yes/no) default: no=strip
yes=prefix (or no=strip) a Fiphdr to this metadata template
metadata-is-fiphdr: (yes/no)
metadata-is-fiphdr1 .. 9: (yes/no) default: no, it is data
is the template extra fiphdr (yes) or data (no)
metadata-xchg: (name of xchg to run on the OUTPUT file) default: none
This can be in FipSeq so it gets resolved at runtime
balance-metadata: (groupName) default: none
balance-metadata1: (groupName) .. balance-metadata9: (groupName)
Balance THIS single metadata file using this GroupName
locale: (locale name) default: whatever the system uses
use-lock-file: (yes/no) default: no
Check, set and clear a lock on the output file (ie add a '.lock' to the
filename)
in the same way that sendmail does. Default is no.
It checks for a file called (outque)/(newname) '.lock'
if not there, it sets/creates one while it is writing and zaps it when
finished.
if there, it waits up to 60 secs in 1sec intervals - then zaps and gets on
with it.
This should really be used on a 2nd instance of ipedsys - or you may find ALL
traffic delayed for up to 60 secs !
fiphdr-script: (name of script plus any optional fipseq)
Run a script to build extra Fiphdr fields BEFORE processing the file
fiphdr-from-remote: (FipHdr, Remote Host and Port strings) - see below
fiphdr-input-filename: (2 letter FipHdr field)
Add the complete Path and filename of the INPUT file as this FipHdr field
eg if the input folder is /fip/spool/2edsys and the filename is
'#SN:tester#DQ:2edsys'
Note '#' are mapped to x9D as it is normally an end of FipHdr field marker.
fiphdr-input-filename:PN
gives PN:/fip/spool/2edsys/^SN:tester^DQ:2edsys
where '^' is used to denote a '#'
fiphdr-input-file-hash:(FipSeq chr)
When changing the filename (which has 'hash' signs) for a script, use this
chr
default is x9D
balance-seqno: Send the Sequence number to this Balance Group (see 'ipbalan')
balance-data: Balance the output file using this Balance Group
balance-delete: Delete the Data on remote systems once sent using this Balance
Group
balance-copy: Balance any 'copy-output; file using this Balance Group
balance-queue: Folder under /fip/spool to leave balance files for a copy of
ipbalan. default 2balance
balance-done-queue: (Balance group) Balance the done queue items
add-scoop-hdr:
scoop-name:
scoop-priority:
scoop-category:
round-robin: (number)
Send to output queues ending with a digit up to this number
default none
ie if round-robin:3 and outque:/data/today
The first file through will be put in a folder /data/today1, the 2nd in
/data/today2 etc
word-length: (number)
line-length: (number)
line length for \$L default is 80 or the contents
of environment variable FIP_EDSYS_LINE
word length for \$W default is 6 or the contents
of environment variable FIP_EDSYS_WORD
max-fiphdr-size: (size)
Normally this limits the size of a single FipHdr field to 1023.
use this if you have some long FipHdr fields you need to preserve.
The max is currently 30000 - but use sparingly as the max size of all Fiphdr
is 64k !
hash-key-ignore-top: (number)
Number of lines at the top of the file to ignore BEFORE starting the hash
Use this if a date/time is on the first line or so.
default: none
hash-key-start: (FipSeq)
Start generating the hash-key (\%x) AFTER this character/string.
default:none - ie no start string meaning ALL data from the start.
hash-key-stop: (FipSeq)
Stop generating the hash-key (\%x) on this character/string.
default:none - ie no end string meaning data continues to the end
; stop hashing at the ^C/end-of-text as the datetime is ALWAYS different
hash-key-stop:\003
hash-key-ignore-xml:(yes/no) default is no, ie all data (up to 32k) is used
for the hash
set to 'yes' to ignore any xml tags when hashing
hash-key-data: (FipSeq)
Use this string to generate the hash
default use the data in the file
eg hash-key-data:\XS-\XN-\XC-\XP-\XK
dirname: Directory queue and name
if this starts with a '/' it is a hard path if not it is under the normal
output path.
The *2 version allows a 2nd directory to be generated
dirline: FipSeq for a Directory line
\SN\t\SU\t\$h:\$n\t\XK
if this is missing, no directory will be created/maintained.
dirsize: Number of Text Chrs to store as part of a directory line
dirsize:200
default is 200, specify -1 for no text, 0 for all text.
diritems: No of lines of Items or entries.
diritems:100 for 100 lines
use whatever number is in FIP Header field QR ie QR:99 -> 99
diritems:\QR
default is 1000 lines; zero signifies all lines
dirchgchr: Exchange this chr from all Fip Hdr fields BEFORE
including the header field in a dirline.
Syntax is dirchgchr: (original chr) (replaced chr)
Note there are no spaces/tabs between the two chrs.
If no replacement is specified, a Space is used.
dirchgchr:\140\047
dirchgchr:Aa
All 256 chrs can be stripped.
dirzap: zap all chrs between FROM to TO in the "dirline" data
dirzap:<:> Delete all <P>, <\HTML> etc
Only one dirzap may be specified.
Any added data with "dirbefore" etc is left as is.
dirbefore: (fipseq)
dirafter: (fipseq)
dirfilbef: (filename)
dirfilaft: (filename)
dirtotal: (digit)
tracker-script: Optional script for tracking purposes (fullpath to script
name)
Note that tracker-script is for external logging-it NOTHING to do with
'track-*' which is tracking individual items inside ipedsys.
default: none
copy-output: (new path and filename) (- same as copy-output0)
copy-output1: (new path and filename)
copy-output2: (new path and filename)
..
copy-output9: (new path and filename)
copy-output-a: (new path and filename)
..
copy-output-z: (new path and filename)
Make up to 36 copies of the output file in new paths and/or filenames.
If a resource fork has been specified, it is also added to these.
The paths are under the same root which will be /fip/spool or that defined in
'-o' input switch (or neither if the path starts with a '/').
On WinNT, these folders must be on the same drive.
If they do NOT exist, use 'createque:' to make them.
On Unix platforms, the file will be 'linked' where possible - so it makes
sense to use 'supercede:yes' with this option for that reason.
eg ; make a copy in a day folder
copy-output:\$d-\$m-\$e\$y/\EN
track-uid: (FipSeq) unique key for this 'group' of files
track-seqno: (FipSeq) item/seqno of this file no default
track-item: (FipSeq) item/seqno of this file no default
These two are the actually the same - seqno implies a number, item a code or
type
- a. Tracking by incrementing seqno ....
track-min-seqno: (FipSeq) minimum seqno of the group defaults to 1
track-max-seqno: (FipSeq) maximum seqno of the group default is 9999
track-inc-seqno: (FipSeq) increment value defaults to 1
FipHdr fields to resolve the unique key of a group of files, its sequence
number of this file - and the min and max seqno expected.
The uid may have slashes in to provide subfolders - such as dates.
track-uid:\HY\HM\HD/\SN
There are no defaults except inc is 1 and min is 1 (ie start seqno is 1, and
they increment by 1).
- b. Tracking by total number of files
track-by-total: (number)
2nd method of tracking - rather than by incrementing seqno, just total the
elements and trigger when this number is achieved.
in this case the seqno is just a unique name
ie 3 files with the same track-uid but with a seqno
track-by-total:3
track-uid:baseball.boxes
track-seqno:\MT
And the 3 files might have one of these
MT:one MT:last MT:middle
- c. Tracking by type ...
track-item:\TY
track-trigger-on-item:ABC
Only trigger if this item exists where the FipHdr field TY:ABC
track-item-list:RECAP, PIX*, PREVIEW, TR, PW
This enables a list of valid items
- other tracking options ...
track-save-folder: (Folder name in FipSeq)
Name of the folder which will hold the individual files that need to be
tracked
There is no default
track-cleanup: (yes/no)
When sent should the original files be deleted (yes) or not (no) - default :
no
track-save-fiphdr: (none/current)
Use the fiphdr from the current file (current); no fiphdr at all (no)
(at some point we aim to add ' from the nth file number - but not yet)
default is current - use the fiphdr from the file that triggered the output
eg no FipHdr track-save-fiphdr:no
track-log: (FipSeq)
Log message for tracking
track-extra-fiphdr: (FipSeq)
Extra FipHdr info to add to any released/tracked items. default: none
track-force-missing-text: (Text in FipSeq)
or track-force-missing-file: (Full pathname in FipSeq)
If the output can be forced (by a file with a FipHdr ZTRACK_FORCE_OUTPUT:),
then replace any missings elements/seqnos with this text (or the contents of
file).
default: none/neither
track-process: (Script in Fipseq)
optional script to run on release (and before nay cleanup)
track-action: (merge/separate/ignore)
what action to take when all elements are present.
merge - merge all the elements into a single file
ignore - ignore them ! (normally because there is a track-process:(script)
which has done something.
separate - send them separately
** Separate files are dropped in spool/2go and MUST have a new DU: set by
track-extra-fiphdr:
** If there is no track-extra-fiphdr or track-save-fiphdr:no, the action
flips to ignore !!!
Where ..........
Fipseq is ths normal sequence of fixed text, unix escape chrs, FIP header
fields and Octal chrs).
Note also that a couple of extra builtin FipHdrs exist
\%x - generate a 32bit hash of the first 64k of the data (note that no-data
has a hash of 474e161d85b5ed6f)
\%z - generate a 6 digit sequence number (as opposed to 3 \$s and 4 \$z)
PCHR is normally a '{' and is used for {M0 for roman etc.
FORMAT is the that of the filename
pc chr set - all special chrs mapped to '$' except for dot and dollar
nt chr set - allow extentions of longer than 3 bytes and allow most special
chars
Only use 'format:nt' for files on a NTFS filesystem not if the disk has been
formatted as FAT.
- all non-conforming chrs are forced to '_'
- As NT boxes do funny things to filenames ending with
one of more dots, they are stripped.
unix chr set (no * ! $ spc )
mac chr set (no : etc )
raw - let anything through
alphanumeric - only letters, numbers and underscore
The default is 'format:unix' so that metacharacters, spaces etc are stripped.
You can be lazy as only the first letter of the parameter is used.
so format:ardvark will also give you alphanumeric
NUMBER defaults to octal and is used for the \000 strings.
BEFORE and AFTER are strings to add before and after the text.
FILEBEFORE and FILEAFTER includes data from standing files to add before or
after the text. If the parameter starts with a '/', it is an absolute path, if
not it is under the Output path which is either 'spool' or the contents of the
'-o' flag.
If both BEFORE and FILEBEFORE are stated, the file is done first while if both
AFTER and FILEAFTER are stated, the file is done last.
For all BEFOREs and AFTERs, the data is assumed to be in FipSeq and is parsed
before adding to the file.
Note that FILEBEFORE and FILEAFTER are expected to be in FipSeq while
BINARY-FILEBEFORE/AFTER, the data is not converted at all.
HdrChgChr: Exchange this chr from all Fip Hdr fields BEFORE including any
header fields in a FileBefore, After etc.
Syntax is hdrchgchr: (original chr) (replaced chr)
Note there are no spaces/tabs between the two chrs. If no replacement is
specified, a Space is used.
hdrchgchr:\140\047
hdrchgchr:<(
All 256 chrs can be stripped.
CHGCHR: ditto for a chr in the data part
OWNER allows you to change the owner of output files to this logon. This MUST
be in /etc/passwd and you must have the correct authority for this. default:
that of ipedsys
FIPHDR-SCRIPT is the name of an optional user script that is started BEFORE the
file has been processed. The script must be in the normal path unless you add
the pathname of the script. The actual input pathname are under Fip Hdr field
name E4. The line is parsed for each new file. eg:
FIPHDR-SCRIPT:/fip/local/find_iim.pl \E4 > \E3
\E3 holds the name of a TMP file to create that will be read for the list.
\E4 holds the name of a TMP file that holds the contents of the incoming file
- this is a copy of the incoming file but with a safe filename
SCRIPT is the name of an optional user script that is started AFTER the output
file has been finished. The script must be in the normal path unless you add
the pathname of the script. The actual output path and name of the new file are
under Fip Hdr field names EP and EN. The line is parsed for each new file. eg:
script:wakeup \SU \SN \EP/\EN
for a file called f8899 from UPI that was put in queue 'eve/biz' with name
upi8899.10 would attempt to start :
wakeup UPI f8899 eve/biz/upi8899.10
If you have embedded spaces or other funnies (for unix) remember to quote the
line.
For NT, remember you are calling a DOS batch file. Also use '/' for paths not
'\'. If you wish to specify a '/' as a parameter, use a \0376 (!).
There can be up to 9 scripts script1: to script9: (script: and script1: are the
same)
SIZE makes sure the filename is EQUAL TO or LESS than the number of chrs
specified. This does NOT include the extention, so if you have an extention of
'.t' for a PC filename, make the 'size:8' ie 8 chr filename plus the '.t'.
DUP for duplicate filenames - If 'dup:inc' then the filename is incremented :
if the name is 8 chrs long, the last chr is incremented; if less than 8 chrs,
an 'a' is suffixed to the name. This is the default. Up to 36 duplicates are
permitted.
If 'dup:$' then the filename is changed to a CCM-like extention of $01 etc. Up
to 99 duplicates are permitted.
If 'dup:zero' then a 3 digit extention is added. Up to 999 duplicates are
permitted.
If 'dup:datetime' then any EXISTING file is renamed with a datatime extension.
SUPERCEDE keyword will attempt to overwrite an existing file (same as the DS:
Fip Hdr field)
APPEND keyword will attempt to either write a new file or append at the END of
an existing file (same as the ZAPPEND: Fip Hdr field)
PREPEND keyword will attempt to either write a new file or prepend at the TOP
of an existing file (same as the ZPREPEND: Fip Hdr field)
(use FipSeq to make conditional - only if 'Y' will it APpend)
option:VC TX,,,,q
fixed:Q1 \VCyes\$o
combie:Q2 Q1,no
eg append:\Q2
only if the FipHdr field TX starts with a 'q' , will the data be appended
ABSTRACT is the no of characters to save as first line of text - the FipSeq
\$T.Normally this is 80 characters but it can be any number up to 2048;
TAKES looks at the field name and looks for 'spc', 'Xtakes', '2ndadd' etc. It
puts into a new header field, a sort key which is :
spc : 0
Xtakes : X where X is the number of takes
XYYadd : addX where X is the take number
For Mac resource forks, 4 parameters specify :
- where a template is to use as the basis
- what software is being used - nfshare, ethershare, ushare
- the file type
- the path for the res fork from the data file
eg
respath: .rsrc/
restype:ethershare
resfile:TEXTFIPO
resorig:rsrc_template
Note that, except for restype, none of these parameters are converted so case
is important.
-- 'resfile' is the TYPE (if only 4 chrs long) and CREATOR (if 8 chrs long) of
the file. This is the default if the incoming file is NOT macbinary. If it is,
the Type and Creator from the MacBinary header will be used.
-- The 'restype' can be 'ethershare', 'ushare' or 'nfsshare'.
ARCHIVE will archive a copy in the log marked with this name. This is the
normal fip-style archive where files are reviewable/resendable from the Resend
function in 'IP' the user interface.
FORCENAME : upper will force the file name Uppercase,
lower to lowercase
nochange to leave 'as is'
The default is lowercase.
FORCEQUE : upper will force the queue name Uppercase,
lower to lowercase
nochange to leave 'as is'
The default is lowercase.
Only the queuename in the EQ field or the 'outque' part is forced upper/lower;
not the main path as defined by the '-o' input switch.
newEN/newEP/newEQ : Ipedsys adds these header fields internally for the new
name and path of the output file. However you may already have an EN and/or EP
and wish to use them. Use newEN/newEP to specify another 2 letter header code
which you are NOT using.
LOG: Is a FipSeq string for stuffing a message in the item log, replacing the
'Sending ..' text as in :
Tue Jan 24 17:25:03 ipedsys !o : Sending apu f008624.t
The first part of the message - up to the ':' before Sending - is fixed.
Remember that EP and EN are still valid (see script: above).
ALWAYS-LOG: Always log files using this parameter file even if the default is
set to no logging.
NEVER-LOG: Never log files using this parameter file even if the default is set
to logging.
LOCALE: use a different 'locale' ( look at the man pages for locale)
Most computers are set to US English and never changed but wat if a type of
traffic needs to interpret accented chrs and force them lowercase for filenames
etc ?.
Rather than amend the whole locale fo the computer, use this parameter to
customise any date/time or Chr translations.
The parameter MUST be a valid locale on your system!
To find out what valid locale exist, type 'locale -a' on unix.
Eg ; Set for for Brazil, portugese
locale:pt_BR
CHKEXISTS is a complete pathname of either a standing file or a queue. This is
used to check that NFS drives are actually mounted BEFORE writing new files.
NOTE THAT PROCESSING BLOCKS FOR THIS FILE AND WILL BE RETRIED AT LEAST ONCE A
MINUTE or whatever time has been specified in 'chkexists-timeout'.
eg : chkexists:/data1/nfs/helios/.Desktop
This is the same processing as for the -c Input Switch.
If a 'chkexists-failover' and a 'output-failover' are specified, then, if the
'chkexists' test fails, the 'chkexists-failover' is tested immediately and if
that succeeds, the output file is moved to the 'outque-failover' folder.
This could be used on Win2k servers to test for remote/UNC drives - if the
first is not available, then put the files on the backup.
This also works for the Meta file IF there is NO folder specified for it ie
'create-metadata-file:\EN.meta'
OUTPUTDRIVE (WINNT ONLY) is a single letter output drive for the output file.
The default is the same drive as the rest of Fip.
CREATEQUE - if the output queue does NOT exist, IPEDSYS should attempt to
create it first.
NODATA - Do NOT copy any of the data of the incoming file into the output. This
is normally used where a message or token needs to be passed over to the
editorial system using FipHdr fields and/or standing text in the 'before',
'filebefore' etc parameters.
PRESERVE-FIPHDR is a flag to NOT strip the incoming FipHdr of the file (and add
the filename to the FipHdr too if appropiate).
FIPHDR-FROM-REMOTE get the data for including in a FipHdr field from a remote
TCP server.
The Host and Port MUST be defined.
Any data returned is put in the FipHdr field and can be used at any time.
A number of optional fields are available :
either - wakeup:(FipSeq) used to logon or prompt the remote server for data.
or - mime:(name of template file of FipSeq) to send as a mime header
or neither (default) - do not send anything extra as a header
either - data:(name of template file) to send as data
or - binary:(name of data file) to send as data
only one of either 'wakeup' or 'mime' may be specified
only one of either 'data' or 'binary' may be specified
the size of the data file is in the temp E0 fiphdr field - and can be used
in the mime template
or neither (default) - do not send anything extra as data
- timeout:(secs) If timeout is > zero then it is the number of seconds to
wait. Default timeout is the normal TCP 2 mins.
Returned data options :
- length: (max length of data)
- process: (flags)
m - strip any MimeHeader - default is the leave any mime header that is
returned
x - strip all xml tags (and attributes), leaving just tag data - default is
all data
- file:(filename in Seq) Save the result in this filename - default none
file:/fip/data/edsys/remote/\$e\$y\$i\$d_lookup
- save:(data in FipSeq) Save this data in the above file - default none
save:\SN|\AB|\DI\n
If an entry containing the new FipHdr info - and probably some metadata from
the file too - then both file: and save: are needed
Syntax:
fiphdr-from-remote:(FipHdr field) host:(hostname or IPaddress)
port:(portNumber on Remote) wakeup:(in FipSeq) timeout:(secs)
Eg: fiphdr-from-remote:AB host:duid-server port:32011 wakeup:\SU
The DIR* keywords are for creating and maintaining a directory or summary file.
Hopefully they are self-explanatory :
dirname - the name and path of the file holding the directory
dirline - actual data for each entry line(s)
dirsize - number of entries
dirbefore - string to add before the directory
dirafter - string to add after the directory
dirfilbef - file whose contents to add before the directory
dirfilaft - file whose contents to add after the directory
dirtotal - number of directory files - see below
If any dirbefores or dirafters, text or file are used, the raw data is held in
a tmp file which is defaults to /fip/fix/EDSYSDIR__(dirname). When a new file
arrives, an extra line is added to the top of the tmp directory, the
dirbef/diraft is added and the new directory replaces the old.
If both dirbefores are stated, the file is done first while if both dirafters
are stated, the file is done last.
In all cases or dirbefore or dirafter, the data is assumed to be in FipSeq and
is parsed before adding to the directory.
If the either file parameter starts with a '/', it is an absolute path, if not
it is under the Output path which is either 'spool' or the contents of the '-o'
flag.
DIRTOTAL allows you to specify more than one directory (up to 10). If
'dirtotal' is specified and the number is greater than 1, then 'dirname',
'dirsize', etc have more than one entry separated by a PIPE '|'.
Alll 'dirXXX' keywords can be split EXCEPT FOR 'dirline', 'dirbefore' and
'dirafter' which can have only one.
Example :
dirtotal:3
dirname:dirbig|dirmedium|dirsmall
diritems:100|350|500
dirsize:1000|500|100
dirfilbef:top|topTicker|topsmall
dirbefore:One line at the top for ALL Directories.....\n
dirafter:One line at the bottom for ALL Directories.....\n
dirline:"\$h:\$m \$d-\$m-\$e\$y","\SU","\SN","\XK","\$1"\n
Take care NOT to embed any spaces in the 'dirname' keyword before and after the
PIPE as they will be considered part of the filename.
Actually for 'diritems' and 'dirsize' any punctuation can be taken as the
separator (except '\' or '$') while for all others a PIPE is required - if you
need to enter a PIPE as part of the data, use '\174'.
Note also that if you specify 'dirtotal', you MUST specify all the 'dirfilbef'
and 'dirfilaft' and 'dirname' for each directory - the default is nothing !
Note that 'dirbeffile' and 'diraftfile' file names are either specified in full
(ie starting with a '/') or under the output queue. For example :
if ipedsys is running in the SYSTEM file as ipedsys -o /dump
and the parameter file has dirfilbef:lovelypix/DirFileHeader
then the Standing file is in /dump/lovelypix/DirFileHeader
The default 'dirline' is
dirline:(filename) \XP \XC (time and date) \XK\n
which will pick up the Priority, Category and Keyword of a normal wire file.
Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).
They are normally specified :
fixed:QZ 1234543
partial:QT ST,3,2,U,<,>
combie:QY ep|na,(0000000)a
option:QE ep,11,7,s
repeat:QK XK,-,3
or repeat:QP PK,,4,#X
style:QS XN,%.03d
Note that EDSYS allows several more FipSeqs :
\$c is the number of bytes or characters in the file
\$w is the number of words (no of chrs divided by Word length which
can be altered by the env variable FIP_EDSYS_WORD)
\$l is the number of lines (no of chrs divided by Line length which
can be altered by the env variable FIP_EDSYS_LINE)
\$t is the first 80 chrs of text
\$1 is the first line of text
\$2 is the second line of text
..
\$9 is the ninth line of text
\%z is a 6 digit seqno (\$z is 5 digits and \$s is 4)
Window mapped drives
--------------------
There is a 'wrinkle' for WinNT/2k Output drives in that 'ipedsys' is started
normally from the service 'ipntsvce' (or ipsvce) which is started under the
logon of 'localsystem'.
If you have any mapped drives on your box they MUST have been pre-mounted and
visible by this logon BEFORE ipedsys starts up.
Otherwise you will get either permission-demied or drive-not-found. You will
probably want to create a new logon (or use an extsing one) which has these
drives mounted first.
Pls read the Microsoft Help pages for how to do this. In fact for 2003 MSFT
recommend you use UNC instead - see below.
WinNT/2k remote drives (also called UNC) may be specified as :
(note the double \\ for each \
and the 'ignore-escapes to ignore the dots in the IP address or hostname being
escaped):
outque:\\\\195.13.83.99\\f$\\saves
or outque://195.13.83.99/d$/saves
ignore-escapes:
The use of fwd slashes '/' rather than '\' make the string much easier to read
!!
Use 'chkexists' to make sure the drive is available first
chkexists://195.13.83.99/d$/.DSstore
TRACKING
--------
When a range of independent files need to be collected and sent once the last
item arrives - either as a single file or en-masse as separate files - use the
'track*' keywords in a parameter file and get each incoming file to use that.
For tracking, there are three methods
-- 1. Sequential elements - at least these 4 parameters must be specified :
track-uid, track-seqno, track-max-seqno and track-save-folder
eg track-uid:\AI
track-seqno:\AQ
track-max-seqno:\AX
track-save-folder:/fip/data/track
The incoming file must have a unique ID and a Sequence number in the collection
(ie 1st, 2nd,,, 30th).
-- 2. trigger when a certain number of files have arrived as specified by
track-by-total:
eg track-uid:\AI
track-seqno:\AQ
track-by-total:22
track-save-folder:/fip/data/track
-- 3. trigger by a certain file - using the seqno to define which one :
TK:BASEBALL_MLB_RED+SOCKS_YANKEES
TY:RECAP
; create tracker uid or key - TK is unique competition code
track-uid:\TK
; track item or type
track-item:\TY
; need an item (RECAP) to trigger an output !
track-trigger-on-item:RECAP
; List - output order - of items - note PIX is '*' for multiples.
track-item-list:RECAP, PIX*, PREVIEW, TR, PW
track-save-folder:/fip/data/sports
In this case, trigger when you get sequence 1 and there are at least 2 items
(max-seqno). Trigger updates if new files arrive.
Note there can be seqno > 2 (or multiple entries) but, while they will be
included in the output, they are ignored for the trigger.
These FipHdrs can be combies, repeats etc or just a number :
eg if the Seqno is 3rd part of the filename :
SN:NAME123-20210229-9981443-HiHoSilver.fip
use
repeat:R1 SN,-,1
repeat:R2 SN,-,2
repeat:R3 SN,-,3
; Uid is the first 2 parts of the filename
track-uid:\R1-\R2
; Seqno is based on the 3rd part and defaults of 999
combie:Q2 R3,999
track-seqno:\Q2
; max element number is 999
track-max-seqno:999
; add the date (2nd part of the name) to the tracking folder
track-save-folder:/fip/data/track/\R2
Various sub-parameters can be used to play with the min and max and in Seqno -
they start at 10 and jump in 20 increments to 210.
Sometimes you need to output SOMETHING when NOT everything is ready - due to
time : elections, or network/system problems : sports results, stoxs, etc
To kick out the existing set, send a file with an extra FipHdr field
ZTRACK_FORCE_OUTPUT:
Optionally it can either be a valid file with UID and Seqno (and data) for the
tracking ..
or these can/should be missing if the data of the file is meaningless
By default, the data files are not 'cleanedup' (deleted) after a Force even
though you might have set 'track-cleaup:yes'
However an optional FipHdr ZTRACK_CLEANUP: (yes/no) can be used to cleanup the
data afterwards (or not if set to 'no')
Input switches (all optional) :
-1 : filename for a single shot default: spooled
-9 : do NOT run in Speedy mode default: no
-c : check this queue or file exists before writing files (for NFS and other
mounted queues - see CHKEXISTS above)
default : no
-d : Output Drive (WINNT only) default: drive with Fip on
This is overridden by the 'outputdrive' keyword.
see note above for output drives and permissions.
-f : done queue for the input file default: file is deleted
This can be overridden by the DONE-QUEUE parameter
-i or -q : queue to scan default: 2edsys
-l : log all files default: do NOT log
-m : file mask - input to umask for file creation. Pls remember this is
input as a DECIMAL number while access is normally an
octal ie -m 420 = 0644. default: 0 for rw-rw-rw- access
-o : Output path name default: /fip/spool
ie the path preceding the EQ field
-q or -i : queue to scan default: 2edsys
-Q : keep quiet if the queue for the incoming file does not exist
or there are two many duplicates. default:no
-R ; default round robin maximum default is none
see also parameter 'round-robin:'
-t : sleep time betwix scans default: 1 sec
-u : default owner for ALL files. default: that of ipedsys
This may be overridden by the 'owner' parameter.
-W : 10 ths of a second - MUST wait if nothing processed default: 0
-X : ignore (delete) files in error default: send to woops queue
-x or -z : default format file in tables/edsys default: FORMAT
note these two -x and -z are the same
-Z : force all traffic to use the default parameter file
(ie either FORMAT or the '-z' switch) default: use DF
-v : print version number and exit
Version Control
;25m74 18jan05 added round-robin: and -R
;a 16feb05 bugette in remote-fiphdr
;b 07apr05 allow up to 9 scripts
;c-d 01jul05 cleanedup UNC-Winremote servers //server/mount/path
;e-j4 7apr06 bugette with metadata-file and copy (;i use-lock-file added)
;k8 17sep07 added repair-xml-output and 10 metadata files
;2 use-sx/use-external-file
;3 31oct07 added ignore-zero-length-files
;5 21nov07 dup now allows up to OS limit of chrs (from 8)
;8 added -1
;l5 01feb09 made Tag structure variable to cope with files > 2million tags
;3-5 added round-robin-fiphdr and tracking
;6 note_balance_action ;7 track-action added ;8 maxdups up to 999 for
inc:zero
;9 minor ;10 added balance-data ;11 added force track with
ZTRACK_FORCE_OUTPUT
;12 added tracker-script ;13-15 no logging if log: resolves to nothing ; 16
parse SX
;17-18 14sep10 added track-by-total and track-action ; 19 added chgchr for
data only
;20-21 wrinkleette in fipSpeedy
;21 added more track-* plus track-item, track-trigger-on-item
;m1-25 28sep11 trackwidget added
;18 19feb12 added hashKeys and extra-fiphdr
;19-24 22mar12 added parse of chkfilo and added
chkfile-failover/outque-failover
;25 added log-level and -L (log-level)
;26-28 18dec12 added fiphdr-from-remote: options mime, data, binary, file and
save
;29 11jul13 added curate to fiphdr-on-remote
;30 30apr14 added file-trace
;31 added output-data:yes/no
;32 better handling of SX in move_text so we can have filebefore/after etc
;33-35 minor mods edsysparse - \$T as last on line
;36 added hash-in-fiphdr ;37 22oct15 added fiphdr-script and done-queue and
-f
;38-40 added balance-metadata ; 40-42 added chggrp/uid
;43 20jul16 added WIN EACCES
;44 29jul16 added 'dup-rename'
;45-47 18sep16 buglette in tracker/items PLUS if there is a track-list, use
that for the order
;48 26jan17 -h added ! ;49-50 added balance-copy and balance-queue ;51-56
strparse balance-*
;57 18dec17 added hash-key-ignore-xml ;58-59 added metadata-fiphdr: ; 60
added metadata-binary ;61 baldonq
;62 7jan18 added parse include:
;63-64 18nov19 allow 36 copy-output+scripts using copy-output-a to z as well
as 0-9
;65 append as parse ;66 hash better ;67 balance-meta is now fipseq too ;68
done-queue better
;69 16jun23 added preserve-metadata-fiphdr:
;70-71 4jul23 added metadata-is-fiphdr:
;72-73 31jul24 added metadata-xchg
;74 21oct24 BUGette with size of SH
;024y 13sep01 added up to 100 ((binary-)file)before/afters
;c 19nov01 added hdrChgChr
;d 28dec01 added locale
;e 02jul02 added metadata file
;f 20aug02 hdrzap to remove <P> etc
;g 15apr03 added fiphdr-from-remote:xx host:xxx port: wakeup:xx
;h 22apr03 added always-log and never-log
;i 24apr03 added chkexists-timeout
;j 11aug03 WINNT - allow outque to be in style \\remsvr\f$
;k 03oct03 timings
;l 09feb04 added word-length and line-length
;m-p 19feb04 tweak for the parameter file.
;q 11may04 added fiphdr-input-filename:
;r-s 04jun04 mod to fiphdr-from-remote:xx wait:(time)
;t-u 01sep04 -9 speedy
;v-z 13dec04 maxFipHdrsize;
;023 25apr01 added Scoop header
;022t 28may99 FMACBIN FipHdr field added.
;b 01sep99 added -Z and -i plus '..' back to '.' bugette
;c/d 09sep99 minor mod to MACBIN
;e 20sep99 parse buffer mod
;f 23sep99 added forcename:alphanumeric
;g/h 01oct99 added append: and ZAPPEND:
;i/j/k 17nov99 added binary-filebefore/after plus mod to NT
;l/m 21jan00 added newEQ to make it simpler
;o 12nov00 NT bugette when BinBefore/After file did not exist
;p 14dec00 bugette when FileBefore/After is bigger than 2 k
;q 10jan01 added preserve-fiphdr
;r/s 20feb01 fipdefs to HDRBUF; better USHARE res forks
;t 12mar01 added copy-output0/9
(copyright) 2024 and previous years FingerPost Ltd.