ipstatd

ipstatd

This is the daemon for the statistics database.

It uses a parameter file is used to route and process incoming files. This
parameter is normally the name of the database to update which is normally the
contents of the FipHdr field DB and defaults to tables/stat/DATABASE.

It first uses a selection table to decide what the type of data this really is.
As the list is top down, only the first valid selection is processed.

IPSTATD then uses the 'type' to run a sequence of programs predefined in the
parameter file.

The syntax for the DATABASE file is :
    ; comment
    ; the following is a selection line
    (hdr field) = string [opt (tab) & (hdr) = string ...] (tab) >typename (nl)
    type: (typename)    (program to run)
    trace: (typename)
    test: (typename)

To describe the Selection syntax in detail :
    (hdr field) = string [opt (tab) & (hdr) = string ...] (tab) >typename (nl)
    Each selection is on a single line. If necessary, multiple conditions
        can be specified with the '&' to 'and' them.
    The operation equal, '=', can also be NOT equal '!='.
    Source Header fields (in SH) are preceeded by X, ie XC for category.
    A '*' is used a wild card string; a '?' is single wild card chr. To
        search for a string/chr embedded somewhere in a field, uses a '*'
        before and after.
    If embedded spaces are needed in the string-to-be-searched, use an '*'.
    Note that the search string is case_insensitive.
    Both the selection file and the main file are scanned completely, so
        that one file may be sent to none, one or several destinations
        according to the same or different criteriae.

For the 'type' parameter :
    - '$i' refers to the input file name (Note \$i is still the FIP
        System Variable 'month')
    - all queues and files are assumed to be under /fip/spool
    - Never assume however that the path environment has been setup, so we advise
        you specify full pathnames for the programs.
    - all 'type' lines MUST precede the selection - ie be above.
    - FIP System variables and Header fields can be accessed.
    - there can be one or many or very many type lines.
    - any program can be run
    - if a script/program returns an error, it is logged in the Item log and
        further processing stops.
If a 'type' exists for a typename, ipstatd will NOT run ipformat but will run
what is specified - which may be ipformat of course.

The 'trace' parameter is used for setup, tuning and testing a new type. All it
does is tell IPSTATD to log each line in the Item log. EG:
    trace:shares
Trace MUST always be specified in the DATABASE file BEFORE the types (ie on a
line nearer the top of the file) and all types must be before the selection
lines.

The 'test' parameter does the same as 'trace' BUT NONE of the programs are
actually run. This allows you just to check syntax etc. EG:
    test:racecards
Test MUST always be specified in the DATABASE file BEFORE the types (ie on a
line nearer the top of the file) and all types must be before the selection
lines.

Example of a sequence of programs for a 'type'. Here, if the file starts
'borsen', IPFORMAT is strted twice (sequencially, serially - ie NOT at the same
time) using two parameter files in tables/form/text : WKENDSHARES and
DAILYSHARES .

; shares : for both weekday and weekend
type:shares      ipformat -p dailyshares -i $i
type:shares      ipformat -p wkendshares -i $i

; Selection for type called 'shares'
        SN=borsen*       > shares

    test:geges
Going into the MUI, ip, and doing a 'l' to list the log (or 'm' to more) gives:
Sat Mar  4 11:52:30 ipstatd !i : Incoming File :  geges :   : geges
Sat Mar  4 11:52:30 ipstatd !f : Test/NotRun : /fip/bin/ipformat -p geges -i
/fip/spool/form/geges -D -S NAGS
Sat Mar  4 11:52:30 ipstatd !f : Test/NotRun : /fip/bin/ipxchg -1 formsave/NAGS
-D geges -F -o formsave
Sat Mar  4 11:52:30 ipstatd !f : Test/NotRun : /bin/sort +0 -3 -o
formsave/NAGS.done formsave/NAGS
Sat Mar  4 11:52:30 ipstatd !f : Test/NotRun : /bin/mv formsave/NAGS.done
2go/#SN:geges#DU:nagsdone

Other Points worth noting (ish) ..
    Break out - If either the input parameter -x or a header field FZBO is
present, the input file is 'broken apart' into blocks, records and fields. The
resultant file is called (dest)_(SN) in spool/formtest where dest is as above
and SN is the filename.

If an incoming file matches none of the tests, it is deleted and an error
logged.

In the selection file, remember to specify long names first. In the following
example, type 'sunrac2' never gets processed as all files will be typped as
'sunrac'

    XK:RAC*     >sunrac
    XK:RACING*  >sunrac2

Input parameters are (all optional) :
    -c : name of a queue into which copies of all incoming files
        are made.               default: no copies
    -f : file creep time                default: 0
    -i : queue to scan              default: spool/2stat
    -l : do NOT log every incoming file/destination default: log
    -n : run the program at reduced priority    default: nice 5
    -p : processing file to use     default: tables/stat/DATABASE
    -s : run files serially (ie one after the other) default: parallel
    -t : scan time of directory         default: 3 secs
    -T : Always trace types. This is the same as the 'trace' parameter
        used for setup, tuning and testing a new type. All it does
        is tell IPSTATD to log each line in the Item log.   def: no
    -x : debugging ON - ALL incoming files will be 'broken out' in formtest
        parameter is 'o'ctal, 'd'ecimal or 'h'ex.   default: off
    -z : calm down time             default: 5 secs
        To attempt to let ipformat finish one type before the next
    -v : display version number and exit.

(copyright) 2024 and previous years FingerPost Ltd.