iptimer

iptimer

This program kicks off another script/program, waits a few seconds, does it
again... and again... and again

----- Main Parameter file -----

The syntax of the Main Parameter File - by default tables/setup/TIMER (or
TIMER.fip on Winnt)
    ; comment line
    group: (groupname)
    track-status:yes/no
    stagger-starts:1
    client:(name)   day:(MTuWThFSaSu)   time:20:30  mustget:
    client:(name)   date:(UTCdate 20040620) time:20:30  mustget:
    client:(name)   day:(MTuWThFSaSu)   every:30    start:07:30 end:19:00   maxtime:3600

Other (ie non client) keywords
    group - name of the group
        this is just a handle
    track-status:(yes/no)
    stagger-starts:(number)
        This is the gap in 100ths of a second between starts.
        The maximum is 100 for 1 second delay.
        Use this so all clients for a particular time are NOT started at the same
millisecond.
        default: none for zero wait
    trace-starts:(yes/no)
        Write a trace file of programs and parameters started in /fip/log/timer
        default: no
    extra-fiphdr: (FipSeq)
        More FipHdr information which is added to ALL starts.
        This is in addition to the information in 'client:xx ... fiphdr: ...'
    proxy-server: If using a proxy, these are the hostname (or IP address) and
port to aim at.
    proxy-port:
    proxy-logon:
    proxy-handshake:yes/no  Does the proxy need to say hello first ?    default: no
    proxy-is-squid:yes/no   Is the proxy a Squid ?  default: no
        Proxy is only available for webgrab clients. Pls see the
        doc for webwire for further information.
    test-fiphdr: (2 letter FH code)         default: none
        if -t input switch is ON, and there is NO -T XX or test-fiphdr, the default
is QZ
        eg if a client is flagged as test, client runline has this T5:TEST added
            test-fiphdr:T5
            client:hoho type:w template:HOHO_GRAB every:1 test:yes

    throttle-output:(parameter file for iproute)
        The output is to be checked for runaway repeat offenders by passing all data
thru 'iproute'
        This adds an extra FipHdr field 'SR:' to the fiphdr and changes the output
folder to /fip/spool/2brouted.
    webgrab-gap: (no of seconds)
        Gap in seconds between file grabs for webwire
        default is 1 (version 2n11 and after) or 5 if previous

    minimum-every:(no of seconds)
        For those clients with 'every', override the settings with this amount if it
is lower.
        This can also be set with the -E input switch
        default: no minimum (ie 1 second)
        ie if you have a client which is being accessed every 5 secs, a second
        every 10 secs and a third every 60: if minimum-every:30 then the
        first two will be scavenged every 30 seconds and the third unaffected at 60
secs.

    bandwidth-stats:yes/no/client
        Force the worker programs - webwire, ipftp etc - to generate statistics on
duration and data volumes (if they can)
        If the parameter is 'client', individual group_client_program files are
created in log/bandwidth_stats
        Otherwise all information is placed in Hour_group_program files for all
accesses.
        If bandwidth stats are on, then logging is turned OFF for the worker
programs. So they will only log errors.
        default is NO

    skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute
        the skip file when changed (see doc on 'ipbalan') - for ipftp and webwire.
        This is often used where a second system could be used as a redundant server
        if the main system fails. (see also -B input switch)

    default-max-timeout: default maximum time in seconds for every process to run.
        default is 21600 for 6 hours (before version 2m3 this was ignored and there
was no max)
    check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
        Use this when iptimer runs on 2 systems with the same parameter file (ie both
run the same jobs)
        - but only one should be live : an FTP grab or web scavenge for example
        This parameter is used to check if the current host should be getting the
files or not
        ie in the parameter file is
            check-primary-server:timersvr
        and in the DEST_REDUN is
            ; psuedohost    primary secondary
            timersvr    fip1    fip2
        and in the SYSTEM file for both fip1 AND fip2 there is  the same 'iptimer'
line with the same parameter file
        If fip1 is up,  'iptimer' on fip1 will always get while on fip2 it will just
check/loop.
        If fip1 is down 'iptimer' on fip2 will start processing the files.

    times-are-gmt:
    times-are-utc: all start/end/time subparameters are based on GMT not localtime
        Use this if the times need to be constant over the year and not change with
summer time
        default use the System time

    client-time-zone:  set a timezone which affects for ALL clients display time,
daylight saving etc for start: end: etc 
    default-time-zone: on systems which do NOT have a system TZ - used ONLY to
reset after using the client-time-zone
        see man tzset           default: system timezone or the env variable "TZ"
        eg if using a file /usr/share/zoneinfo/, add a colon eg default-time-zone:
:America/New_York

In detail :
- 'client' or 'source' keyword (these are identical):
    client: name of a client or source
        which should match a profile defined for the times and dates
    url:
    host: url to grab or hostname for ftp, nntp etc (url and host are identical)
    port:   port number if not default (webwire, ftp, soap etc)
    match: (FipSeq string)
        (optional and possibly up to 10 may be specified)
    ignore: (FipSeq string)
        (optional and possibly up to 10 may be specified)
    fiphdr: (FipSeq string)
        (optional extra FipHdr information)
        This will add the W3 fiphdr field. To add other fields, use a '#' to
separate:
        fiphdr:abc
            will give a value of 'abc' in the FipHdr W3
        fiphdr:WY:123#WA:online
            will give a (probabaly wrong) value of 'WY:abc' in the FipHdr W3
            plus 'online' in WA
        fiphdr:#WY:123#WA:online
            will give a value of nothing in the FipHdr W3
            plus 'online' in WA and '123' in WY
    test: (yes/no)
        adds the FipHdr field QZ:TEST if test-fiphdr:QZ or -T QZ
        default is NO
    type: (webwire/ipftp/...)
        Type of grabs for the Internet Grabbing  - pls see note 2 below
        default is 'webwire'
    switch: (FipSeq string)
        Optional extra program input switches to use when the program is started
        default is none
    template: (name of template parameter file)
        Mandatory for webwire or ipftp
    logon:
    passwd: (webwire, soap, ftp etc where needed)
        Logon and password for any authentication
        no default
    data: (webwire)
        Do we need the data of the file - data Yes/no
        default: yes
    dest: (all)
        extra destinations (FipHdr DU on output)
        default: no extra
    outque: (webwire, imap, soap, ipspool and ftp)
        Different Output queue under spool - for webwire this could be for a
rpt-offenders check.
        default: spool/2go
    queue: (ipspool and ftp send only)
        Input folder under spool
        default: none !
    ignorekey: (webwire)
        add ignore-key:\W5 to the webwire parameter file.
        default: none
    spare: (all)
        Extra information which is added to the FipHdr \W4
        default: none
    primary-server: (override check-primary-server)
    primary: (override check-primary-server)
        This allows a single client to use a different primary-server
        Default is that defined by check-primary-server - or not set/checked if the
latter is not specified
    script: (path/script options)
        Script name and options for type:x, external script.
        default: none
    maxtime: Maximum time for the job in seconds
        default: 21600 for 6 hours (before version 2m3 - none)
    cwd: (folder name)
        for ftp only, the folder name to 'cd' to

There can be up to 1000 clients in the main parameter file.
either
    date: Actual Date to run this
        Format is UTC style : CCYYMMDD
            ie 20091225     for Christmas Day 2009
        You can use FipSeq for 'date' - but it is resolved ONLY when the parameter
file is changed. So that if you want it to change, stop;/restart iptimer or
touch the parameter file to trigger a new read.
            eg only get on the 15th of each month ... date:\$e\$y\$i15
or
    day:    Day of week to run the job :
            M   Monday
            Tu  Tuesday
            W   Wednesday
            Th  Thursday
            F   Friday
            Sa  Saturday
            Su  Sunday
            X   Every day.
            Z   Every weekday (ie Monday to Friday)
                Case is NOT important.
                Commas or dots (but NOT spaces) may be used to separate.
                Default is every day.
either
    time:   Time of the day on 24 hour clock.   Default is 18:00.
or
    every:  interval in minutes between grabs   Default: none
        every: (mins)   [(optional) start:(starttime) end:(endtime)
        for seconds, add 's' or 'secs' after the number - 45s for 45 secs
        for hours, add 'h' or 'hour' after the number - 6h
        eg  every:30    start:07:30 end:19:00
            every:30secs
        The minimum interval is 15 secs (every:0.15) unless modified with
'minimum-every:' (see above)

    sync:   for 'every', syncronise on this minute of the hour  default: time when
iptimer starts
        eg  every:20    sync:11
            will kick off the item when iptimer is started ..
            and then every 20 mins at 11, 31 and 41 mins past
        The default is at start and a GAP of 20 mins from the end of one grab to the
next
        Specify seconds either with a decimal or tailing 's' : sync:17s sync:0.17
sync:3:24
        Note that the sync resets on the hour - so the first one after the hour MAY
be at a odd, extra time.
eg:
    client:UKcompanies      start:07:20 end:17:45
    client:Forex    day:MTuWThF time:16:30
    client:Forex    day:MTuWThF time:16:40
        Get the Page file tables/setup/FOREX every week day at 16:30 and 16:40

Input options (all optional) :
    -1 : name of a single script/program            default: none
        (same as -z - ie OldStyle; does not use the parameter file at all
    -B : name of Balance Group for skip files       default: none
        This should exist in sys/BALANCE (see ipbalan)
    -c : Name of a single client or source to trigger an instant snap   default:
spooled
        after which the program stops
        The name MUST match an entry in the parameter file
    -E : minimum access (see minimum-every: above)      default: no minimum
    -h : extra FipHdr data to add to each run       default: none
        use this in the SYSTEM file entry to use the same webwire/ftp/etc parameter
file but vary the results
    -H : alternate host name                default: none
        use this for checking primary/secondary for clustered systems where the
actual hostname differs from the cluster name
        ie if the host name may be fipcluster-a or fipcluster-b but there is a
cluster hostname of fipcluster
    -i : queue to scan (-i and -q are identical)        default: none
    -l : log everything through
    -L : log nothing (timer and worker programs)
        default: timer reports errors only but worker programs log all
    -n : parameter file in 'tables/setup'.          default: TIMER or TIMER.fip
    -q : queue to scan (-i and -q are identical)        default: none
        The DA FipHdr field should contain the client to trigger
    -Q : quiet mode - message if NOT the primary system default: do not message
    -R : do NOT resolve hostnames               default: do
        This will resolve any hostnames in the DEST_REDUN file (if there is one)
    -t : Set the test flag on               default: no
    -T : Set the test flag on and the test FH to this 2 letter code     default: no
    -v : version and exit
    -V : show the run lines of each program and exit    default: no
        -V on its own will show only those jobs that should run NOW - on this day at
this time
        -V -A will show all jobs - ignoring the time they should run
    -w : wait time in seconds               default: 60 secs
    -x : external proxy file to use             default: no
        This should be in tables/setup and ONLY have the 5 proxy keywords in.
        This overrules any 'proxy-server' etc commands in the main parameter file
        AND any proxy commands in the program parameter files too.
        AND bandwidth-stats:yes/no/client
    -z : name of a single script/program            default: none
        (same as -1)
    -Z : parameters for the script              default: none
-- NOTES --
1. FipHdr fields
    WN  client name
    WG  group
    W0  url
    W1  matchlinks
    W2  ignorelinks
    W3  extra fiphdr
    W4  spare
    W5  ignorekey
    W6  logon
    W7  passwd
    W8  data yes/no
    W9  extra destinations
    (plus any specified in the extra-fiphdr: keyword)

days:X for all.Z for weekdays (or M-F ??)

2. Types of grabs for the Internet Grabbing are (only the 1st chr is
significant except for ftp-push)
    type:w  WebGrabber  (using webwire)
    type:f  Ftp Get     (using ipftp)
    type:ftp-push   Ftp Send (using ipftp)
    type:u  Ftp Send    (using ipftp)
    type:e  E-mail receive  (using ipsmtp)
    type:x  External Script - no default
    type:s  Soap get    (using ipsoap)
    type:n  NNTP receive    (using nntpwire)
    type:3  POP3 receive    (using ipsmtp)
    type:i  IMAP receive    (using imapwire)
    type:imap   IMAP receive    (using imapwire)
    type:ipspool    ipspool

3. changing the program to be use
If you need to use different programs for each of the stages :
    program-webgrab: (path/filename)
    program-ftp:     (path/filename)
    program-ftp-push: (path/filename)
    program-email:   (path/filename)
    program-nntp:    (path/filename)
    program-pop3:    (path/filename)
    program-imap:    (path/filename)
    program-soap:    (path/filename)
    program-im:  (path/filename)
    program-ipspool: (path/filename)
++ NOTE you will need to add this to the parameter file if you are using SSL
versions :
    program-webgrab:/fip/bin/webwiressl
Use this for testing new versions :
    program-ftp:/fip/zsav/v3.21/ipftp

4. iptimer can do a one-off snap of a client by just appending the clientname
to the snap file :
    /fip/fix/setup/FIPtimer.snaps.(name of TimerParameter File forced uppercase)

5. For ipspool, the default wait is 5 seconds, use this to alter
    ipspool-wait:22

Version Control
;2n56   13may05 bandwidth stats for ipftp, added skip-balance-group
    ;c 24jun05 added minimum-every and -M
    ;d-f 06sep05 added -V and -x
    ;g 09jan06 external script better
    ;h-i 09feb06 added throttle-output
    ;j 02aug06 added -Z param switches (k 64bit wait)
    ;l 07sep06 bugette with ipftp and output folders
    ;m4 05may07 added snaps ;3 tweak to time and added maxtime:
    ;n2 20nov08 added check-primary-server
        ;3 6jan09 redid imap ;5 17feb09 added source/cwd/host ;6 allow "" in client:
lines
        ;7-9 22jun09 added type:s for ipsoap
        ;11 6jul09 added -c (single client)
        ;12 3sep09 better reporting of errors
        ;13-14 22sep09 added trace-starts:yes
        ;15-16 28dec09 bugette Win2K primary host
        ;17-21 19apr10 tuning timings - webgrab-gap added ;20 added extra-fiphdr,
fiphdr now parsed
        ;22 13oct10 added client:.. switch:.. ;23 program-soap ;24 22jun11 allow
webwire -w0
        ;25 16jan12 mix up with every, start and end
        ;26 02apr12 allow -B with imap
        ;27-29 7dec12 bugette with zargs and added ftp-push
        ;30-32 20oct14 overnights better ; date with FipSeq ;33 sparc null strerror
bug ;34 proxyHandshake
        ;35 bugette - sometimes ran once on startup for unwanted day
        ;36 28jan16 added -h so we can use one parameter file
        ;37 12feb16 arrgh 64bit WEXITCODE are rubbish
        ;38-39 26jun17 added sync and times-are-gmt
        ;40-41 12aug17 added type:ipspool -1 ; 42 added hours every:6h
        ;43-46 16dec17 only end : if the start is < end - for the case where it goes
over midnight
        ;47 13may19 allow override of the primary server ...
        ;48-49 11jan20 sync can now be in secs too
        ;50-54 remove QZ from webwire run line; logging ; better offset-utc ; minor
        ;55 28aug22 Buglete with vbig -h switch
;001z   09may04 added TIMER etc
    ;i-j 11jul04 status E only for failed starts plus added SU and SA
    ;k 18aug04 added check if anything running before start OR new pram file
    ;l-n 01oct04 added Seconds to 'every:'
    ;o 01dec04 added input folder for one-off triggers rather than timers.
    ;p 10dec04 WINNT - not closing Handles properly
    ;r 31dec04 added use-proxy:yes/no
    ;s-t 07feb05 FTP - added logon and passwd and data and dests and -q
    ;u-y 18apr05 added outque,ignorekey,spare to our list
;000    22jul99 original version

(copyright) 2024 and previous years FingerPost Ltd.