iptimer
iptimer
This program kicks off another script/program, waits a few seconds, does it
again... and again... and again
----- Main Parameter file -----
The syntax of the Main Parameter File - by default tables/setup/TIMER (or
TIMER.fip on Winnt)
; comment line
group: (groupname)
track-status:yes/no
stagger-starts:1
client:(name) day:(MTuWThFSaSu) time:20:30 mustget:
client:(name) date:(UTCdate 20040620) time:20:30 mustget:
client:(name) day:(MTuWThFSaSu) every:30 start:07:30 end:19:00 maxtime:3600
Other (ie non client) keywords
group - name of the group
this is just a handle
track-status:(yes/no)
stagger-starts:(number)
This is the gap in 100ths of a second between starts.
The maximum is 100 for 1 second delay.
Use this so all clients for a particular time are NOT started at the same
millisecond.
default: none for zero wait
trace-starts:(yes/no)
Write a trace file of programs and parameters started in /fip/log/timer
default: no
extra-fiphdr: (FipSeq)
More FipHdr information which is added to ALL starts.
This is in addition to the information in 'client:xx ... fiphdr: ...'
proxy-server: If using a proxy, these are the hostname (or IP address) and
port to aim at.
proxy-port:
proxy-logon:
proxy-handshake:yes/no Does the proxy need to say hello first ? default: no
proxy-is-squid:yes/no Is the proxy a Squid ? default: no
Proxy is only available for webgrab clients. Pls see the
doc for webwire for further information.
test-fiphdr: (2 letter FH code) default: none
if -t input switch is ON, and there is NO -T XX or test-fiphdr, the default
is QZ
eg if a client is flagged as test, client runline has this T5:TEST added
test-fiphdr:T5
client:hoho type:w template:HOHO_GRAB every:1 test:yes
throttle-output:(parameter file for iproute)
The output is to be checked for runaway repeat offenders by passing all data
thru 'iproute'
This adds an extra FipHdr field 'SR:' to the fiphdr and changes the output
folder to /fip/spool/2brouted.
webgrab-gap: (no of seconds)
Gap in seconds between file grabs for webwire
default is 1 (version 2n11 and after) or 5 if previous
minimum-every:(no of seconds)
For those clients with 'every', override the settings with this amount if it
is lower.
This can also be set with the -E input switch
default: no minimum (ie 1 second)
ie if you have a client which is being accessed every 5 secs, a second
every 10 secs and a third every 60: if minimum-every:30 then the
first two will be scavenged every 30 seconds and the third unaffected at 60
secs.
bandwidth-stats:yes/no/client
Force the worker programs - webwire, ipftp etc - to generate statistics on
duration and data volumes (if they can)
If the parameter is 'client', individual group_client_program files are
created in log/bandwidth_stats
Otherwise all information is placed in Hour_group_program files for all
accesses.
If bandwidth stats are on, then logging is turned OFF for the worker
programs. So they will only log errors.
default is NO
skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute
the skip file when changed (see doc on 'ipbalan') - for ipftp and webwire.
This is often used where a second system could be used as a redundant server
if the main system fails. (see also -B input switch)
default-max-timeout: default maximum time in seconds for every process to run.
default is 21600 for 6 hours (before version 2m3 this was ignored and there
was no max)
check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
Use this when iptimer runs on 2 systems with the same parameter file (ie both
run the same jobs)
- but only one should be live : an FTP grab or web scavenge for example
This parameter is used to check if the current host should be getting the
files or not
ie in the parameter file is
check-primary-server:timersvr
and in the DEST_REDUN is
; psuedohost primary secondary
timersvr fip1 fip2
and in the SYSTEM file for both fip1 AND fip2 there is the same 'iptimer'
line with the same parameter file
If fip1 is up, 'iptimer' on fip1 will always get while on fip2 it will just
check/loop.
If fip1 is down 'iptimer' on fip2 will start processing the files.
times-are-gmt:
times-are-utc: all start/end/time subparameters are based on GMT not localtime
Use this if the times need to be constant over the year and not change with
summer time
default use the System time
client-time-zone: set a timezone which affects for ALL clients display time,
daylight saving etc for start: end: etc
default-time-zone: on systems which do NOT have a system TZ - used ONLY to
reset after using the client-time-zone
see man tzset default: system timezone or the env variable "TZ"
eg if using a file /usr/share/zoneinfo/, add a colon eg default-time-zone:
:America/New_York
In detail :
- 'client' or 'source' keyword (these are identical):
client: name of a client or source
which should match a profile defined for the times and dates
url:
host: url to grab or hostname for ftp, nntp etc (url and host are identical)
port: port number if not default (webwire, ftp, soap etc)
match: (FipSeq string)
(optional and possibly up to 10 may be specified)
ignore: (FipSeq string)
(optional and possibly up to 10 may be specified)
fiphdr: (FipSeq string)
(optional extra FipHdr information)
This will add the W3 fiphdr field. To add other fields, use a '#' to
separate:
fiphdr:abc
will give a value of 'abc' in the FipHdr W3
fiphdr:WY:123#WA:online
will give a (probabaly wrong) value of 'WY:abc' in the FipHdr W3
plus 'online' in WA
fiphdr:#WY:123#WA:online
will give a value of nothing in the FipHdr W3
plus 'online' in WA and '123' in WY
test: (yes/no)
adds the FipHdr field QZ:TEST if test-fiphdr:QZ or -T QZ
default is NO
type: (webwire/ipftp/...)
Type of grabs for the Internet Grabbing - pls see note 2 below
default is 'webwire'
switch: (FipSeq string)
Optional extra program input switches to use when the program is started
default is none
template: (name of template parameter file)
Mandatory for webwire or ipftp
logon:
passwd: (webwire, soap, ftp etc where needed)
Logon and password for any authentication
no default
data: (webwire)
Do we need the data of the file - data Yes/no
default: yes
dest: (all)
extra destinations (FipHdr DU on output)
default: no extra
outque: (webwire, imap, soap, ipspool and ftp)
Different Output queue under spool - for webwire this could be for a
rpt-offenders check.
default: spool/2go
queue: (ipspool and ftp send only)
Input folder under spool
default: none !
ignorekey: (webwire)
add ignore-key:\W5 to the webwire parameter file.
default: none
spare: (all)
Extra information which is added to the FipHdr \W4
default: none
primary-server: (override check-primary-server)
primary: (override check-primary-server)
This allows a single client to use a different primary-server
Default is that defined by check-primary-server - or not set/checked if the
latter is not specified
script: (path/script options)
Script name and options for type:x, external script.
default: none
maxtime: Maximum time for the job in seconds
default: 21600 for 6 hours (before version 2m3 - none)
cwd: (folder name)
for ftp only, the folder name to 'cd' to
There can be up to 1000 clients in the main parameter file.
either
date: Actual Date to run this
Format is UTC style : CCYYMMDD
ie 20091225 for Christmas Day 2009
You can use FipSeq for 'date' - but it is resolved ONLY when the parameter
file is changed. So that if you want it to change, stop;/restart iptimer or
touch the parameter file to trigger a new read.
eg only get on the 15th of each month ... date:\$e\$y\$i15
or
day: Day of week to run the job :
M Monday
Tu Tuesday
W Wednesday
Th Thursday
F Friday
Sa Saturday
Su Sunday
X Every day.
Z Every weekday (ie Monday to Friday)
Case is NOT important.
Commas or dots (but NOT spaces) may be used to separate.
Default is every day.
either
time: Time of the day on 24 hour clock. Default is 18:00.
or
every: interval in minutes between grabs Default: none
every: (mins) [(optional) start:(starttime) end:(endtime)
for seconds, add 's' or 'secs' after the number - 45s for 45 secs
for hours, add 'h' or 'hour' after the number - 6h
eg every:30 start:07:30 end:19:00
every:30secs
The minimum interval is 15 secs (every:0.15) unless modified with
'minimum-every:' (see above)
sync: for 'every', syncronise on this minute of the hour default: time when
iptimer starts
eg every:20 sync:11
will kick off the item when iptimer is started ..
and then every 20 mins at 11, 31 and 41 mins past
The default is at start and a GAP of 20 mins from the end of one grab to the
next
Specify seconds either with a decimal or tailing 's' : sync:17s sync:0.17
sync:3:24
Note that the sync resets on the hour - so the first one after the hour MAY
be at a odd, extra time.
eg:
client:UKcompanies start:07:20 end:17:45
client:Forex day:MTuWThF time:16:30
client:Forex day:MTuWThF time:16:40
Get the Page file tables/setup/FOREX every week day at 16:30 and 16:40
Input options (all optional) :
-1 : name of a single script/program default: none
(same as -z - ie OldStyle; does not use the parameter file at all
-B : name of Balance Group for skip files default: none
This should exist in sys/BALANCE (see ipbalan)
-c : Name of a single client or source to trigger an instant snap default:
spooled
after which the program stops
The name MUST match an entry in the parameter file
-E : minimum access (see minimum-every: above) default: no minimum
-h : extra FipHdr data to add to each run default: none
use this in the SYSTEM file entry to use the same webwire/ftp/etc parameter
file but vary the results
-H : alternate host name default: none
use this for checking primary/secondary for clustered systems where the
actual hostname differs from the cluster name
ie if the host name may be fipcluster-a or fipcluster-b but there is a
cluster hostname of fipcluster
-i : queue to scan (-i and -q are identical) default: none
-l : log everything through
-L : log nothing (timer and worker programs)
default: timer reports errors only but worker programs log all
-n : parameter file in 'tables/setup'. default: TIMER or TIMER.fip
-q : queue to scan (-i and -q are identical) default: none
The DA FipHdr field should contain the client to trigger
-Q : quiet mode - message if NOT the primary system default: do not message
-R : do NOT resolve hostnames default: do
This will resolve any hostnames in the DEST_REDUN file (if there is one)
-t : Set the test flag on default: no
-T : Set the test flag on and the test FH to this 2 letter code default: no
-v : version and exit
-V : show the run lines of each program and exit default: no
-V on its own will show only those jobs that should run NOW - on this day at
this time
-V -A will show all jobs - ignoring the time they should run
-w : wait time in seconds default: 60 secs
-x : external proxy file to use default: no
This should be in tables/setup and ONLY have the 5 proxy keywords in.
This overrules any 'proxy-server' etc commands in the main parameter file
AND any proxy commands in the program parameter files too.
AND bandwidth-stats:yes/no/client
-z : name of a single script/program default: none
(same as -1)
-Z : parameters for the script default: none
-- NOTES --
1. FipHdr fields
WN client name
WG group
W0 url
W1 matchlinks
W2 ignorelinks
W3 extra fiphdr
W4 spare
W5 ignorekey
W6 logon
W7 passwd
W8 data yes/no
W9 extra destinations
(plus any specified in the extra-fiphdr: keyword)
days:X for all.Z for weekdays (or M-F ??)
2. Types of grabs for the Internet Grabbing are (only the 1st chr is
significant except for ftp-push)
type:w WebGrabber (using webwire)
type:f Ftp Get (using ipftp)
type:ftp-push Ftp Send (using ipftp)
type:u Ftp Send (using ipftp)
type:e E-mail receive (using ipsmtp)
type:x External Script - no default
type:s Soap get (using ipsoap)
type:n NNTP receive (using nntpwire)
type:3 POP3 receive (using ipsmtp)
type:i IMAP receive (using imapwire)
type:imap IMAP receive (using imapwire)
type:ipspool ipspool
3. changing the program to be use
If you need to use different programs for each of the stages :
program-webgrab: (path/filename)
program-ftp: (path/filename)
program-ftp-push: (path/filename)
program-email: (path/filename)
program-nntp: (path/filename)
program-pop3: (path/filename)
program-imap: (path/filename)
program-soap: (path/filename)
program-im: (path/filename)
program-ipspool: (path/filename)
++ NOTE you will need to add this to the parameter file if you are using SSL
versions :
program-webgrab:/fip/bin/webwiressl
Use this for testing new versions :
program-ftp:/fip/zsav/v3.21/ipftp
4. iptimer can do a one-off snap of a client by just appending the clientname
to the snap file :
/fip/fix/setup/FIPtimer.snaps.(name of TimerParameter File forced uppercase)
5. For ipspool, the default wait is 5 seconds, use this to alter
ipspool-wait:22
Version Control
;2n56 13may05 bandwidth stats for ipftp, added skip-balance-group
;c 24jun05 added minimum-every and -M
;d-f 06sep05 added -V and -x
;g 09jan06 external script better
;h-i 09feb06 added throttle-output
;j 02aug06 added -Z param switches (k 64bit wait)
;l 07sep06 bugette with ipftp and output folders
;m4 05may07 added snaps ;3 tweak to time and added maxtime:
;n2 20nov08 added check-primary-server
;3 6jan09 redid imap ;5 17feb09 added source/cwd/host ;6 allow "" in client:
lines
;7-9 22jun09 added type:s for ipsoap
;11 6jul09 added -c (single client)
;12 3sep09 better reporting of errors
;13-14 22sep09 added trace-starts:yes
;15-16 28dec09 bugette Win2K primary host
;17-21 19apr10 tuning timings - webgrab-gap added ;20 added extra-fiphdr,
fiphdr now parsed
;22 13oct10 added client:.. switch:.. ;23 program-soap ;24 22jun11 allow
webwire -w0
;25 16jan12 mix up with every, start and end
;26 02apr12 allow -B with imap
;27-29 7dec12 bugette with zargs and added ftp-push
;30-32 20oct14 overnights better ; date with FipSeq ;33 sparc null strerror
bug ;34 proxyHandshake
;35 bugette - sometimes ran once on startup for unwanted day
;36 28jan16 added -h so we can use one parameter file
;37 12feb16 arrgh 64bit WEXITCODE are rubbish
;38-39 26jun17 added sync and times-are-gmt
;40-41 12aug17 added type:ipspool -1 ; 42 added hours every:6h
;43-46 16dec17 only end : if the start is < end - for the case where it goes
over midnight
;47 13may19 allow override of the primary server ...
;48-49 11jan20 sync can now be in secs too
;50-54 remove QZ from webwire run line; logging ; better offset-utc ; minor
;55 28aug22 Buglete with vbig -h switch
;001z 09may04 added TIMER etc
;i-j 11jul04 status E only for failed starts plus added SU and SA
;k 18aug04 added check if anything running before start OR new pram file
;l-n 01oct04 added Seconds to 'every:'
;o 01dec04 added input folder for one-off triggers rather than timers.
;p 10dec04 WINNT - not closing Handles properly
;r 31dec04 added use-proxy:yes/no
;s-t 07feb05 FTP - added logon and passwd and data and dests and -q
;u-y 18apr05 added outque,ignorekey,spare to our list
;000 22jul99 original version
(copyright) 2024 and previous years FingerPost Ltd.