ipftp (Sat Oct 25 2014 01:31:01)

ipftp

This program loops around a queue and sends or gets or lists files to/from a
Remote Server via FTP.

As FTP does hang on (regular) occasion, no response for a timeout period will
abort that file or transmission and the program attempt to restart where
appropiate.

By default, the program can be spooled against a queue or kicked off for a
single file or queue from a script using the '-1' input switch, which we call
'single-shot'.

For single shot, the files are NOT deleted unless the keyword 'del-files' is
specified.

Files are only deleted if sent correctly or, for getting files, if a reasonable
response was received like 'File Not Found'.

The FipHdr of each file to be sent is checked for the 'DF' field which, as per
normal for Fippies, is the name of a parameter file in tables/ftp. This
parameter file, tables/ftp/FORMAT by default, is used to describe the
transmission.

The Remote Host Name can be a Fip Pseudo-Host (ie with an entry in
'tables/sys/DEST_REDUN' ). In this case, the primary host is checked for being
down and, if so, data sent to the secondary. If that too is down, files are not
sent.

File for remote systems that are offline or un-pingable are send to
'spool/ftp-offline' which will be rescanned on a regular basis. When such a
system recovers, the program will automatically send the collected data files
form 'ftp_offline'.

Files for retrieving can be left on the remote server untouched, deleted or
moved to a specified Done folder. There is a Poll input switch which will just
look once during the period for any files and get them accordingly.

A skip file may be created and maintained automatically for GET files which we
do NOT need to re-get. This is used where a 'get-and-zap' or 'get-and'move'
cannot be used.

ipftp can also be used to merely list the files on the remote server using the
-D input switch or the 'list-folder-only:' parameter.

Syntax for the parameter file are :
	; comment
	remhost:	Hostname on remote system
		This overrides the '-s' Input switch.
		defaults to that specified in the '-s' Input switch
		This can be in FipSeq or just fixed text.
	remport:	Port on remote system
		This overrides the '-p' Input switch.
		defaults to that specified in the '-p' Input switch
		or failing that, defaults to 21
		This can be in FipSeq or just fixed text.
	logon:		Logon for remote system		default: anonymous
		This can be in FipSeq or just fixed text.
	password:	Password for remote system		default: none
		This can be in FipSeq or just fixed text.
	nopassword:	The remote system does not need a password
		To send a password of nothing (which is different to NO pwd),
		specify the keyword with no parameter :
		eg	password:
		Normally a remote system wants a password even if it is blank.
	remhost,remport,logon,password,nopassword may be repeated another 9
		times for roundrobin which is enabled by the -R input switch.
		remhost:mainsys.bigdom.com
		logon:main
		password:passo
		remhost1:backup1.bigdom.com
		logon1:back
		password1:passo
		remhost2:backup1.bigdom.com
		logon2:backo
		password2:passo
	ftpbefore:  FTP command to execute after logon		default: none
	ftpafter:   FTP command to execute before closing up	default: none
	ftpbeffile: FTP command to execute before each file.	default: none
	ftpaftfile: FTP command to execute after  each file.	default: none
		Run some raw Ftp commands....
		There can be several commands for each section, each on a separate line.
		Note that the actual commands are NOT those from the user interface,
		but the raw ones as used by the FTP protocol. For weirdos you might
		want to check that the remote system will actually allow you to do these !
		Valid commands include :
			dele	file delete ** See notes below
			cwd	change directory
			mkd	make queue
			rmd	remove queue
			pwd	get current queue
			list	long  queue list - like an 'ls -l' or 'dir'
			nlst	short queue list - like an 'ls'	or 'dir/w'
			nlst -lrt : long list, sorted by time.
				For lists, always 'cwd' into the queue beforehand.
			type	'a' for ascii or 'i' for image or binary
			rnfr	rename file : these two work in tandem ..
			rnto		rnfr=old name, rnto=new name.
		eg:
			; Do a MKDIR (who cares if it fails) and a Chg Dir before anything
			ftpbefore:mkd /pub/pporange
			ftpbefore:cwd /pubmp/pporange

			; before each file, make sure we delete an old version before
			ftpbeffile:dele SN

			; at the end do a List to see what is really there.
			; the list will be in the log file
			ftpafter:nlst -lt
			logfile:/fip/log/ftp/FTPCLIENTS.$d-$m-$e$y
		Note also there are some Fip-enhanced keywords :
			fiptouch
			fipdelay
			fipblockfile
			fipallowfile
			fipdelete
			fipduplicate
			fipon-error
			fipdir
			fiplongdir
			fipcd
		see below for explanations on what these do !
	ftpproxy:   FTP command to execute after logon		default: none
		to logon to yet-another FTP server.
		Please see the notes below

	ignore-all-preparation: ignore ALL the before/after/beffile/aftfiles
		and just send that RAW file out. Use this for big binary files that
		need NO alterations or additions.
	before:	FipSeq string added before data of each file.	default: none
	after:	FipSeq string added after  data of each file.	default: none
	beffile: FipSeq file added before data of each file.	default: none
		Beffile is added after any 'before' string and before normal data.
	aftfile: FipSeq file added after  data of each file.	default: none
		Aftfile is added after normal data and before any 'after' string
	binary-beffile: binary file added before data of each file.  def: none
		Beffile is added after any 'before' string and before normal data.
	binary-aftfile: binary file added after  data of each file.  def: none
		Aftfile is added after normal data and before any 'after' string
	first:	Name of optional file in tables/ftp that is sent
		on startup and on each scan of the queue when a new file has
		been found.  The text of the file is in FipSeq. default:none
	last:	Name of optional file in tables/ftp that is sent when all the
		files in the queue have been sent and before we rescan the
		queue.  The text of the file is in FipSeq.	default:none

	hostname: The hostname for the system 'ipftp' is running.
		This defaults to the hostname the system was booted with.
		However for systems with multiple ethernet addresses,
		this is used to tell the remote system the IP address to use.
		(Note for versions 06c+, this is no longer required).
	del-files: Delete the files once sent. This is for -1 single file only.
			default: for -1 send one file, do NOT delete input file
			default: for spooled folder, delete ALL input files
				once sent correctly.
	timeout: Timeout period when sending for no response	default:120 secs
		The default timeout can be modified by the -T input switch.
		If you are sending to a host which is only across the computer
		room, you should take this timeout down to 10 or 15 secs.
	connection-retries: No of retries if the first connection
		fails. The default is 5. Which means a remote system
		which has disappeared can hold up the next traffic by up
		to (5) * (120 secs timeout) = 10 mins !.
 	passive-connection: (yes/no) This causes the remote server to enter PASV mode
(if it can).
 		The default is NO for 'active' if the command is NOT specified
 		The default is YES for 'passive' if the command is specified with no
options.
	passive-force-address:yes/no					default: yes
		Force the IP address for a Passive Data connection to be the same as the
Control
		Sometimes the remote server has a funny NAT address - use this to 'clean' it
up
	linger-on-close: Timeout in secs for the data to be sent.
		For slow connections, increase this,
		default: 3 (secs)

	doneque: Queue to move original files once sent.	default: none
	errorque: Queue to move original files if NOT sent.	default: none
		This is used ONLY where systems do NOT have a valid IP address
		because there  is no entry in the host file, or it has been
		missplet or, more likely, DNS has flipped.
		If the remote host is out there but the Logon is refused or,
		once logged on, we were unable to write the file, the files
		are stuffed in 'spool/ftp_offline' or if the input queue
		is NOT the default, spool/ftp_(inputque)_offline
	outque: For get files only, this is the output folder for incoming files
		This overrides the '-o' input switch and the default is 'spool/2go'.

	onefile: Send one file at a time to this host.
		After each file, 'ipftp' disconnects, waits and then reconnects
		before sending the next.  default: send all files in the queue.
	maxfiles: Send (or Get, from version 18j54) this number of files at a time
to/from this host.
		After so many files, 'ipftp' disconnects, waits and then reconnects before
sending/getting the next bunch.
		Use this to 'throttle' a connection where too many files (in or out) might
cause a system overload.
								default: send/get all files
	logfile: Log file name					default: none
		This should be the full path and filename of the the log file.
		The contents of the Log file are preserved and appended to.
		Use this to debug new feeds.
		You must make sure the queue exists for the log file as 'ipftp' will not
create it.
	logfile-dest: Destination for the log file		default: none
		This is the Fip DU destination for a log of the transmission.
		Use this to debug new feeds.
		Only specify EITHER 'logfile' or 'logfile-dest' (or neither)
		but not both.
	minimum-log:	Only log starts. stops and errors in the 'logfile'
		Normally the whole session is logged.
	dest:	Fip Destination for any Directory Listing or	default: none
		Retreive files. This should be a valid DU in the tables/sys/USERS file.
	send-no-data: Do NOT send any data from this file	default: send data
		Use this option to send just a headline with a before or beffile
		or to place a marker in a directory in the remote host.
	fiphdr:	Send Fip Hdr if part of the file.		default: send data only
		Normally old the data part is sent and the FipHdr stripped off.
	newname: Newname for the file(s) in FipSeq		default: as original
			newname:SN.$Z
		The default is the same as 'newname:SN'.
		Note that the 'first' and 'last' files are NOT renamed.
	uniquename: Force the remote server to store the file	default: no
		with a new name if it is a duplicate.
		Some systems allow you to give a filename, others do not.
	append: Force the remote server to either append to	default: no
		an existing file or create a new the file
		Some systems allow you to give a filename, others do not.
		This is not available on all remote systems !
	forcename: force the filename on the remote system to be upper or lowercase
		forcename:upper					default: no change
	formatname: Make sure the format of the filename on the remote system
		is correct for that system
		options are	raw, unix, mac, nt, alphanumeric,pc
		default: unix on *nix and nt on Win2k
		YOU WILL NEARLY ALWAYS want to format the filename as different systems only
allow a subset of characters.
		Note that Win2K boxes do not like trailing dots so 'formatname:nt' also
strips them.
		To preserve the filename - such as if you are using ZO - use formatname:raw
for no changes at all.
	sizename: Maximum size of the filename on the remote system
		eg sizename:32					default: no limit
		Note that normally the max for Windows/NTFS is 127, Unix is 255 and Mac is 31
		The maximum for sizename is therefore 255.
	send-external-file-name: Name on the remote system for any external files
		Use this parameter to send the external file separately to this name
		default - if there is an external file it is sent in the one file.
	send-external-fiphdr:yes/no
		if there is an external file containing the data (pointed to in the
FTP_EXTERNAL_FILE: fiphdr field), it may (or may not) have a FipHdr.
		Use this parameter to NO=strip it (or YES=leave it on) when sending.
		default is yes to send the fiphdr on the external file
	rename-prefix: (FipSeq) prefix for files that will be moved in get-and-move:
		If the remote server is a Windows server, permissions may prohibit a straight
move of duplicate filenames.
		Use this to make the filename unique.
		It is a Prefix as often, the extension needs to be kept!
		Eg	rename-prefix:$e$y$i$d$h$n$b_
		If the original file is POPE.JPG, this will prefix the date and time to the
moved file.
	rename-newname: (FipSeq) newname for files that will be moved in get-and-move:
		If the remote server is a Windows server, permissions may prohibit a straight
move of duplicate filenames.
		Use this to make the filename unique. Remember FipHdr E1 will have the
filename
		Eg	rename-newname:E1.$z
	send-empty-files:put/send
	send-empty-files:put/send
or	put-empty-files:put/send
	put-empty-files:ignore
		For SEND, if the file is zero length or empty this parameter will allow you
to :
			put or send	- send the file and process as normal
			ignore		- skip to next file
		default is send

-- Listing Folders instead of sending .....	(note the default is to SEND)
		list-folder-only:

		Only one parameter file is used - specify with the '-z' input switch.
		Also specify '-G 0' for a one-off grab or '-G 99' for a grab every 99 secs
		This MUST have at least one of the following in the parameter file:
			ftpbefore:fipdir	(for short listing of just the filename)
		or	ftpbefore:fiplongdir	(for a long list in the remote servers own style)
		or	ftpbefore:fipstddir	(for a long list in the generic style - using MLSD)


-- Getting files instead of sending ...... 	(note the default is to SEND)
	getallfiles:		Get ALL files from the remote server.
	getallfiles:199*	Get ALL files starting '199' from the remote.
	getfile:		Get a file with this name from the remote.
	get-and-move:
	get-and-moveall:
	get-and-zap:
	get-and-zapall:
	getallfiles-except:(mask)
	get-and-moveall-except:(mask)
	get-and-zapall-except:(mask)
		In this case no data is sent, as only the FipHdr is used to find
		the filename. There can be several lines of 'getfile'.
		Watch out for case sensitive remote systems !!
		No wild cards are allowed for single file gets but you can use FipSeq :
		eg	; get a file from the remote system with the same name ..
			; .. as the input file on the local system.
			getfile:SN
			; always get the readme
			getfile:README
			; get todays file which has a 8 digit day extension
			style:QD	$D,%.08d
			getfile:News.QD
		You should also define the Fip destination DU using 'dest:',
		if not all the files all files will be sent to a DU of 'woops'.

		Use the 'except' versions where the mask is used to IGNORE files.
		eg	get-and-zapall-except:.pl
			Get all the files EXCEPT those with '.pl' in the filename
		NOTE the except string is just a simple compare and CANNOT include
wildstrings/chrs
		This is slightly inconsistent
		eg	get-and-zapall-except:*.pl
			will look for a filename INCLUDING the 4 chrs '*.pl'
		BUT it can be FipSeq
		eg	get-and-zapall-except:$e$y$i$d
			Get all files EXCEPT those with todays date in format YYYYMMDD like 20081231

		Commands 'get-and-zap', 'get-and-zapall' and 'get-and-zapall-except' :
			get then zap each file.
		Commands 'get-and-move', 'get-and-moveall' and 'get-and-moveall-except' :
			get then move each file to a done folder on the remote
			box setup by the 'remote-done-folder' keyword.
			PLEASE see below for comments on getmove and getzap.

		Note that if the remote FTP server is a Win2K box, the name of the
		file MAY have a '/' in it (or a Unix FTP server may have a file with a
		'' in the name). Normally 'ipftp' ignores such files as it assumes they
		are sub-folders which should not be scavenged.
		Use the parameter 'walk-remote-folder-tree:' to grab these - see below!
	get-filter-case-sensitive:no FOR SSH connections only, you can force the
filter to be case INsensitive using this parametrer.
		default is YES it is case sensitive

	remote-done-folder: Relative path on the remote box for the data files
		after they have been GET'ted.
		eg	remote-done-folder:../done
	remote-done-exists: Action if the file already exists in the
remote-done-folder
		Options are 'replace' 	- replace existing with this one - (default)
			or 'add-ext'	- add the new version with a date_time extention
			or 'ignore-new'	- just delete the NEW version once the transmission has been
completed successfully
		eg	remote-done-exists:ignore-new
	get-extra-fiphdr: extra FipHdr to attach to incoming, GET'ted files
		eg	get-extra-fiphdr:#ZI:#SU:REMOTE
		This will tell 'ipwheel' to archive the incoming data under
		the archive log REMOTE.
	no-fiphdr-on-getfiles: do NOT add a FipHdr to the GET'ted files.
		default is to add one which Date and Time fields etc.
	standalone-filename: (for Get files, do NOT put a Fip-style filename
		but use the original filename (suitably modified for the
		actual system - ie strip '/' if on UNIX, ':' for Win2k)
		The default is the normal Fip filename/fiphdr malarky.
		eg	standalone-filename:SN_$e$y$i$d.fip
	get-copy-file: (full path name in FipSeq)	default:no
		Make a copy of each incoming file as this folder/name
		eg get-copy-file:/fip/data/raw.data/$e$y$i$d_DF/ZZ_$h$n$b_$z
		this leaves a copy in raw.data/(date)_(format)/(filename)_(time)_(seqno)
	hash-in-filename: (FipSeq chr)			default:235
		A hash/pound (#) in a GET filename is mapped to this chr
		Normally hashes are end-of-field in a FipHdr. So it needs
		to be mapped to something else in order to preserve it.
	get-empty-files:get
	get-empty-files:ignore
		For GET, if the file at the other end is zero length or empty
		this parameter will allow you to :
			get the file and process as normal
			ignore	skip to next file
		default is ignore
	skip-files: Name of a file in /fip/fix/ftp for holding the names of files
			brought over; so that only new files are received. It is
			remade from the LIST on every Get ALL.
			Note for unix/linix systems, the name is cases-SENSitive.
		eg	skip-files:fromFTP
		default: none
	check-skip-details: Yes/no
		The default is only to check if the file-to-get exists or not. But sometimes
		you need to track files which are updated - ie the name remains the same
		but the contents differ. Set this option for tracking changes too.
		Note it is slower and does take a bit more system resource as well
		as an extra skip file in /fip/fix/skip to hold the existing file times etc.
		default: no
	skip-purge-after: (hours) Number of hours to keep the skip entry
		default is 1.  You might want to tune this :
			make bigger if sites add/take off old material
			reduce the time if the same link is used for differnet data
	skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute
		the skip file when changed (see doc on 'ipbalan')
		This is oftern used where more than one system is GETTING the same remote
		system/folder (usually with 'check-primary-server-for-getfiles').
	shadow-skip-file: (full path/filename)
		Name of a duplicate of the skip-file - probably on a network drive.
		Whichever file is newest is used.
	minimum-poll-interval: (secs)
		minumum delay between polls for gets	default: 10 secs
	log-skip-details:
		Log why a file is being retrieved - new file, size change etc
	fixed-data-port: (number)
		if the number is >= 20, fix the data channel port number to this number and
do NOT change it.
	min-data-port: (number)
	max-data-port: (number)
		if the number is >= 20, the data channel port number will be in the range of
min <= number <= max
		defaults are min-32800, max 60000
	check-primary-server-for-getfiles: pseudo-host name that is specified in
tables/sys/DEST_REDUN
		that is used whether the current host should be getting the files or not.
		ie in the ftp parameter file REMOTEGET is
			check-primary-server-for-getfiles:remotewire
		and in the DEST_REDUN is
			; psuedohost	primary	secondary
			remotewire	fip1	fip2
		and in the SYSTEM file for both fip1 AND fip2 there is a line
			rem1	local	ipftp -G 600 -Z -z REMOTEGET
		Then if fip1 is up, the ipftp on fip1 will always get while on fip2 it will
just check/loop.
			if fip1 is down, the 'ipftp' on fip2 will start getting.

	log-each-file: (dest) or
	logeachfile:(dest) Send a Success/failed msg to this destination
			for each file. There is no default. This log file is
			just a FipHdr with the following extra fields :
				DR-File Sent OK		DR:ok or DR:error
				DG-Will Retry later	DG:retrying, DG:stopped
				DT-Some message text	DT:No connection
			default: no log created.
		The text for the DR and DG can be in FipSeq and so can contain
		FipHdr and other variables. As they are FipHdr fields, please
		do NOT put NL, CR etc in the fields.
		Note that System Variable $q holds the time taken for transmission.
	log-last-error-file: (dest) or
	loglasterrfile:(dest) Same as for 'logeachfile' but it is sent ONLY
		after the last failed attempt where 'maxattempts' is specified.
		Only log-EACH-file or log-LAST-ERROR-file can be specified - not both.
	log-max-size: If logging is ON and a file is bigger than this size
	DRgood:(text)	Message for the FipHdr field DR on a   successful tx
			default: ok
	DRbad: (text)	Message for the FipHdr field DR on a unsuccessful tx
			default: error
	DGcont:(text)	Message for the FipHdr field DG if, after an
			unsuccessful tz, another attempt will be made.
			default: retrying
	DGstop:(text)	Message for the FipHdr field DG if no further
			attempts will be made as the file was sent successfully
			or the maximum no of attempts has been tried.
			default: stopped
	fiphdr-for-logeachfile: (FipSeq) or
	msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
			'logeachfile' or 'loglasterrfile' msg. This should be in FipHdr
			format and be in FipSeq. It can be used to pass FipHdr fields
			in the outgoing file into the log file.
			eg	msgeachfile:	DF:logdialnSS:SSn
			default: nothing added
	stop-on-error:	Stop if you get a '500' series error in those commands
		you have specified in the 'ftpbefore', 'ftpafter' etc keywords.
		Normally these are ignored ..
		.. which is what you want if you have something like :
			ftpbefore:dele SN
		to delete a similar file beforehand and the file does not exist :
			550 no such file or directory
		is the message received and can be ignored ..
		but then the same message is also given for 'cwd' that does not work !
		Where it is important, use 'stop-on-error' or 'on-error'.
		This is overriden by the 'on-error' keyword - see below.
	fip-syn-log:	add a Item Log line to the Fip Syndication Log file log/SYN.
			normally only the normal Fip log is kept up to date.
	newEN:	A 2 letter FipHdr field which will be the filename of the output file
		or, for GET, the name of the file on the remote system.
		This defaults to EN but if you are already using EN, map it to
		another field.
		For GetFiles this can be the full pathname (especially with Lists),
		so use E1 (below) for the filename only.
	newE1:	A 2 letter FipHdr field which will be the name of the file to GET
		This defaults to E1 but if you are already using E1, map it to
		another field. This field does NOT have any '/' or '\' chrs.
		(for the original name on the remote server, use the fiphdr ZO and not E1)
	newEQ:	A 2 letter FipHdr field which will be the name of the folder
		on the remote system for SENDing. Default is none
		This is usable only for the messaging back
	max-single-fiphdr-size: size that a single fiphdr field can be.
		default is 8196-sep2012 (was 2024)
		the minimum is 1024 and maximum is 31000

	log-max-size: If logging is ON and a file is bigger than this size
		then the first 64 chrs of each block are NOT stuffed in the log.
		This is because the log file can get massive !
		default is 'log-max-size:30000'
	check-message: FipSeq string to replace default check message if the -C
		input switch has been enabled.
		The default string is "Check $d-$m-$e$y $h:$nn"
	failover: Used with the Round Robin switch (-R) and multiple
		remhost/logon/password, this keyword will NOT round robin
		but will always try the highest named 'remhost's first
		and, if not there, will failover to the second etc.
	offline: If Offline : wait in secs between attempts	default: 60 secs
		For the first couple of attempts the program will wait about
		4 seconds or so but all attempts after that will be at least
		60 seconds apart.
	ascii:	The end-of-line (whether CR, NL or CRNL) will be converted
		by the remote to whatever it requires. 'ipftp' converts to
		CRNL and ignores NULs before sending.
		default: binary (ie files received are the same as sent)
	remote-wants-crnl:			default: file is sent  unchanged
		Line endings are converted to CR NL and the file is sent 'binary'
	remote-wants-nl:			default: file is sent  unchanged
		Line endings are converted to NL only and the file is sent 'binary'
	remote-wants-cr:			default: file is sent unchanged
		Line endings are converted to CR only and the file is sent 'binary'
	locale: use a different 'locale' ( look at the man pages for locale)
		Most computers are set to US English and never changed.
		Use this parameter to customise any date/time or Chr translation
s
		The parameter MUST be a valid locale on your system!
		To find out what valid locale exist, type 'locale -a' on unix.
		Eg	; Set for for Brazil, portugese
			locale:pt_BR
	balance-seqno:  Send the Sequence number to this Balance Group
		(see 'ipbalance') Use this to make sure the sequence number
		is always updated on any companion systems.
	balance-delete: Send a note for ipbalan/ipsvrd to delete the mirrored
		file on any remote server.
	redun-balance: balance group for redundant balance.
		Note you should only use balance or redun-balance but not both.
	ignore-timeouts: do NOT message if the sender timed out. Use this with
		caution! We recommend you use it ONLY for GET-polling where
		there may be times when the remote system dies but you do not
		care to know OR you have other tools to check for the problem.
		The actual message which is ignored is :
Thu Nov  4 16:49:33 ipftp !x : **Error - FROMZZ - 192.130.52.10 - NULL : **
Timed Out - no response from remote   0
	on-error:	ignore/abort
		If we have an error from an 'ftpbefore'/'ftpbefile' etc,
		should the program stop processing the file or continue ?
		This sets up the default for that file.
		** See also 'fipon-error' below.
		The default is 'ignore'
		Ths overrides the 'stop-on-error' keyword as it is more flexible
	log-line: Extra information in FipSeq for the Item Log when sending
	log-level:(number)  adjust the amount of logging in the ALL log
		More is less ! so the higher the number, the less you get..
		log-level:99 is the same as input switch '-q' ie minimum logging
		log-level:9 does not report every file, but does report end of tx.
		default is -1 for all logging
	log-get-every: No of seconds to log Get Accesses where nothing new was found
		this reduces the amount of repetitive log messages.
	maxattempts: No of attempts to send this file before stopping
		default is unlimited attempts, files is error are sent to woops
		maxattempts is only checked if the we are unable to send the file.
	slow-down: No of secs to pause between commands (between 1 and 5)
		If the remote host is old or overloaded, use this to slow down the link.
	inc-seqno: or increment-seqno:
	max-seqno: or maximum-seqno
	min-seqno: or minimum-seqno
		Set the Max and Min sequence numbers for this service
		Default is min of 1 and max of 99999 and increment is 1
		Use FipSeq %Z to pull out
		Note that $z and $s will always produce a 4 or 3 digit sequence number.
		Eg	min-seqno:100
			max-seqno:1000
			inc-seqno:20
			; Start at 100 and by jumps to 20 until and including 1000
	zapresforks: Path to Mac ResourceForks to zap those too (sending only)
		This is the path to the resource fork from the data folder.
		It assumes the filename is appended.
		This is NOT valid for '-1' single shot items, only spooled.
			default is to NOT zap
		eg for Helios 	zapresforks:.rsrc/
		eg for Ushare 	zapresforks:%
	script: Optional Script to run AFTER the file has been sent successfully.
		eg	script:/fip/local/FTP_NEXT_STAGE YI-YS
	tracker-script: Optional Tracking script - run AFTER script (if any)
		default: none

	resfork-type: ethershare/ushare
		Add a resource fork on the file at the remote site (sending only)
	resfork-creator: 4 letter creator type eg 'ZILA'	default:FIPO
	resfork-filetype: 4 letter filetype eg '8BIM'		default:TEXT
	resfork-template: template ResFork to copy
	resfork-path: Path to resfork
		eg for ethershare this is '.rsrc/'
	use-ssh:yes/no
		The commands are for a sftp on the remote server - see below
		Both ipftp and ipftpssl may be used for SSH/SFTP
		default is NO
	use-tls:yes/implicit/explicit/auth/no
		The commands are for a ftp running over SSL/TLS on the remote server
		NOTE - ipftpssl and NOT ipftp must be used for SSL/TLS
		default is NO
		no		- normal, standard FTP on (normally) port 21 for the control
		yes or explicit - connect (normally) on port 21 in clear then use SSL for
USER, PASS and data
		auth		- connect (normally) on port 21 in clear, use SSL for USER, PASS then
return to clear for non-data commands - but use SSL for all data
		implicit	- connect (normally) on port 990: use SSL for all control and data
 	tls-auth: (XXX)
 		AUTH type for TLS/SSL			default: TLS
		Valid entries are TLS, SSL, TLS-C (whatever that is !) and something starting
'X-' which will be something homegrown !
		NOTE that for all versions of SSL the method string is "SSL" (this string is
case sensitive according to the RFC)
		eg tls-auth:SSL

	ssl-method: (1,2,3,23,999)
		Version number to use for TLS/SSL		default: 999 for current default (2 or 3)
	ssl-password: (password)
	ssl-passwd: (password)				  default: none
		Optional password if the handshake requires a shared secret
	ssl-cert: (name of a PEM certificate file)		default: none
	ssl-root-cert: (name of a root PEM certificate file)	defaunt: none
		Optional certificates - held in tables/ssl

	throttle-speed: (no of KILOBITS per second maximum)
		If the TCP pipe being used is only 64kbps, then large files may timeout.
		ie a small 1k file should take under a second, but a 1 mb file will take
about 60 seconds to send normally. So if you have set the 'timeout' value at 60
secs, they it might abort before the data could reasonably be send.
		Use this command to pace the link better. Note the parameter is in KBITS/s
which is the normal way of describing a link and NOT BYTES which is the normal
way of describing a file.
		This does NOT stop/slow the data being sent - just delays the timeout for big
files.
		eg for an E1 (2 megabit) line	throttle-speed:2048
		T1	(1.54 mps)		throttle-speed:1544
		56kb line			throttle-speed:56
		Line you know is overloaded - ADSL perhaps - throttle-speed:33

	keep-connection-open: (secs)
		For sending files, do NOT close the connection after each
		series of files and keep the connection open. This means any
		AFTER commands (ftpafter:) are ONLY done on error.
		The parameter is the number of seconds between accesses (which
		is just a 'PWD') to make sure the line has not been closed
		prematurely by the remote end or a network device.

	add-md5-signature: (FipHdr field)
		Create an MD5 signature for the outbound files (Data Part only) and put it in
a FipHdr field.
		This FipHdr can then be used as a 'ftpbeffile' or 'ftpaftfile'.
		default: none

	send-multiple-file:yes
		Sending multiple files that are related (eg XML companion file to a JPEG)
		These files are located in a separate folder and MUST exist before sending
		NOTE - the best way is to zip all the files up and send the single zip !
Otherwise you can get all sorts of problems when the transmission breaks (which
it will do for big files on occasion) half way thru sending.
	send-multiple-que: (name of folder)
		specify a folder where the files are - this is a must
	send-multiple-key: (FipSeq)
		As the folder MAY have lots of files, specify a KEY or STUBname which can be
used to find all related files
		(ie if you do an ls for this key in this folder you ONLY get the files you
want)
		Often the key is a part of the incoming filename, or the contents of a FipHdr
field
	You must also specify ONE of these
		send-multiple-all:yes
			send everything matching the key
		send-multiple-ext:(fileextension)
			only send files with this extension - this can be a FipHdr field
	And thes remaining parameters are all optional
		send-multiple-fiphdr: yes/no
			depending on whether you need to sent the FipHdr too
		send-multiple-zap: yes/no
			zap after successful send ?
		send-multiple-remote-name:$e$y$i$d_$h$n$b_E1.$z
			name for remote if different - might want to add a unique string - date/time
for example

  Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).

	They are normally specified :
		fixed:QZ	1234543
		partial:QT	ST,3,2,U,<,>
		combie:QY	ep|na,(0000000)a
		option:QE	ep,11,7,s
		repeat:QK	XK,-,3
	or	repeat:QP	PK,,4,#X
		style:QS	XN,%.03d
		replace:QN	NN	abc=DEF def=GHI
		newdate:QT	hours-3	"ZD-ZM"
		unique:QU	XC

The input file can optionally be just a FipHdr file with pointers to where the
data resides. This can be useful where large PDFs or JPEGs or other binary
files need to be ftp'ed but you do NOT want to move/copy them inside the
system.  To do this, use FipHdr fields
	FTP_EXTERNAL_FILE: (full path name to file)
A single file before and binary file before (or after) may be specified by :
	FTP_BEFORE:		syntax - same as 'before'
	FTP_AFTER:		syntax - same as 'after'
	FTP_FILE_BEFORE:	syntax - same as 'beffile'
	FTP_FILE_AFTER:		syntax - same as 'aftfile'
	FTP_BINARY_BEFORE:	syntax - same as 'binary-beffile'
	FTP_BINARY_AFTER:	syntax - same as 'binary-aftfile'
Another FipHdr field can also have bearing on this :
	FTP_ZAP_EXTERNAL:	which deletes the external file if transmitted with no
errors (default is to leave the external file alone).
	FTP_LIST_FILE: (filename or path/filname in FipSeq)
		The filename is used for a new file holding a LIST of the current remote
folder.  (same as -F)

Input Parameters are : (all Optional)
	-i : queue to scan for input			default: spool/2ftp
		This can be specified as a queue under /fip/spool or
		if it starts with a '/', the complete pathname.
OR	-1 : name of file to send where we are sending a single file. def:none
		This can be specified as a queue under /fip/spool or
		if it starts with a '/', the complete path and filename.
		The input file is NOT deleted in this case.
OR	-G : poll remote queue for files to Get		default: local spool
		This parameter sets the number of seconds to wait between Polls.
		Only one parameter file is used - specify with the '-z' input switch.
		This MUST have at least one 'getfile', 'getallfiles', 'get-and-zap'
		or 'get-and-zapall' line (or be used with the -D switch)
		If the parameter is zero -  '0' - then it is polled once only
		If > 0, the minimum time is 10 seconds unless modified by
'minimum-poll-interval:'
OR	-D : get the directory listing
		Only one parameter file is used - specify with the '-z' input switch.
		Also specify '-G 0' for a one-off grab or '-G 99' for a grab every 99 secs
		This MUST have at least one of the following in the parameter file:
			ftpbefore:fipdir	(for short listing of just the filename)
		or	ftpbefore:fiplongdir	(for a long list in the remote servers own style)
		or	ftpbefore:fipstddir	(for a long list in the generic style - using MLSD)
	-z : default parameter file			default: tables/ftp/FORMAT

Less often used parameters ...
	-A : spool a folder - then stop when it is empty	default: keep spooling or
(-1) do single
		normally you will -Y for no offlines for this
		see below
	-b : generate bandwidth statistics			default: no
	-B : default balance group for skip files		default: none
		(see skip-balance-group parameter)
	-c : list of hosts to track for check messages.	default: none
	-C : Interval in secs for Check Messages		default: none
		If no data file has been sent for a certain number of seconds
		then a Check Message may be sent. Change the text of a Check
		Message with the 'check-message' keyword.
	-d : done queue for sent files			default: none
		Normally files are deleted after sending.
	-F : force all files to be JUST the List File.		default: no
	-h : extra FipHdr info 					default: none
		Used for cases where external information is not in the
		FipHdr of the file - system variables for example
		This is generally more use for GET files rather than SEND.
		It can be used for extra logging information or to fill in parameters from a
script.
		EG a parameter file GENERIC.GET might have an line 'remhost:BN'
		while the script has ipftp -z generic.get -h '#BN:10.10.20.31' etc etc
	-H : alternate host name				default: none
		use this for checking primary/secondary for clustered systems where the
actual hostname differs from the cluster name
		ie if the host name may be fipcluster-a or fipcluster-b but there is a
cluster hostname of fipcluster

	-k : block size to send data				default: 32 for 32k
		This can be any number from 1 to 32 (in kilo bytes).
		For clients with bad connections, smaller blocks sometimes work better.
	-K : ignore the skip file and skipdetails files.	default: use if specified
	-l : do NOT log files sent/received			default: log
	-m : Use main Sequence number for Check Messages	default: use different
	-M : File to replace "check-message" keyword		default: none
	-o : output queue for get files			default: spool/2go
	-O : offline queue for send files		default: spool/ftp_offline_(-i name)
		where (-i name) is the name of the input spooler (normally '2ftp')
		See also -Y do NOT move un-sent files to the offline queue
		Note that if -W (do NOT watch offline q) is specified, the Offline queue
		is just that specified and the Input queuename is NOT appended.
	-p : Default control port number (for remote)	default: 21
	-P : Minimum data port number			default: 32800
	-q : quiet mode - do not message if we find directories or other
		non-files are found in the input queue - just ignore them.
					default: log message if directory found
		For get files this will NOT message if no file(s) are found
		or the file is to be skipped.
	-R : round robin if more than one host specified	default: no
	-s : Default Remote host name			default: none
	-S : log the trace of each transaction		default: do not
		This generates a one line log of each file sent is stored in
		log/remote_trace with a name of 'date_(DF)'.
	-t : network file wait for files arriving	default: no wait
		from across a network - using NFS perhaps
	-T : default timeout for messages back from	default: 120 secs
		the remote system. Shorten this for quick or
		internal networks to 20-30 secs or shorter.
		This MUST be between 10 and 3000 seconds.
	-V : make all connections Passive / PASV		default: no
		This should be used with care as not all FTPd's can handle PASV connections
		It is better practice to use the parameter 'passive-connection:' for
		those sessions that must be PASV than to use -V for all.
	-W : do NOT scan the offline queue for sends		default: do
		Use this flag where a 2nd 'ipftp' (or other program) is scanning the
		offline queue of this ftp.
	-x : default no of files to send (or get) before closing the connection	def.
all
		This is overridden in the parameter file by 'onefile' or 'maxfiles'
	-y : (secs) wait this time if an error occurs
		AND there is no offline queue at all		default: oh yes there is
		On error, wait this number of seconds but do not ignore any files
		Ie keep going, but pause after an error.
	-Y : there is no offline queue at all			default: oh yes there is
		and on error, handle files in the normal files (ie ignore all files for the
same address for 60 secs)
	-Z : force all traffic to use the default parameter file
		(ie either FORMAT or the '-z' switch)	default: use DF
	-v : print version no and exit

-------------------------------------------
Notes ...

-- For a single shot GET - getallfiles or whatever - try
	ipftp -G 0 -z jerry
where 'jerry' is the name of your parameter file

-- When Sending - Watch out for remote machines which cannot handle long
filenames. Use 'newname' to clean that up.

-- When Sending - Watch out for remote machines which cannot handle certain
chrs in filenames. Use 'formatname' to clean that up.
Especially 'formatname:nt' for Win2K boxes (which HATE ':' etc) or older Macs
(which needed the filename trimmed to 31 chrs)

-- When Sending - Watch out for duplicate file names on the remote box. Often
you may want to delete the file of the same name first or use 'uniquename' to
force the remote server to accept it.
	eg	ftpbeffile:dele SN
	or	uniquename:

-- When Sending - if using 'uniquename' (which is an FTP STOU command) and
something goes wrong, some remote hosts will leave an empty file with another
seqno extention in the same queue.

-- Any commands specified by 'ftpbefore' are done once only before the first
file of a series of files all with the same Parameter file.

-- Note that in the standard FTP on older versions of Win2K, there is a
'feature' that says you HAVE to wait a few milliseconds after a 'delete', so
there is an extra FIP ftp command called 'fipdelay' which will wait a second
before continuing.
	ftpbeffile:dele /tmp/pporange/oinky
	ftpbeffile:fipdelay
	ftpbeffile:fipdelay
	ftpbeffile:fipdelay
Note from version 18i2 'fipdelay can have a number after for the number of
seconds to wait which is in the range 1 to 60. eg to wait 25 seconds :
	ftpbeffile:fipdelay 25
This is also useful where the remote system is slow(ish) and the files to grab
are large and may take more than a second to build - Audio, Video and some
pix/jpegs for example - so add a second of two to the 'ftpbeffile'.. just in
case.
	ftpbeffile:fipdelay 2

- On old Windows2K servers, there is also another feature where the Server
loses all the permissions on a drive. Which means you can only create
zero-length files. Microsoft says the problem does not reappear if you reboot!

-- For GETs and SENDs FipHdr field 'EN' should be the filename on the remote
system. This can be changed using the "newEN" parameter.
For GETs a second field 'E1' (and 'ZZ' if required) will be just the filename
with no path information.
Note that both these fields have been cleanedup of funny chrs which are bad for
the local server.
So for GETS, another field 'ZO' will hold the name as it was on the remote.

-- There is also a means to add a flag file - a unix 'touch' - using an extra
FIP ftp command called 'fiptouch' which will create a zero length file.
	ftpafter:fiptouch /trigger/incomingFile/EN

-- An extra FIP Ftp command 'fipblockfile' will check to see if a file of this
name is on the remote server. If it is there, the file to send is NOT sent.
	eg	ftpbefore:fipblockfile stop.all
	If there is a file called 'stop.all' on the remote, NO files are to be sent.

-- An extra FIP Ftp command 'fipallowfile' will check to see if a file of this
name is on the remote server. ONLY if it is there, will the file be sent.
	eg	ftpbefore:fipallowfile ok.txt
	If there is a file called 'ok.txt' on the remote, files will be sent.

-- An extra FIP ftp command 'fipdelete' will delete ALL files in the current
folder of the remote server. Use with Care ! Make sure you are in the right
area first. In fact you may wish to use a mask to make sure you are only
deleting certain files:
	eg	ftpbefore:fipdelete *.txt

-- For continuous feeds where data is dribbled to a/several remote hosts, Check
Messages allow you to notify the remote site that everyting is still working.

-- For Check Messages to be sent, an interval (-C) must be specified. In each
parameter file the 'check-message' string can have the message to send OR the
default will be used OR the contents of the (-M) check message file is sent.

- Using 'get-and-zap', 'get-and-zapall'  and 'get-and-zapall-except'. These
three commands will get a file(s) and delete off the remote server once it has
been retrieved. BUT you MUST be very careful that you are in the right folder
in order to do this - NEVER use 'root' or 'administrator' as the logon! or you
risk pulling files from the root/top folder and totally damage the remote box
!! Usually a specific logon with very restricted access is used for these
commands.

- Note that if there are multiple 'get-and-zap's and 'getfile'/'getallfiles'
then it is the last specified which determines whether ALL are zapped or not. -
Basically the message is do NOT mix Zaps with Non-zaps in the same parameter
file.

-- An extra FIP ftp command 'fipduplicate' will copy a file a second time with
a different filename. It can ONLY be used for Send and in 'ftpaftfile'
	eg	ftpaftfile:fipduplicate EN.copy
This differs from 'fiptouch' which creates a zero-length file as the WHOLE data
is resent.

-- Individual commands may be considered more important - such as :
	ftpbefore:cwd /sgt/bilko
If the folder does not exist, you normally wish to STOP.  But if you have a
'mkd' beforehand, it will work the first time but not thereafter
So the fip builtin 'fipon-error' can be used BEFORE the line to set how to
process :
	ftpbefore: fipon-error:ignore
	ftpbefore: mkd /sgt/bilko
	ftpbefore: fipon-error:abort
	ftpbefore: cwd /sgt/bilko
or from version 18i, these can be replaced by the fipcd builtin which will
mkidr if the folder does not exists and the cd to it
	ftpbefore: fipcd /sgt/bilko

- Always rename files (ie RNTO/RNFR combinations) using the 'ftpaftfile'
keyword  and NOT 'ftpafter' because by the time 'ipftp' gets to the 'ftpafter'
statements it is not at all bothered whether any files were sent or not!
For ordinary FTP - and FTP over SSL use :
	newname:SN.tmp
	ftpaftfile:rnfr	SN.tmp
	ftpaftfile:rnto SN.gogogo
For SFTP (over SSH) you can (only) use the builtin 'REN (oldname) (newname)'
	use-ssh:yes
	newname:RP$z.tmp
	ftpaftfile:REN RP$z.tmp SN.pdf

- Directory lists of the remote folder
	ftpbefore: cd /pub/secret/EQ
	ftpbefore: fipdir
	ftpbefore: fiplongdir /fip/web/pages/junk/SHOWME.TXT
	ftpbefore: fipstddir /fip/web/pages/junk/STDDIR.TXT
'fipdir' gives a short and 'fiplongdir' a long list in whatever format the
remote system gives !
There can optionally be a filename after the 'fipdir' to create/replace a file.
Also 'fipbefore:fipstddir will do the same but in the system-independent format
as used by MLSD. Note this will only work on servers that can process the
enhanced MLSD command !

-- GET - There is a peculiar quirk of some FTPs - often on Linux or Win2k -
that will display a complete tree of all files and sub-folders under a given
folders. 'ipftp' assumes you do NOT want to drill down this list and, by
default, will only pick up files from the top folder specified. If you do need
to drill down, use parameter 'walk-remote-folder-tree:'.
	USE WITH CARE - you could cwd/cd to root on such a box and start pulling over
EVERY FILE ON THE DISK !!!

-- For Mac resource forks, 4 parameters specify :
	- where a template is to use as the basis
	- what software is being used - nfshare, ethershare, ushare
	- the file type
	- the path for the res fork from the data file
	eg
	resfork-path: .rsrc/
	resfork-type:ethershare
	resfork-filetype:JPEG
	resfork-creator:FIPO
	resfork-template:rsrc_template

The 'resfork-type' can be 'ethershare', 'ushare' or 'nfsshare'.

Note that, except for resfork-type,  none of these parameters are converted so
case is important.

The FileType and Creator are NORMALLY 4 uppercase letters/numbers and default
to TEXT and FIPO respectively.

-- Proxy

To use ipftp thru a proxy server, there is an 'ftpproxy:' parameter to add the
extra information.

This uses the same syntax as the 'ftpbefore' etc above.

The flow is that you connect and logon to the local proxy server first..
.. then put something in to tell it where the remote FTP server is.

Generally there are two type of Proxies
	Case 1 - where only USER and PASS are needed.
	Case 2 - where ACCT is needed.

Case 1 - 'ftpproxy' is used to state the actual IP address or hostname of the
proper remote host is added to the USER after an '@' sign.  USER and PASS then
give the logon and password for the remote.
- 'remhost', 'logon' and 'password' should be the proxy server
hostname/logon/password
- 'ftpproxy' holds any logon and password of the remote
	eg	ftpproxy:USER abcde@195.13.83.99
		ftpproxy:PASS eyeAteFigs

Case 2 - the word 'proxy_server_user' is added as an extra word on the logon
- 'remhost'	is the proxy server
- 'logon' 	is the REMOTE logon '@ REMOTE host
- 'password'	is the REMOTE password
- 'ftpproxy' uses ACCT to hold the password of the PROXY server

eg
remhost:proxy_host
logon:remote_logon@remote_host proxy_server_user
password:remote_password
ftpproxy:ACCT proxy_password
; use log to check what is REALLY happening !!
logfile:/fip/log/ftp/FIPTEST.$D

and the log file gives
ftp> open proxy_server

Connected to proxy_server.
220 Blue Coat FTP Service

Name (proxy_server:fip): remote_ftp_server_user@remote_ftp_server
proxy_server_user

331 Enter password.
Password:(remote_ftp_server_password)

332 Enter proxy password.
Account:(proxy_server_password)

230 User xxxx logged in.  Access restrictions apply.
Remote system type is UNIX.
Using binary mode to transfer files.

... etc

-- Secure FTP --------------------------------------

Confusingly there are two - completely different - 'Secure' FTPs plus a defunct
company called SecoueFTP etc etc
	1. a more secure version of ordinary FTP which uses SSL/TLS in the same way
http and httpS work for secure web sites.
	2. a file copy layer which sits ontop of SSH - which has nothing at all to do
with ordinary FTP

The extra confusion is that SSH uses SSL - so dont mix the two up !

How do you know which is the one you want ?

- What is the port number on the remote server ?
	port 21		- it is BOTH normal FTP and the SSL/TLS version (port 21 is the same
as normal FTP)
				test with ordinary 'ftp' client
	port 990	- it is ONLY SSL/TLS version
				test with 'telnet' to (remhost) 990 and cut the connection once you are
satisfied it connects
	port 22		- it is ONLY the sftp on top of SSH
				test with 'ssh' or 'sftp'

--- 1. SSL/TLS
	This is the easy one..
Prerequsite is the SSL layer which nowadays comes as standard on most
platforms. Otherwise it can be downloaded from the installation kit or the
website of the OS - Sun for Solaris, RedHat, Suse etc
	http://www.openssl.org
For *nix, if you have gcc installed, it is usually easier (!) to compile from
the latest sources at http://www.openssl.org/source/
For Win2k, there are precompiled versions at
	http://www.openssl.org/related/binaries.html
which points at
	http://www.slproweb.com/products/Win32OpenSSL.html
NOTE you generally have to also add the 'MicroSoft Visual C++ 2008
Redistributables' (vcredit)
Pick the 32bit Light version unless you are running very very high volume
stuff.

To get ipftp to use SSL/TLS, all you need to do is add the keyword
	use-tls:(param)
where param is
		no		- normal, standard FTP on (normally) port 21 for the control (ie DO NOT
use SSL/TLS)
		yes or explicit - connect (normally) on port 21 in clear then use SSL/TLS for
USER, PASS and data
		auth		- connect (normally) on port 21 in clear, use SSL/TLS for USER, PASS
then return to clear
				 (but use SSL/TLS for all data) This is normally the only version which
works if using a Proxy server or a really nasty Firewall
		implicit	- connect (normally) on port 990: use SSL/TLS for all control and
data

If passwords and/or certicates are required, use the ssl-cert etc to add

--- 2.a SSH/SFTP

The Fip SFTP is layered over OpenSSH (www.openssh.org) in a similar way to
'sftp'. So before using 'ipftp' in secure mode, openssh will need to be
installed by your or your system manager.

OpenSSH is available either as an installed option on most current Linux/Unix
builds or prebuilt binaries can be downloaded (for Solaris,
www.sunfreeware.com, Linux redhat etc, AIX www.bull.de).
For Win2k - watch this space - the implementation is not tested or tuned yet.

Once OpenSSH is installed, use the 'sftp' program to check it works with the
remote site.

Then to use secure ftp in 'ipftp', add the parameter
	use-ssh:yes

Note that the commands for 'ftpbefore'. 'ftpbeffile' etc are restricted to
those available in SSH/sftp

---- 2.b SSH/Sending to a new system using sshpass

For most modern systems, use 'sshpass' (it can be downloaded from
sourceforge.net) - beware it will not work on old versions of some OS as it
needs a certain minimum version of OpenSSL. This layers the sshpass in the
parameter file eg :
	sshpass-program:/usr/local/bin/sshpass

The passwd is added from the 'passwd' parameter.

See the doc on sshpass for which parameter to use - and change the paths for
both sshpass and ssh to what they are on your system.

---- 2.c SSH/Sending to a new system using shared keys

If sshpass does NOT exist (and cannot be loaded for some reason) to make
'ipftp' work using SSH, the remote site needs to let you in without prompting
for a password.

This entails generating a pub key on one system and installing it on the other
- and then generation a public key on the other and copying it to the first.

You should read (and digest) the man pages for the ssh-keygen to work out what
data needs to be added to which files. But generally for Unix/Linux, the
default enc in 'rsa' so :
	 the public key is in ~/.ssh/id_rsa.pub
The contents of this file needs to be added (without changing, adding NL,
spaces etc) to the file on the remote system called :
	   $HOME/.ssh/authorized_keys
and in some cases
	   $HOME/.ssh/authorized_keys2
Email or somehow send it to the remote site but do not  Note the key is
normally too long to be edited, so email the whole file rather than
view,cut,paste.email as that tends to mangle lineends. The '2' version should
really have just the 'dsa' and 'rsa' but it does not seem to matter.

Example - assume the hostname of the Fip sending system is 'sendersys' (logon
fip) and the remote host is 'remotesys' (logon pif)
	1. On sendersys system, logon with the same logon as Fip runs under - in this
case 'fip'
		Generate the 3 different keys using ssh-keygen, cat them together and send
them as a single file to the remote
		(some systems do not use the -t switch for ssh-keygen, so 'man' or 'howto' to
find what it should be)
			ssh-keygen -t dsa
			ssh-keygen -t rsa
			ssh-keygen -t rsa1
			cd
			cd .ssh
			cat *.pub > sender.k
			scp sender.k remotesys:
				use scp or sftp for this as it will add the remotesys into the
'known-hosts' file too.

	2. On remotesys, logon with the same logon as Fip  - in this case 'pif'
		ssh-keygen -t dsa
		ssh-keygen -t rsa
		ssh-keygen -t rsa1
		cd
		cd .ssh
		cat *.pub > remote.k
		scp remote.k sendersys:

	3. On sendersys system, create or add to the 'authorized_keys' file
		cd
		cd .ssh
		ls						just to see what is there
		cat ../remote.k >> authorized_keys		to add to an existing keys file
	or	cat ../remote.k > authorized_keys		if it does not exist
		cp authorized_keys authorized_keys2		some older versions need this
		chmod 644 authorized_keys*			sort out permissions or the keys files
		chmod 755 .					.. and the .ssh folder

	4. On remotesys system, create or add to the 'authorized_keys' file
		cd
		cd .ssh
		ls						just to see what is there
		cat ../sender.k >> authorized_keys		to add to an existing keys file
	or	cat ../sender.k > authorized_keys		if it does not exist
		cp authorized_keys authorized_keys2		some older versions need this
		chmod 644 authorized_keys*			sort out permissions or the keys files
		chmod 755 .					.. and the .ssh folder

	5. try it using ssh on the sendersys
		- first using just the hostname
			ssh pif@remotesys
			if you get a 'Add to known hosts message'
				- type 'yes'
				- then once logged on, exit and retry
		- second with host.full.internet.com
			ssh pif@remotesys.hoho.com
			if you get a 'Add to known hosts message'
				- type 'yes'
				- then once logged on, exit and retry
		Do you get prompted for a password - NO=it worked, YES=back to the drawing
board.

Some versions of openssh do not work automatically if the main folder for that
logon does NOT have the correct permissions 'drwxr-xr-x' (chmod 755 .) or is
owned by another logon like 'root'. If you are the only user of that folder,
just change it - if not you will have to 'negociate' with the other users....

BEFORE running ipftp, 'ssh' must be able to connect and logon WITHOUT a logon.
So if it does not, go back to the doc for you system as it may differ slightly
to that described above.

If you are unsure whether the ssh is working correctly, ssh debugging can be
turned on by adding '-v -v -v' to the command line.
	ssh -v -v -v pif@remotesys

Similarly ipftp can be put in a similar mode using the 'ssh-debug:yes'
parameter and 'ipftp' and 'ssh' both spit out reams of meaningful information.

If it does not work (!), it could be that 'ssh' is in a different folder ipftp
looks in /usr/bin first and then /usr/local/bin - and if not found gives up 
with an error. Use 'ssh-program:/your/path/tossh/ssh' to state where it really
is.

If you need to use a non-standard ssh program or set of options, the defaults
can be changed. The defaults are below - note the use of single quotes around
each option.
	ssh-program:/usr/bin/ssh
	ssh-options:-q -oForwardX11=no -oForwardAgent=no -oProtocol=2"
	Plus ...
	if the number of connection-retries is NOT 5	-oConnectionAttempts=(number)
	if the remote port is NOT 22			-oPort=(number)
	if the timeout is NOT 120 seconds		-oConnectTimeout=(number)
	if you are connection to newr systems, try	-oBatchMode=yes
	For proxy connection, try			-oProxyCommand='nc -v -x(proxy host):(proxy port)
%h %p'
(pre 18j59 these options defaulted to :
	ssh-options:-q '-oForwardX11 no' '-oForwardAgent no' '-oProtocol 2'

If you are having problems with the SSH link - bad dns, flaky network etc - you
can specify a log filename for just the SSH messages.
THIS FILE CAN GET VERY large - so pls remember to turn off (and also use FipSeq
to make day/hour files)
This file is in addtion to the normal fip 'logfile'
	ssh-debug:yes
	ssh-debug-logfile:/fip/log/ftp/SSH_NAME_$e$y%i$d.log
There is also a ssh-debug-loghead: for adding more information at the start of
exach transmission.

--- Running parallel streams and an error stream

Use ipwheel, ippost, iprobin or just the USERS file to route data into any of
the 4 (or so) main ftp queues, and use -W, -Y and -O to handle errors.

ftp1	ipftp -i 2ftp1 -O ftp_error -W
ftp2	ipftp -i 2ftp2 -O ftp_error -W
ftp3	ipftp -i 2ftp3 -O ftp_error -W
ftp4	ipftp -i 2ftp4 -O ftp_error -W
ftperr	ipftp -i ftp_error -Y

--- Troubleshooting a problematic connection

To find where the problem lies ..  our checklist is
- Try it manually first to check it is ok - INCLUDING sending a file to make
sure permissions are ok
- Always try it manually from the Fip server to make sure the Firewall is ok
too.
- Use the log file parameter and see what it brings up.
- check with both active and passive and check timings - which works better.
- Check both servers are running FULL-duplex
- if it still failing test at the TCP level with snoop/tcpdump/wireshark (which
can lead to total obfuscation as you get too much info)

The log file does give you everything sent and received on the Control Port
plus the first chunk of data on the Data Port
- so you can tell :
	-  if the FipHdr has been stripped off by looking at the first part of the
file sent part.
or   - if the ftpbefore has worked - as the CD or whatever will be noted with
the reply (positive or neg)

The FULL-duplex one is quite a tricky one as most network programs work of in
half-duplex, including the FTP control port. The symptom is that small files
are FTPed with no problem but larger ones (generally over 200K) always bomb
out.

With most servers being auto-sensing, all it takes is a network glitch for them
to drop down to a slower speed or half-duplex - and they dont always readjust
upwards when it has cleared. (On a sparc, /sbin/mii-tool is a useful thing to
hack with)

For the Active/Passive thing, if a firewall is eating the data connection in in
one direction, it is quite hard to diagnose - except for any 425 error messages
(which are always generated on the side which is starting the data connection,
and so may not be ipftp).
So swopping between passive and active is definitely one thing to try.
The difficulty is when FTPing thru both your firewall and the remote client's
to get to the remote server.
One of the Firewalls MUST be setup to allow either the High port connections or
traffic on a specified fixed-data-port.
Note	- active means the remote server will start the connection to the client
(ipftp in this case) for any data (LS or FILE)
	- passive means ipftp will start the connect to the remote server for any data
(LS or FILE)

---- Sending all files in a folder .. and stopping

Use -A for where you want all the files from a folder to be sent - and then
ipftp stops/exits.

The wrinkle is - what happens if the connection fails; we want ipftp to keep
pounding away until all the files have gone.

It gets used for things like sending all 900 files to Amazon for a Kindle
edition for example - where the chances of a network blip in the early hours of
the morning are quite high, and we need ipftp to reesablish the connection and
continue.

Try it with no offline queue : (where QD is a date like QD:20130126)

ipftp -A -i /fip/data/ebooks/QD_kindle -h'WD:QD' -Z -z goa_kindle -y 10 -Y

------------------------------------------------------------------------------
Version control
;018j60	29dec09	;1-3 added TLS/SSL and fipstddir ;4 bugette for balance and
programname ;5 %q is time taken
	;6 08mar10 added tracker-script ;7-8 added ConnectionAttempts an
ConnectTimeout to ssh/sftp
	;10-11 16apr10 only read pram file once for GETS
	;12-17 7jun10 added error message in drain to describe what ! (15 added
linger-on-close) ; 16 better wilds with SSH ;17 TLS'C'
	;18 25jul10 bugette with -o not adding /fip/spool
	;19 10aug10 added rename-prefix and rename-newname for SFTP
	;20 29oct10 bugette with false error msg for gets
	;21-23 10nov10 GET-skips fudge for W2k FTPD bug ;22 %c bytes RCV or TX
	;23 added -A onePass and doneque is now FipSeq and fixed GENERIC_GET bugette
	;24 added send-external-fiphdr:yes/no
	;25-31 redid keepConnectionOpen so it did NOT logon each time /
do_connect-OpenData
	;32-33 6dec11 woops remport NOT working for ssh and added remote-folder-exists
	;34 12*425 is an error ;35 sshpass now works
	;36 added list-folder-only plus added an error if Unknown keyword in parameter
file
	;37 15feb12 added max-data-port and min-data-port (min is same as
start-data-port)
	;38 20feb12 woops TLS/SSL only worked 'onefile:'
	;39 $c and CTRL bugette
	;40 6jul12 added hidden 'CD ~' if 2 sequential files have different Parameter
files but the same host/logon
	;41 4sep12 added max-single-fiphdr-size
	;42 21sep12 added fiphdr ZO original filename
	;43-45 10oct12 added timeout alarm for sshpass plus added sshSftpVersionNumber
	;46-47 17jan13 added send-multiples ;48 added isHash for Renames ; 49 cleanup
; 50 outque is now parseable
	;51 added get-copy-file ; 52-53 tuned
fipdelete/fipduplicate/cd/fipallowfile/fipblockfile for spaces in filename
	;54 5dec13 fixed max-files for GETS too ;55 added file_trace ;56-57 minor
logfile additions
	;58 8aug14 passive-force-address added
	;59-60 5sep14 added ssh-debug-logfile plus sshpass quotes to " not ' ** needs
fipssl 01d
;018i35	12dec06 added skip-purge-after
	;b 18dec06 PASV and 425
	;c 09feb07 ssh wait cleanup
	;d-g 23mar07 added -x default max files and added fipallowfile
	;h 20sep07 small internal chg to link with StingRay FTP Server - no PORT/PASV
on logon
	;i3 24oct07 added 'fipcd' and fipdelay can now have a noOfSecs after
	;i4  1feb08 redunBalanced
	;i5-10 19feb08 added send-external-file-name ;i6 max skip files upped
1000->20000
		;10-11 27may08 bug in fipdelay (from i3) and some remote servers do NOT allow
skipdetails
		;12-17 5jun08 added rename-prefix, rename-newname and BUG in PASV ; 17 for
log-level:
		;18 -justskip added
		;19-21 added log-skip-details:(and stackoverflow doc)
		;22-25 note_balance_action
		;26-27 added -H for cluster host name and allow ZZ to be used in
get-extra-fiphdr
		;28-35 14sep09 check if proxy for failure and added minimum-poll-interval ;
30 HE+HR in GET FipHdr
;017z	24jul05 made H1-3, DR, DG, DT available to script, added -k and -y and
script for GET
	;b-d 12aug05 cleaned up all those extra PORT/PASV on skipped files
	;e 15sep05 keep-connection-open added for sending
	;f 14oct05 STOU/uniquename need tuning
	;g 21oct05 Reworked ZeroLen GET files and ignore - now we do NOT add to skip
notr zap/move etc
	;h 23nov05 -W and -O mean that the offline queu is that specified and '2ftpX'
is not appended
	;j-k 15dec05 added shadow skip file - possibly on another drive. and added -K
for
	;l-o 09jan06 added add-md5-signature and cleanup of getfiles
	;p 23jan06 added fixed-data-port
	;q-r 23feb06 added put-empty-files (allow format: same as formatname)
	;s-t 28jul06 added remote_trace style timings to send file done msg as H4
	;u 10aug06 wait3 64 bit and added -X and timing-stats
	;v-w 07sep06 added priority-sends plus bugette with fip-onerror
		plus for GET, must have a dest:, so added a default.
	;x-y 04oct06 new DestRedun
	;z 06dec06 added start-data-port
;016z	10aug04 add skips to old too
	;b 24sep04 speedy
	;c-h 27oct04 added ssf-secure shell stuff
		(d,h 19nov04 hash-in-filename added)
	;i-j 31jan05 better Win2K Gets
	;k-m 07feb05 'dests' can be FipSeq and added outque:
	;n-p 23apr05 remote-done-que is now FipSeq
	;r-v 13may05 redid old skip list for WINNT and added -B for skip-balance-group
	;w-x 15jun05 added -Y there is no offline queue
	;y-z 30jun05 only go PASV if no Data port is open
;015z	06jun02 better control over filename.....
	;a/b 18jun02 WINNT get was broken
		plus AS400 gives a funny file ended message.
	;c/d 28jul02 added ignore-all-preparation
	;e 12sep02 added creating ResForks on the remote system
	;f 15sep02 bugette for GET with many entries - may miss the last ones
	;g/h 10oct02 bugette with zero length files plus added
		check-primary-server-for-getfiles: and balance-skip-files
	;i 16oct02 to strip the './' at the begining of WarFTP files.
	;j 23jan03 added inc/min/max Seqno
	;k 30jun03 bugette - skip file not zapped when nothing there.
	;l-m 12sep03 redid timeout on connect
	;n 30sep03 added logon to checking host
	;o 06oct03 quiet on DataPorts.....
	;p 12dec03 added -h for extra-fiphdr on GETS
	;q 06jan04 default formatname for Gets
	;r 19feb04 small mod - comments only at start of line
	;s 26feb04 better skiplist
	;t-w 25mar04 added FTP_LIST_FILE: or -F and fipdir
	;y 29jun04 added -O offline queue
	;z 02jul04  bugette in PASV/passive sends
;014z	08sep00 added minimum-log
	;a 29nov00 improved logeachmsg
	;b 20dec00 bugette for the case of maxattempts exceeded and no errque
	;c-d 10jan01 bugette in fiphdr:
	;e-i 30jan01 cleanups
	;j 27mar01 better exit codes for single-shot and errors
	;k 10jun01 added FTP_EXTERNAL_FILE: and FTP_ZAP_EXTERNAL:
	;l-m 12jul01 do NOT normally walk the remote folders for getfiles.
	;n-o 25jul01 added no-fiphdr-on-getfiles
	;p 09aug01 PASV/passive-connection tuned. and -V added
	;r-u 17aug01 added 'except's
	;v 01oct01 fipdelete mods for 'fipon-error'
	;w 16jan02 added loglasterrfile/log-last-error-file
	;x-y 14mar02 added remote-trace and zap DO on attempts from filename
	;z 30apr02 added script plus FTP_FILE/BINARY_BEFORE/AFTER
;013b	15aug00 added skip-file plus possible buggette in GetFiles and lousy
networks
	;a 19aug00 bugette in get-extra-fiphdr
	;b 05sep00 added log of skip files
;012e	03apr00 bugette in LOG file - could get confused with many files in
offline all withh different log files.
	;a 26apr00 chkmsg seqno not right
	;b 01may00 NT getfiles was buggy
		added fipon-error/on-error and maxattempts and slowdown
	;c 17may00 added maxfiles:
	;d 21jun00 better handling of the Done Queue
	;e 09aug00 added 'zapresforks'
;011g	30nov99 added ftpproxy
	;a 06dec99 mod to drain_it - occasionally missing a result code.
	;b-c 10dec99 v occasionally, may miss change of seqno
	;d-g 14jan00 allow space in NT filenames - redo seqno

(copyright) 2014 and previous years FingerPost Ltd.