ipftp
ipftp
This program loops around a queue and sends or gets or lists files to/from a
Remote Server via FTP.
As FTP does hang on (regular) occasion, no response for a timeout period will
abort that file or transmission and the program attempt to restart where
appropiate.
By default, the program can be spooled against a queue or kicked off for a
single file or queue from a script using the '-1' input switch, which we call
'single-shot'.
For single shot, the files are NOT deleted unless the keyword 'del-files' is
specified.
Files are only deleted if sent correctly or, for getting files, if a reasonable
response was received like 'File Not Found'.
The FipHdr of each file to be sent is checked for the 'DF' field which, as per
normal for Fippies, is the name of a parameter file in tables/ftp. This
parameter file, tables/ftp/FORMAT by default, is used to describe the
transmission.
The Remote Host Name can be a Fip Pseudo-Host (ie with an entry in
'tables/sys/DEST_REDUN' ). In this case, the primary host is checked for being
down and, if so, data sent to the secondary. If that too is down, files are not
sent.
File for remote systems that are offline or un-pingable are send to
'spool/ftp-offline' which will be rescanned on a regular basis. When such a
system recovers, the program will automatically send the collected data files
form 'ftp_offline'.
Files for retrieving can be left on the remote server untouched, deleted or
moved to a specified Done folder. There is a Poll input switch which will just
look once during the period for any files and get them accordingly.
A skip file may be created and maintained automatically for GET files which we
do NOT need to re-get. This is used where a 'get-and-zap' or 'get-and'move'
cannot be used.
ipftp can also be used to merely list the files on the remote server using the
-D input switch or the 'list-folder-only:' parameter.
Syntax for the parameter file are :
; comment
remhost: Hostname on remote system
This overrides the '-s' Input switch.
defaults to that specified in the '-s' Input switch
This can be in FipSeq or just fixed text.
remport: Port on remote system
This overrides the '-p' Input switch.
defaults to that specified in the '-p' Input switch
or failing that, defaults to 21
This can be in FipSeq or just fixed text.
logon: Logon for remote system default: anonymous
This can be in FipSeq or just fixed text.
password: Password for remote system default: none
This can be in FipSeq or just fixed text.
nopassword: The remote system does not need a password
To send a password of nothing (which is different to NO pwd),
specify the keyword with no parameter :
eg password:
Normally a remote system wants a password even if it is blank.
remhost,remport,logon,password,nopassword may be repeated another 9 times for
roundrobin which is enabled by the -R input switch.
remhost:mainsys.bigdom.com
logon:main
password:passo
remhost1:backup1.bigdom.com
logon1:back
password1:passo
remhost2:backup1.bigdom.com
logon2:backo
password2:passo
ftpbefore: FTP command to execute after logon default: none
ftpbeffile: FTP command to execute before each file. default: none
ftpaftfile: FTP command to execute after each file. default: none
ftpafter: FTP command to execute before closing up default: none
Run some raw Ftp commands....
There can be several commands for each section, each on a separate line.
Note that the actual commands are NOT those from the user interface,
but the raw ones as used by the FTP protocol. For weirdos you might
want to check that the remote system will actually allow you to do these !
Valid commands include :
dele file delete ** See notes below
cwd change directory
mkd make queue
rmd remove queue
pwd get current queue
list long queue list - like an 'ls -l' or 'dir'
nlst short queue list - like an 'ls' or 'dir/w'
nlst -lrt : long list, sorted by time.
For lists, always 'cwd' into the queue beforehand.
type 'a' for ascii or 'i' for image or binary
rnfr rename file : these two work in tandem ..
rnto rnfr=old name, rnto=new name.
eg:
; Do a MKDIR (who cares if it fails) and a Chg Dir before anything
ftpbefore:mkd /pub/pporange
ftpbefore:cwd /pubmp/pporange
; before each file, make sure we delete an old version before
ftpbeffile:dele \SN
; at the end do a List to see what is really there.
; the list will be in the log file
ftpafter:nlst -lt
logfile:/fip/log/ftp/FTPCLIENTS.\$d-\$m-\$e\$y
Note also there are some Fip-enhanced keywords :
fiptouch
fipdelay
fipblockfile
fipallowfile
fipdelete
fipduplicate
fipon-error
fipdir
fiplongdir
fipcd
see below for explanations on what these do !
ftpproxy: FTP command to execute after logon default: none
to logon to yet-another FTP server.
Please see the notes below
ignore-all-preparation: ignore ALL the before/after/beffile/aftfiles
and just send that RAW file out. Use this for big binary files that
need NO alterations or additions.
before: FipSeq string added before data of each file. default: none
after: FipSeq string added after data of each file. default: none
beffile: FipSeq file added before data of each file. default: none
Beffile is added after any 'before' string and before normal data.
aftfile: FipSeq file added after data of each file. default: none
Aftfile is added after normal data and before any 'after' string
binary-beffile: binary file added before data of each file. def: none
Beffile is added after any 'before' string and before normal data.
binary-aftfile: binary file added after data of each file. def: none
Aftfile is added after normal data and before any 'after' string
first: Name of optional file in tables/ftp that is sent
on startup and on each scan of the queue when a new file has
been found. The text of the file is in FipSeq. default:none
last: Name of optional file in tables/ftp that is sent when all the
files in the queue have been sent and before we rescan the
queue. The text of the file is in FipSeq. default:none
check-compare-file: For Grabbing, Compare the contents of this file BEFORE
checking for new/changes default: none
Use this for timestamp file which is changed on the remote when there are new
or corrected data files available.
hostname: The hostname for the system 'ipftp' is running.
This defaults to the hostname the system was booted with.
However for systems with multiple ethernet addresses, this is used to tell
the remote system the IP address to use.
(Note for versions 06c+, this is no longer required).
del-files: Delete the files once sent. This is for -1 single file only.
default: for -1 send one file, do NOT delete input file
default: for spooled folder, delete ALL input files once sent correctly.
timeout: Timeout period when sending for no response default:120 secs
The default timeout can be modified by the -T input switch.
If you are sending to a host which is only across the computer room, you
should take this timeout down to 10 or 15 secs.
connection-retries: No of retries if the first connection fails. The default
is 5.
Which means a remote system which has disappeared can hold up the next
traffic by up to (5) * (120 secs timeout) = 10 mins !.
passive-connection: (yes/no) This causes the remote server to enter PASV mode
(if it can).
The default is NO for 'active' if the command is NOT specified
The default is YES for 'passive' if the command is specified with no
options.
From version 18j61, NO or YES can be in a FipHdr field in the file being sent
in the FipHdr : PC:yes
eg passive-connection:\PC
passive-force-address:yes/no default: yes
Force the IP address for a Passive Data connection to be the same as the
Control.
Sometimes the remote server has a funny NAT address - use this to 'clean' it
up
linger-on-close: Timeout in secs for the data to be sent.
For slow connections, increase this;
default: 20 (secs) to allow up to 20 secs max before cutting the call
outque: For get files only, this is the output folder for incoming files
This overrides the '-o' input switch and the default is 'spool/2go'.
doneque: Queue to move original files once sent. default: none
balance-done-queue: (Balance group) Balance the done queue items
errorque: Queue to move original files if NOT sent. default: none
This is used ONLY where systems do NOT have a valid IP address because there
is no entry in the host file, or it has been missplet or, more likely, DNS has
flipped.
If the remote host is out there but the Logon is refused or, once logged on,
we were unable to write the file, the files are stuffed in 'spool/ftp_offline'
or if the input queue is NOT the default, spool/ftp_(inputque)_offline
To use this, you must specify max-attempts:(number) where number can 1
onefile: Send one file at a time to this host.
After each file, 'ipftp' disconnects, waits and then reconnects before
sending the next. default: send all files in the queue.
max-files:
or maxfiles: Send (or Get, from version 18j54) this number of files sent to (or
got from) this host before disconnecting
After so many files, 'ipftp' disconnects, waits and then reconnects before
sending/getting the next bunch.
Use this to 'throttle' a connection where too many files (in or out) might
cause a system overload or when 'ftpbefore' might change between files for the
same host
default: send/get all files
max-get-sessions: For GET, this the number of sessions (input files or -G 99)
before disconnecting
Use this when 'ftpbefore' might change between input Lists for the same host
default: keep grabbing !
logfile: Log file name default: none
This should be the full path and filename of the the log file.
The contents of the Log file are preserved and appended to.
Use this to debug new feeds.
You must make sure the queue exists for the log file as 'ipftp' will not
create it.
logfile-dest: Destination for the log file default: none
This is the Fip DU destination for a log of the transmission.
Use this to debug new feeds.
Only specify EITHER 'logfile' or 'logfile-dest' (or neither)
but not both.
minimum-log: Only log starts. stops and errors in the 'logfile'
Normally the whole session is logged.
verbose-logging: yes/no default: no
add more detailed logging in the item log ALL
dest: Fip Destination for any Directory Listing or default: none
Retreive files. This should be a valid DU in the tables/sys/USERS file.
send-no-data: Do NOT send any data from this file default: send data
Use this option to send just a headline with a before or beffile
or to place a marker in a directory in the remote host.
fiphdr: Send Fip Hdr if part of the file. default: send data only
Normally old the data part is sent and the FipHdr stripped off.
hash-in-fiphdr: (FipSeq chr) default:none
A hash/pound (#) in a FipHdr field is mapped to this chr
Normally hashes are end-of-field in a FipHdr. So it needs to be mapped to
something else and we changed it here on outbound.
newname: Newname for the file(s) in FipSeq default: as original
newname:\SN.\$Z
The default is the same as 'newname:\SN'.
Note that the 'first' and 'last' files are NOT renamed.
uniquename: Force the remote server to store the file default: no
with a new name if it is a duplicate.
Some systems allow you to give a filename, others do not.
append: Force the remote server to either append to default: no
an existing file or create a new the file
Some systems allow you to give a filename, others do not.
This is not available on all remote systems !
forcename: force the filename on the remote system to be upper or lowercase
forcename:upper default: no change
formatname: Make sure the format of the filename on the remote system
is correct for that system
options are raw, unix, mac, nt, alphanumeric,pc
default: unix on *nix and nt on Win2k
YOU WILL NEARLY ALWAYS want to format the filename as different systems only
allow a subset of characters.
Note that Win2K boxes do not like trailing dots so 'formatname:nt' also
strips them.
To preserve the filename - such as if you are using ZO - use formatname:raw
for no changes at all.
sizename: Maximum size of the filename on the remote system
eg sizename:32 default: no limit
Note that normally the max for Windows/NTFS is 127, Unix is 255 and Mac is 31
The maximum for sizename is therefore 255.
send-external-file-name: Name on the remote system for any external files
Use this parameter to send the external file separately to this name
default - if there is an external file it is sent in the one file.
send-external-fiphdr:yes/no
if there is an external file containing the data (pointed to in the
FTP_EXTERNAL_FILE: fiphdr field), it may (or may not) have a FipHdr.
Use this parameter to NO=strip it (or YES=leave it on) when sending.
default is yes to send the fiphdr on the external file
rename-prefix: (FipSeq) prefix for files that will be moved in get-and-move:
If the remote server is a Windows server, permissions may prohibit a straight
move of duplicate filenames.
Use this to make the filename unique.
It is a Prefix as often, the extension needs to be kept!
Eg rename-prefix:\$e\$y\$i\$d\$h\$n\$b_
If the original file is POPE.JPG, this will prefix the date and time to the
moved file.
rename-newname: (FipSeq) newname for files that will be moved in get-and-move:
If the remote server is a Windows server, permissions may prohibit a straight
move of duplicate filenames.
Use this to make the filename unique. Remember FipHdr E1 will have the
filename
Eg rename-newname:\E1.\$z
send-empty-files:put/send
send-empty-files:put/send
or put-empty-files:put/send
put-empty-files:ignore
For SEND, if the file is zero length or empty this parameter will allow you
to :
put or send - send the file and process as normal
ignore - skip to next file
default is send
-- Listing Folders instead of sending ..... (note the default is to SEND)
list-folder-only:
Only one parameter file is used - specify with the '-z' input switch.
Also specify '-G 0' for a one-off grab or '-G 99' for a grab every 99 secs
This MUST have at least one of the following in the parameter file:
ftpbefore:fipdir (for short listing of just the filename)
or ftpbefore:fiplongdir (for a long list in the remote servers own style)
or ftpbefore:fipstddir (for a long list in the generic style - using MLSD)
-- Getting files instead of sending ...... (note the default is to SEND)
getallfiles: Get ALL files from the remote server.
getallfiles:199* Get ALL files starting '199' from the remote.
getfile: Get a file with this name from the remote.
get-and-move:
get-and-moveall:
get-and-zap:
get-and-zapall:
getallfiles-except:(mask)
get-and-moveall-except:(mask)
get-and-zapall-except:(mask)
In this case no data is sent, as only the FipHdr is used to find the
filename.
There can be several lines of 'getfile' and variants.
Watch out for case sensitive remote systems !!
No wild cards are allowed for single file gets but you can use FipSeq :
eg ; get a file from the remote system with the same name ..
; .. as the input file on the local system.
getfile:\SN
; always get the readme
getfile:README
; get todays file which has a 8 digit day extension
style:QD $D,%.08d
getfile:News.\QD
You should also define the Fip destination DU using 'dest:',
- if not all the files all files will be sent to a DU of 'woops'.
Use the 'except' versions where the mask is used to IGNORE files.
eg get-and-zapall-except:.pl
Get all the files EXCEPT those with '.pl' in the filename
NOTE the except string is just a simple compare and CANNOT include
wildstrings/chrs
This is slightly inconsistent
eg get-and-zapall-except:*.pl
will look for a filename INCLUDING the 4 chrs '*.pl'
BUT it can be FipSeq
eg get-and-zapall-except:\$e\$y\$i\$d
Get all files EXCEPT those with todays date in format YYYYMMDD like 20081231
Commands 'get-and-zap', 'get-and-zapall' and 'get-and-zapall-except' :
get then zap each file.
Commands 'get-and-move', 'get-and-moveall' and 'get-and-moveall-except' :
get then move each file to a done folder on the remote box setup by the
'remote-done-folder' keyword.
PLEASE see below for comments on getmove and getzap.
Note that if the remote FTP server is a Win2K box, the name of the file MAY
have a '/' in it (or a Unix FTP server may have a file with a '\' in the name).
Normally 'ipftp' ignores such files as it assumes they are sub-folders which
should not be scavenged.
Use the parameter 'walk-remote-folder-tree:' to grab these - see below!
get-filter-case-sensitive:no FOR SSH connections only, you can force the
filter to be case INsensitive using this parametrer.
default is YES it is case sensitive
use-client-side-filter:yes/no
or local-filter:yes/no Filter in ipftp NOT on the remote server - use this for
use with slightly-odd FTP sites like smartfile.com who do NOT allow wildcard
gets - try the remote manually using 'ftp', check that files exists and do a
'mget *'. If it returns '550 No files to get' when it should get something, set
this flag to YES and try with ipftp.
remote-done-folder: Relative path on the remote box for the data files after
they have been GET'ted.
eg remote-done-folder:../done
remote-done-exists: Action if the file already exists in the
remote-done-folder
Options are 'replace' - replace existing with this one - (default)
or 'add-ext' - add the new version with a date_time extention
or 'ignore-new' - just delete the NEW version once the transmission has been
completed successfully
eg remote-done-exists:ignore-new
get-extra-fiphdr: extra FipHdr to attach to incoming, GET'ted files
eg get-extra-fiphdr:#ZI:#SU:REMOTE
This will tell 'ipwheel' to archive the incoming data under the archive log
REMOTE.
no-fiphdr-on-getfiles: do NOT add a FipHdr to the GET'ted files.
default is to add one with Date and Time fields etc.
add-fiphdr-on-getfiles: add a FipHdr to the GET'ted files.
merge-fiphdr-on-getfiles: add a FipHdr to the GET'ted files.
default is to add one with Date and Time fields etc.
NOTE that if there IS a FipHdr on the GET'ted files, the default is to merge
it with normal ipftp ones
standalone-filename: (for Get files, do NOT put a Fip-style filename but use
the original filename (suitably modified for the actual system - ie strip '/'
if on UNIX, ':' for Win2k)
The default is the normal Fip filename/fiphdr malarky.
eg standalone-filename:\SN_\$e\$y\$i\$d.fip
get-copy-file: (full path name in FipSeq) default:no
Make a copy of each incoming file as this folder/name
eg get-copy-file:/fip/data/raw.data/\$e\$y\$i\$d_\DF/\ZZ_\$h\$n\$b_\$z
this leaves a copy in raw.data/(date)_(format)/(filename)_(time)_(seqno)
hash-in-filename: (FipSeq chr) default:\235
A hash/pound (#) in a GET filename is mapped to this chr
Normally hashes are end-of-field in a FipHdr. So it needs to be mapped to
something else in order to preserve it.
get-empty-files:get
get-empty-files:ignore
For GET files, if the remote file zero length or empty this parameter will
allow you to :
GET - get the file and process a zero-length file as a normal file
IGNORE - ignore it completely : default is ignore
skip-files: Name of a file in /fip/fix/ftp for holding the names of files
brought over; so that only new files are received. It is
remade from the LIST on every Get ALL.
Note for unix/linix systems, the name is cases-SENSitive.
eg skip-files:fromFTP
default: none
check-skip-details: Yes/no
The default is only to check if the file-to-get exists or not. But sometimes
you need to track files which are updated - ie the name remains the same
but the contents differ. Set this option for tracking changes too.
Note it is slower and does take a bit more system resource as well
as an extra skip file in /fip/fix/skip to hold the existing file times etc.
default: no
skip-purge-after: (hours) Number of hours to keep the skip entry
default is 1. You might want to tune this :
make bigger if sites add/take off old material
reduce the time if the same link is used for differnet data
skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute
the skip file when changed (see doc on 'ipbalan')
This is oftern used where more than one system is GETTING the same remote
system/folder (usually with 'check-primary-server-for-getfiles').
shadow-skip-file: (full path/filename)
Name of a duplicate of the skip-file - probably on a network drive.
Whichever file is newest is used.
minimum-poll-interval: (secs)
minumum delay between polls for gets default: 10 secs
log-skip-details:
Log why a file is being retrieved - new file, size change etc
public-address-from-aws-meta: (FipSeq)
Access the internal AWS metadata for the public-ipv4 address
public-address-from-aws-meta:/latest/meta-data/public-ipv4
or fixed:Q1 /latest/meta-data/public-ipv4
public-address-from-aws-meta:\Q1
Either will grab the ip address at
'http://169.254.169.254/latest/meta-data/public-ipv4'
public-address:(IP address)
nat-pasv-address:(IP address)
if using a proxy or firewall, which does NOT automatically NAT internal to
external addresses, this is the external address of THIS host to the Extranet
world.
For ipftp, it is used for the DATA port of ACTIVE connections; ie the PORT
command sent to the remote server.
fixed-data-port: (number)
if the number is >= 20, fix the data channel port number to this number and
do NOT change it.
min-data-port: (number)
max-data-port: (number)
if the number is >= 20, the data channel port number will be in the range of
min <= number <= max
defaults are min-32800, max 60000
check-primary-server-for-getfiles: pseudo-host name that is specified in
tables/sys/DEST_REDUN
that is used whether the current host should be getting the files or not.
ie in the ftp parameter file REMOTEGET is
check-primary-server-for-getfiles:remotewire
and in the DEST_REDUN is
; psuedohost primary secondary
remotewire fip1 fip2
and in the SYSTEM file for both fip1 AND fip2 there is a line
rem1 local ipftp -G 600 -Z -z REMOTEGET
Then if fip1 is up, the ipftp on fip1 will always get while on fip2 it will
just check/loop.
if fip1 is down, the 'ipftp' on fip2 will start getting.
log-each-file: (dest) or
logeachfile:(dest) Send a Success/failed msg to this destination
for each file. There is no default. This log file is
just a FipHdr with the following extra fields :
DR-File Sent OK DR:ok or DR:error
DG-Will Retry later DG:retrying, DG:stopped
DT-Some message text DT:No connection
default: no log created.
The text for the DR and DG can be in FipSeq and so can contain
FipHdr and other variables. As they are FipHdr fields, please
do NOT put NL, CR etc in the fields.
Note that System Variable \$q holds the time taken for transmission.
log-last-error-file: (dest) or
loglasterrfile:(dest) Same as for 'logeachfile' but it is sent ONLY
after the last failed attempt where 'maxattempts' is specified.
Only log-EACH-file or log-LAST-ERROR-file can be specified - not both.
log-max-size: If logging is ON and a file is bigger than this size
DRgood:(text) Message for the FipHdr field DR on a successful tx
default: ok
DRbad: (text) Message for the FipHdr field DR on a unsuccessful tx
default: error
DGcont:(text) Message for the FipHdr field DG if, after an
unsuccessful tz, another attempt will be made.
default: retrying
DGstop:(text) Message for the FipHdr field DG if no further
attempts will be made as the file was sent successfully
or the maximum no of attempts has been tried.
default: stopped
fiphdr-for-logeachfile: (FipSeq) or
msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
'logeachfile' or 'loglasterrfile' msg. This should be in FipHdr
format and be in FipSeq. It can be used to pass FipHdr fields
in the outgoing file into the log file.
eg msgeachfile: DF:logdial\nSS:\SS\n
default: nothing added
stop-on-error: Stop if you get a '500' series error in those commands
you have specified in the 'ftpbefore', 'ftpafter' etc keywords.
Normally these are ignored ..
.. which is what you want if you have something like :
ftpbefore:dele \SN
to delete a similar file beforehand and the file does not exist :
550 no such file or directory
is the message received and can be ignored ..
but then the same message is also given for 'cwd' that does not work !
Where it is important, use 'stop-on-error' or 'on-error'.
This is overriden by the 'on-error' keyword - see below.
fip-syn-log: add a Item Log line to the Fip Syndication Log file log/SYN.
normally only the normal Fip log is kept up to date.
newEN: A 2 letter FipHdr field which will be the filename of the output file
or, for GET, the name of the file on the remote system.
This defaults to EN but if you are already using EN, map it to another field.
For GetFiles this can be the full pathname (especially with Lists),
so use E1 (below) for the filename only.
newE1: A 2 letter FipHdr field which will be the name of the file to GET
This defaults to E1 but if you are already using E1, map it to another field.
This field does NOT have any '/' or '\\' chrs.
(for the original name on the remote server, use the fiphdr ZO and not E1)
newEQ: A 2 letter FipHdr field which will be the name of the folder on the
remote system for SENDing. Default is none
This is usable only for the messaging back
newE2: A 2 letter FipHdr field which will be the name of the input folder for
SENDs for logging only. default: E2
newE3: A 2 letter FipHdr field which will be the name of the extension matched
for sendMultiples. default: E3
max-single-fiphdr-size: size that a single fiphdr field can be.
default is 8196-sep2012 (was 2024)
the minimum is 1024 and maximum is 31000
log-max-size: If logging is ON and a file is bigger than this size then the
first 64 chrs of each block are NOT stuffed in the log.
This is because the log file can get massive !
default is 'log-max-size:30000'
check-message: FipSeq string to replace default check message if the -C input
switch has been enabled.
The default string is "Check \$d-\$m-\$e\$y \$h:\$n\n"
failover: Used with the Round Robin switch (-R) and multiple
remhost/logon/password, this keyword will NOT round robin
but will always try the highest named 'remhost's first and, if not there,
will failover to the second etc.
offline: If Offline : wait in secs between attempts default: 60 secs
For the first couple of attempts the program will wait about 4 seconds or so
but all attempts after that will be at least 60 seconds apart.
ascii: The end-of-line (whether CR, NL or CRNL) will be converted by the
remote to whatever it requires. 'ipftp' converts to CRNL and ignores NULs
before sending.
default: binary (ie files received are the same as sent)
remote-wants-crnl: default: file is sent unchanged
Line endings are converted to CR NL and the file is sent 'binary'
remote-wants-nl: default: file is sent unchanged
Line endings are converted to NL only and the file is sent 'binary'
remote-wants-cr: default: file is sent unchanged
Line endings are converted to CR only and the file is sent 'binary'
locale: use a different 'locale' ( look at the man pages for locale)
Most computers are set to US English and never changed.
Use this parameter to customise any date/time or Chr translations
The parameter MUST be a valid locale on your system!
To find out what valid locale exist, type 'locale -a' on unix.
Eg ; Set for for Brazil, portugese
locale:pt_BR
balance-seqno: Send the Sequence number to this Balance Group
(see 'ipbalance') Use this to make sure the sequence number
is always updated on any companion systems.
balance-delete: Send a note for ipbalan/ipsvrd to delete the mirrored
file on any remote server.
balance-done-queue: (Balance group) Balance the done queue items
balance-queue: Folder under /fip/spool to leave balance files for a copy of
ipbalan. default 2balan
ce
redun-balance: balance group for redundant balance.
Note you should only use balance or redun-balance but not both.
ignore-timeouts: do NOT message if the sender timed out. Use this with
caution! We recommend you use it ONLY for GET-polling where
there may be times when the remote system dies but you do not
care to know OR you have other tools to check for the problem.
The actual message which is ignored is :
Thu Nov 4 16:49:33 ipftp !x : **Error - FROMZZ - 192.130.52.10 - NULL : **
Timed Out - no response from remote 0
on-error: ignore/abort
If we have an error from an 'ftpbefore'/'ftpbefile' etc,
should the program stop processing the file or continue ?
This sets up the default for that file.
** See also 'fipon-error' below.
The default is 'ignore'
Ths overrides the 'stop-on-error' keyword as it is more flexible
log-line: Extra information in FipSeq for the Item Log when sending
log-level:(number) adjust the amount of logging in the ALL log
More is less ! so the higher the number, the less you get..
log-level:99 is the same as input switch '-q' ie minimum logging
log-level:9 does not report every file, but does report end of tx.
default is -1 for all logging
log-get-every: No of seconds to log Get Accesses where nothing new was found
this reduces the amount of repetitive log messages.
max-attempts: (number)
maxattempts: No of attempts to send this file before stopping.
number can be 1 or more
Default is unlimited attempts and files in error are sent to 'woops' or the
'errorque' or zapped
maxattempts is only checked if the ipftp is unable to send the file or the
transmission fails
slow-down: No of secs to pause between commands (between 1 and 5)
If the remote host is old or overloaded, use this to slow down the link.
inc-seqno: or increment-seqno:
max-seqno: or maximum-seqno
min-seqno: or minimum-seqno
Set the Max and Min sequence numbers for this service
Default is min of 1 and max of 99999 and increment is 1
Use FipSeq \%Z to pull out
Note that \$z and \$s will always produce a 4 or 3 digit sequence number.
Eg min-seqno:100
max-seqno:1000
inc-seqno:20
; Start at 100 and by jumps to 20 until and including 1000
zapresforks: Path to Mac ResourceForks to zap those too (sending only)
This is the path to the resource fork from the data folder.
It assumes the filename is appended.
This is NOT valid for '-1' single shot items, only spooled.
default is to NOT zap
eg for Helios zapresforks:.rsrc/
eg for Ushare zapresforks:%
script: Optional Script to run AFTER the file has been sent successfully.
eg script:/fip/local/FTP_NEXT_STAGE \YI-\YS
tracker-script: Optional Tracking script - run AFTER script (if any)
(Note FipHdr fields E7 (log-line message) and E8 (result code 0=success,
-ve=neg)
default: none
resfork-type: ethershare/ushare
Add a resource fork on the file at the remote site (sending only)
resfork-creator: 4 letter creator type eg 'ZILA' default:FIPO
resfork-filetype: 4 letter filetype eg '8BIM' default:TEXT
resfork-template: template ResFork to copy
resfork-path: Path to resfork
eg for ethershare this is '.rsrc/'
use-ssh:yes/no
The commands are for a sftp on the remote server - see below
Both ipftp and ipftpssl may be used for SSH/SFTP or SCP
default is NO
use-tls:yes/implicit/explicit/auth/no
The commands are for a ftp running over SSL/TLS on the remote server
NOTE - ipftpssl and NOT ipftp must be used for SSL/TLS
default is NO
no - normal, standard FTP on (normally) port 21 for the control
yes or explicit - connect (normally) on port 21 in clear then use SSL for
USER, PASS and data
auth - connect (normally) on port 21 in clear, use SSL for USER, PASS then
return to clear for non-data commands - but use SSL for all data
implicit - connect (normally) on port 990: use SSL for all control and data
tls-auth: (XXX)
AUTH type for TLS/SSL default: TLS
Valid entries are TLS, SSL, TLS-C (whatever that is !) and something starting
'X-' which will be something homegrown !
NOTE that for all versions of SSL the method string is "SSL" (this string is
case sensitive according to the RFC)
eg tls-auth:SSL
ssl-method: tls tls1 tls1.1 tls1.2 sslv2 sslv3 sslv2and3
Version number to use for TLS/SSL default: 999 for current default (2 or 3)
(only the digits are significant, so add other text to make it readable)
For 'modern' connection, pls do NOT use sslv2 ! as it is deemed insecure
If default it will check the available list and pick the highest.
The default is currently 23 which on a modern server is sslv3 and tls1_2 !)
ssl-password: (password)
ssl-passwd: (password) default: none
Optional password if the handshake requires a shared secret
ssl-cert: (name of a PEM certificate file) default: none
ssl-root-cert: (name of a root PEM certificate file) defaunt: none
Optional certificates - held in tables/ssl
ssl-verify: yes/no verify certificates default: yes
ssl-ciphers: (list) acceptable ciphers
(use 'openssl ciphers' to list)
default: "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS"
throttle-speed: (no of KILOBITS per second maximum)
If the TCP pipe being used is only 64kbps, then large files may timeout.
ie a small 1k file should take under a second, but a 1 mb file will take
about 60 seconds to send normally. So if you have set the 'timeout' value at 60
secs, they it might abort before the data could reasonably be send.
Use this command to pace the link better. Note the parameter is in KBITS/s
which is the normal way of describing a link and NOT BYTES which is the normal
way of describing a file.
This does NOT stop/slow the data being sent - just delays the timeout for big
files.
eg for an E1 (2 megabit) line throttle-speed:2048
T1 (1.54 mps) throttle-speed:1544
56kb line throttle-speed:56
Line you know is overloaded - ADSL perhaps - throttle-speed:33
keep-connection-open: (secs)
For sending files, do NOT close the connection after each series of files and
keep the connection open. This means any AFTER commands (ftpafter:) are ONLY
done on error.
The parameter is the number of seconds between accesses (which is just a
'PWD') to make sure the line has not been closed prematurely by the remote end
or a network device.
add-md5-signature: (FipHdr field)
Create an MD5 signature for the outbound files (Data Part only) and put it in
a FipHdr field.
This FipHdr can then be used as a 'ftpbeffile' or 'ftpaftfile'.
default: none
send-multiples:yes
Sending multiple files that are related (eg XML companion file to a JPEG)
These files are located in a separate folder and MUST exist before sending
NOTE - the best way is to zip all the files up and send the single zip !
Otherwise you can get all sorts of problems when the transmission breaks (which
it will do for big files on occasion) half way thru sending.
send-multiple-que: (name of folder)
specify a folder where the files are - this is a must
send-multiple-key: (FipSeq)
As the folder MAY have lots of files, specify a KEY or STUBname which can be
used to find all related files
(ie if you do an ls for this key in this folder you ONLY get the files you
want)
Often the key is a part of the incoming filename, or the contents of a FipHdr
field
You must also specify ONE of these
send-multiple-all:yes
send everything matching the key
send-multiple-ext:(fileextension in fipseq)
only send files with this extension - this can be a FipHdr or FipSeq field
And the remaining parameters are all optional :
send-multiple-fiphdr: yes/no
depending on whether you need to sent the FipHdr too
send-multiple-zap: yes/no
zap after successful send ?
send-multiple-remote-name:\$e\$y\$i\$d_\$h\$n\$b_\E1.\$z
name for remote if different - might want to add a unique string - date/time
for example
Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).
They are normally specified :
fixed:QZ 1234543
partial:QT ST,3,2,U,<,>
combie:QY ep|na,(0000000)a
option:QE ep,11,7,s
repeat:QK XK,-,3
or repeat:QP PK,,4,#X
style:QS XN,%.03d
replace:QN NN abc=DEF def=GHI
newdate:QT hours-3 "\ZD-\ZM"
unique:QU XC
The input file can optionally be just a FipHdr file with pointers to where the
data resides. This can be useful where large PDFs or JPEGs or other binary
files need to be ftp'ed but you do NOT want to move/copy them inside the
system. To do this, use FipHdr fields
FTP_EXTERNAL_FILE: (full path name to file)
A single file before and binary file before (or after) may be specified by :
FTP_BEFORE: syntax - same as 'before'
FTP_AFTER: syntax - same as 'after'
FTP_FILE_BEFORE: syntax - same as 'beffile'
FTP_FILE_AFTER: syntax - same as 'aftfile'
FTP_BINARY_BEFORE: syntax - same as 'binary-beffile'
FTP_BINARY_AFTER: syntax - same as 'binary-aftfile'
Another FipHdr field can also have bearing on this :
FTP_ZAP_EXTERNAL: which deletes the external file if transmitted with no
errors (default is to leave the external file alone).
FTP_LIST_FILE: (filename or path/filname in FipSeq)
The filename is used for a new file holding a LIST of the current remote
folder. (same as -F)
Input Parameters are : (all Optional)
-i : queue to scan for input default: spool/2ftp
This can be specified as a queue under /fip/spool or
if it starts with a '/', the complete pathname.
OR -1 : name of file to send where we are sending a single file. def:none
This can be specified as a queue under /fip/spool or
if it starts with a '/', the complete path and filename.
The input file is NOT deleted in this case.
OR -G : poll remote queue for files to Get default: local spool
This parameter sets the number of seconds to wait between Polls.
Only one parameter file is used - specify with the '-z' input switch.
This MUST have at least one 'getfile', 'getallfiles', 'get-and-zap'
or 'get-and-zapall' line (or be used with the -D switch)
If the parameter is zero - '0' - then it is polled once only
If > 0, the minimum time is 10 seconds unless modified by
'minimum-poll-interval:'
OR -D : get the directory listing
Only one parameter file is used - specify with the '-z' input switch.
Also specify '-G 0' for a one-off grab or '-G 99' for a grab every 99 secs
This MUST have at least one of the following in the parameter file:
ftpbefore:fipdir (for short listing of just the filename)
or ftpbefore:fiplongdir (for a long list in the remote servers own style)
or ftpbefore:fipstddir (for a long list in the generic style - using MLSD)
-z : default parameter file default: tables/ftp/FORMAT
Less often used parameters ...
-? : prompt bettwen files. Use this to debug a feed outbound
before each file, you are prompted whether to send the next one
-4 : use socks4 to skip thru a proxy default: no proxy
-5 : use socks5 to skip thru a proxy default: no proxy
-A : spool a folder - then stop when it is empty default: keep spooling or
(-1) do single
normally you will -Y for no offlines for this
see below
-b : generate bandwidth statistics default: no
-B : default balance group for skip files default: none
(see skip-balance-group parameter)
-c : list of hosts to track for check messages. default: none
-C : Interval in secs for Check Messages default: none
If no data file has been sent for a certain number of seconds
then a Check Message may be sent. Change the text of a Check
Message with the 'check-message' keyword.
-d : done queue for sent files default: none
Normally files are deleted after sending.
-F : force all files to be JUST the List File. default: no
-h : extra FipHdr info default: none
Used for cases where external information is not in the
FipHdr of the file - system variables for example
This is generally more use for GET files rather than SEND.
It can be used for extra logging information or to fill in parameters from a
script.
EG a parameter file GENERIC.GET might have an line 'remhost:\BN'
while the script has ipftp -z generic.get -h '\#BN:10.10.20.31' etc etc
-H : alternate host name default: none
use this for checking primary/secondary for clustered systems where the
actual hostname differs from the cluster name
ie if the host name may be fipcluster-a or fipcluster-b but there is a
cluster hostname of fipcluster
-justskip : just build a skip file and do NOT download any data default:
download data and update skipfiles (if specified)
Use this to create a skip file of whatever files are available
-k : block size to send data default: 32 for 32k
This can be any number from 1 to 32 (in kilo bytes).
For clients with bad connections, smaller blocks sometimes work better.
-K : ignore the skip file and skipdetails files. default: use if specified
-l : do NOT log files sent/received default: log
-m : Use main Sequence number for Check Messages default: use different
-M : File to replace "check-message" keyword default: none
-o : output queue for get files default: spool/2go
-O : offline queue for send files default: spool/ftp_offline_(-i name)
where (-i name) is the name of the input spooler (normally '2ftp')
See also -Y do NOT move un-sent files to the offline queue
Note that if -W (do NOT watch offline q) is specified, the Offline queue
is just that specified and the Input queuename is NOT appended.
-p : Default control port number (for remote) default: 21
-P : Minimum data port number default: 32800
-q : quiet mode - do not message if we find directories or other
non-files are found in the input queue - just ignore them.
default: log message if directory found
For get files this will NOT message if no file(s) are found or the file is to
be skipped.
-R : round robin if more than one host specified default: no
-s : Default Remote host name default: none
-S : log the trace of each transaction default: do not
This generates a one line log of each file sent is stored in log/remote_trace
with a name of 'date_(DF)'.
-t : network file wait for files arriving default: no wait
from across a network - using NFS perhaps
-T : default timeout for messages back from the remote system. Shorten this
for quick or internal networks to 20-30 secs or shorter.
This MUST be between 10 and 3000 seconds. default: 120 secs
-u : logon on fip server - used when ipftp is run by root in rc.fip default:
ignore logon
-U : make all connections Passive / PASV default: no
This should be used with care as not all FTPd's can handle PASV connections
It is better practice to use the parameter 'passive-connection:' for those
sessions that must be PASV than to use -V for all.
-W : do NOT scan the offline queue for sends default: do
Use this flag where a 2nd 'ipftp' (or other program) is scanning the offline
queue of this ftp.
-x : default no of files to send (or get) before closing the connection def.
all
This is overridden in the parameter file by 'onefile' or 'maxfiles'
-y : (secs) wait this time if an error occurs
AND there is no offline queue at all default: oh yes there is
On error, wait this number of seconds but do not ignore any files
ie: keep going, but pause after an error.
-Y : there is no offline queue at all default: oh yes there is
and on error, handle files in the normal files
ie: ignore all files for the same address for 60 secs)
-Z : force all traffic to use the default parameter file default: use DF
ie: either FORMAT or the '-z' switch
-v : print version no and exit
-------------------------------------------
A Checklist for a new Grab or Push using ipftp ...
Things that you can check (you probably have, but here is a small list)
1. Which FTP ? - Is it normal FTP or normal FTP over secure link (FTPS) or
Secure FTP over SSH (SFTP or SCP )?
ipftp can handle all three on Linux or Solaris but you have to add extra
middleware (openssl and openssh) for the secure ones on Windoze. Note that for
the middle option FTP over a secure link, you must use ipftpssl not ipftp.
2. Connectivity - Test it manually using FTP (from the Fip Server - NOT from
your desktop or your home) to check if your/their firewalls have been setup
correctly. It also takes Fip out of the equation - if the manual tests do not
work, dont expect Fip to do any better !
3. Active/Passive - If the connection passes through more than one firewall,
one will have to be set correctly to allow 'High-Port FTP access'
So when testing manually, look for any message with the word 'Passive' in. You
can (normally - ie if not the dreadful Windows FTP program) toggle
active/passive using the PASS command.
Try doing a 'dir' of the remote folder - if it hangs, you have the wrong
setting.
Try doing a 'get of a file' if it hangs, either you have the wrong setting or
permissions do not allow you to get.
In the ipftp parameter file, add the line 'passive-connection:yes' if you want
to make the link passive. To make the link active, comment the line out.
4. Sub-folders - if there are any subfolders you need to check, use
ftpbefore:cd (subfoldername)
5. Permissions on the remote server - Do you have permission to delete file (if
using get-and-zapall) or rename (if using get-and-moveall) ? Test it manually.
So many sites lock things down and it is common for the remote administrators
to lock it down TOO much - especially if the data is in a subfolder !
-------------------------------------------
Notes ...
-- For a single shot GET - getallfiles or whatever - try
ipftp -G 0 -z jerry
where 'jerry' is the name of your parameter file
-- When Sending - Watch out for remote machines which cannot handle long
filenames. Use 'newname' to clean that up.
-- When Sending - Watch out for remote machines which cannot handle certain
chrs in filenames. Use 'formatname' to clean that up.
Especially 'formatname:nt' for Win2K boxes (which HATE ':' etc) or older Macs
(which needed the filename trimmed to 31 chrs)
-- When Sending - Watch out for duplicate file names on the remote box. Often
you may want to delete the file of the same name first or use 'uniquename' to
force the remote server to accept it.
eg ftpbeffile:dele \SN
or uniquename:
-- When Sending - if using 'uniquename' (which is an FTP STOU command) and
something goes wrong, some remote hosts will leave an empty file with another
seqno extention in the same queue.
-- Any commands specified by 'ftpbefore' are done once only before the first
file of a series of files all with the same Parameter file.
-- Note that in the standard FTP on older versions of Win2K, there is a
'feature' that says you HAVE to wait a few milliseconds after a 'delete', so
there is an extra FIP ftp command called 'fipdelay' which will wait a second
before continuing.
ftpbeffile:dele /tmp/pporange/oinky
ftpbeffile:fipdelay
ftpbeffile:fipdelay
ftpbeffile:fipdelay
Note from version 18i2 'fipdelay can have a number after for the number of
seconds to wait which is in the range 1 to 60. eg to wait 25 seconds :
ftpbeffile:fipdelay 25
This is also useful where the remote system is slow(ish) and the files to grab
are large and may take more than a second to build - Audio, Video and some
pix/jpegs for example - so add a second of two to the 'ftpbeffile'.. just in
case.
ftpbeffile:fipdelay 2
-- For Get files, Parameter 'check-compare-file' will GET the contents of this
file and compare it to the previous contents.
ONLY if the contents diff, will any new files be grabbed.
eg check-compare-file:.timestamp
If there is a file called '.timestamp' on the remote, the CONTENTS are checked
and if different, the folder is scanned for the changes.
- On old Windows2K servers, there is also another feature where the Server
loses all the permissions on a drive. Which means you can only create
zero-length files. Microsoft says the problem does not reappear if you reboot!
-- For GETs and SENDs FipHdr field 'EN' should be the filename on the remote
system. This can be changed using the "newEN" parameter.
For GETs a second field 'E1' (and 'ZZ' if required) will be just the filename
with no path information.
Note that both these fields have been cleanedup of funny chrs which are bad for
the local server.
So for GETs, another field 'ZO' will hold the name as it was on the remote.
(For SENDS, field E2 will have the input folder name for Logging purposes only)
-- There is also a means to add a flag file - a unix 'touch' - using an extra
FIP ftp command called 'fiptouch' which will create a zero length file.
ftpafter:fiptouch /trigger/incomingFile/\EN
-- An extra FIP Ftp command 'fipblockfile' will check to see if a file of this
name is on the remote server. If it is there, the file to send is NOT sent.
eg ftpbefore:fipblockfile stop.all
If there is a file called 'stop.all' on the remote, NO files are to be sent.
-- An extra FIP Ftp command 'fipallowfile' will check to see if a file of this
name is on the remote server. ONLY if it is there, will the file be sent.
eg ftpbefore:fipallowfile ok.txt
If there is a file called 'ok.txt' on the remote, files will be sent.
-- An extra FIP ftp command 'fipdelete' will delete ALL files in the current
folder of the remote server. Use with Care ! Make sure you are in the right
area first. In fact you may wish to use a mask to make sure you are only
deleting certain files:
eg ftpbefore:fipdelete *.txt
-- For continuous feeds where data is dribbled to a/several remote hosts, Check
Messages allow you to notify the remote site that everyting is still working.
-- For Check Messages to be sent, an interval (-C) must be specified. In each
parameter file the 'check-message' string can have the message to send OR the
default will be used OR the contents of the (-M) check message file is sent.
- Using 'get-and-zap', 'get-and-zapall' and 'get-and-zapall-except'. These
three commands will get a file(s) and delete off the remote server once it has
been retrieved. BUT you MUST be very careful that you are in the right folder
in order to do this - NEVER use 'root' or 'administrator' as the logon! or you
risk pulling files from the root/top folder and totally damage the remote box
!! Usually a specific logon with very restricted access is used for these
commands.
- Note that if there are multiple 'get-and-zap's and 'getfile'/'getallfiles'
then it is the last specified which determines whether ALL are zapped or not. -
Basically the message is do NOT mix Zaps with Non-zaps in the same parameter
file.
-- An extra FIP ftp command 'fipduplicate' will copy a file a second time with
a different filename. It can ONLY be used for Send and in 'ftpaftfile'
eg ftpaftfile:fipduplicate \EN.copy
This differs from 'fiptouch' which creates a zero-length file as the WHOLE data
is resent.
-- Individual commands may be considered more important - such as :
ftpbefore:cwd /sgt/bilko
If the folder does not exist, you normally wish to STOP. But if you have a
'mkd' beforehand, it will work the first time but not thereafter
So the fip builtin 'fipon-error' can be used BEFORE the line to set how to
process :
ftpbefore: fipon-error:ignore
ftpbefore: mkd /sgt/bilko
ftpbefore: fipon-error:abort
ftpbefore: cwd /sgt/bilko
or from version 18i, these can be replaced by the fipcd builtin which will
mkidr if the folder does not exists and the cd to it
ftpbefore: fipcd /sgt/bilko
- Always rename files (ie RNTO/RNFR combinations) using the 'ftpaftfile'
keyword and NOT 'ftpafter' because by the time 'ipftp' gets to the 'ftpafter'
statements it is not at all bothered whether any files were sent or not!
For ordinary FTP - and FTP over SSL use :
newname:\SN.tmp
ftpaftfile:rnfr \SN.tmp
ftpaftfile:rnto \SN.gogogo
For SFTP (over SSH) you can (only) use the builtin 'REN (oldname) (newname)'
use-ssh:yes
newname:RP\$z.tmp
ftpaftfile:REN RP\$z.tmp \SN.pdf
- Directory lists of the remote folder
ftpbefore: cd /pub/secret/\EQ
ftpbefore: fipdir
ftpbefore: fiplongdir /fip/web/pages/junk/SHOWME.TXT
ftpbefore: fipstddir /fip/web/pages/junk/STDDIR.TXT
'fipdir' gives a short and 'fiplongdir' a long list in whatever format the
remote system gives !
There can optionally be a filename after the 'fipdir' to create/replace a file.
Also 'fipbefore:fipstddir will do the same but in the system-independent format
as used by MLSD. Note this will only work on servers that can process the
enhanced MLSD command !
-- GET - There is a peculiar quirk of some FTPs - often on Linux or Win2k -
that will display a complete tree of all files and sub-folders under a given
folders. 'ipftp' assumes you do NOT want to drill down this list and, by
default, will only pick up files from the top folder specified. If you do need
to drill down, use parameter 'walk-remote-folder-tree:'.
USE WITH CARE - you could cwd/cd to root on such a box and start pulling over
EVERY FILE ON THE DISK !!!
-- For Mac resource forks, 4 parameters specify :
- where a template is to use as the basis
- what software is being used - nfshare, ethershare, ushare
- the file type
- the path for the res fork from the data file
eg
resfork-path: .rsrc/
resfork-type:ethershare
resfork-filetype:JPEG
resfork-creator:FIPO
resfork-template:rsrc_template
The 'resfork-type' can be 'ethershare', 'ushare' or 'nfsshare'.
Note that, except for resfork-type, none of these parameters are converted so
case is important.
The FileType and Creator are NORMALLY 4 uppercase letters/numbers and default
to TEXT and FIPO respectively.
-- Proxy
There are several types of Proxy servers
- http
- ftp
- socks4 and socks5
You will need to check with the network team what type it is.
For Socks 4/5 - use these parameters to control
use-socks:4/5 yes/no (yes is same as 5)
socks-host: (hostname of the socks proxy) no default
socks-port: (port number of the socks proxy) default: 1080
socks-user: (user name for the socks proxy) no default
if nothing specified, assumed that there is none
socks-pwd: (password for the socks proxy) no default
It can also be forced to ALWAYS use socks4 or socks5 with the -4 or -5 input
switches
To use ipftp thru a proxy server, there is an 'ftpproxy:' parameter to add the
extra information.
This uses the same syntax as the 'ftpbefore' etc above.
The flow is that you connect and logon to the local proxy server first..
.. then put something in to tell it where the remote FTP server is.
Generally there are two type of Proxies
Case 1 - where only USER and PASS are needed.
Case 2 - where ACCT is needed.
Case 1 - 'ftpproxy' is used to state the actual IP address or hostname of the
proper remote host is added to the USER after an '@' sign. USER and PASS then
give the logon and password for the remote.
- 'remhost', 'logon' and 'password' should be the proxy server
hostname/logon/password
- 'ftpproxy' holds any logon and password of the remote
eg ftpproxy:USER abcde@195.13.83.99
ftpproxy:PASS eyeAteFigs
Case 2 - the word 'proxy_server_user' is added as an extra word on the logon
- 'remhost' is the proxy server
- 'logon' is the REMOTE logon '@ REMOTE host
- 'password' is the REMOTE password
- 'ftpproxy' uses ACCT to hold the password of the PROXY server
eg
remhost:proxy_host
logon:remote_logon@remote_host proxy_server_user
password:remote_password
ftpproxy:ACCT proxy_password
; use log to check what is REALLY happening !!
logfile:/fip/log/ftp/FIPTEST.\$D
and the log file gives
ftp> open proxy_server
Connected to proxy_server.
220 Blue Coat FTP Service
Name (proxy_server:fip): remote_ftp_server_user@remote_ftp_server
proxy_server_user
331 Enter password.
Password:(remote_ftp_server_password)
332 Enter proxy password.
Account:(proxy_server_password)
230 User xxxx logged in. Access restrictions apply.
Remote system type is UNIX.
Using binary mode to transfer files.
... etc
-- SCP ---------------------------------------------
Note that SCP is pretty lightweight - so while the transmission may be quicker,
the lack of error checking (at the remote end) the lack of a final ack to say
the file has been received is really an issue !
from Version 18j91 on FOR SEND ONLY
; logon password and host as normal
remhost:10.3.3.32
logon:fip
; flag we need ssh
use-ssh:
; onefile only
onefile:
; getting the folder right
; AQ is in the FipHdr of the incoming file
; BE CAREFUL - make sure you get the right folder that you have permissions for
!!
combie:QQ AQ,tmp
; -t for push/put/send
; -p for preserve times
ssh-scp-name:"scp -vpt /\QQ"
; check where your sshpass is !! and add password
ssh-program:/fip/3rdparty/sshpass -pXXX /usr/bin/ssh
; possibly play with these settings
;;ssh-options:-oForwardAgent=no -oPermitLocalCommand=no
-oClearAllForwardings=yes
possible changes to these
;;ssh-scp-response-wait:30
; permissions of files on the remote server - be very careful; can be FipSeq;
make octal !
;;ssh-scp-permissions:0644
ssh-source-logon: logon name if not the one ipftp is started by
the logon which ipftp is running under MUST have setguid permissions to be
able to assume the logon specified
-- Secure FTP --------------------------------------
Confusingly there are two - completely different - 'Secure' FTPs plus a defunct
company called SecoueFTP etc etc
1. a more secure version of ordinary FTP which uses SSL/TLS in the same way
http and httpS work for secure web sites.
2. a file copy layer which sits ontop of SSH - which has nothing at all to do
with ordinary FTP
The extra confusion is that SSH uses SSL - so dont mix the two up !
How do you know which is the one you want ?
- What is the port number on the remote server ?
port 21 - it is BOTH normal FTP and the SSL/TLS version (port 21 is the same
as normal FTP)
test with ordinary 'ftp' client
port 990 - it is ONLY SSL/TLS version
test with 'telnet' to (remhost) 990 and cut the connection once you are
satisfied it connects
port 22 - it is ONLY the sftp on top of SSH
test with 'ssh' or 'sftp'
--- 1. SSL/TLS
This is the easy one..
YOU MUST USE ipftpssl for any/all ssl/tls traffic as ipftp blocks these
commands.
Prerequsite is the SSL layer which nowadays comes as standard on most
platforms. Otherwise it can be downloaded from the installation kit or the
website of the OS - Sun for Solaris, RedHat, Suse etc
http://www.openssl.org
For *nix, if you have gcc installed, it is usually easier (!) to compile from
the latest sources at http://www.openssl.org/source/
For Win2k, there are precompiled versions at
http://www.openssl.org/related/binaries.html
which points at
http://www.slproweb.com/products/Win32OpenSSL.html
Pick the 32bit Light version unless you are running very very high volume
stuff.
NOTE you usually have to add the 'MicroSoft Visual C++ 2015 Redistributables's
to run OpenSSL
(x86/32bit please - vcredist_x86.exe - NOT the 64 version)
To get ipftp to use SSL/TLS, all you need to do is add the keyword
use-tls:(param)
where param is
no - normal, standard FTP on (normally) port 21 for the control (ie DO NOT
use SSL/TLS)
yes or explicit - connect (normally) on port 21 in clear then use SSL/TLS for
USER, PASS and data
auth - connect (normally) on port 21 in clear, use SSL/TLS for USER, PASS
then return to clear
(but use SSL/TLS for all data) This is normally the only version which
works if using a Proxy server or a really nasty Firewall
implicit - connect (normally) on port 990: use SSL/TLS for all control and
data
If passwords and/or certicates are required, use the ssl-cert etc to add
--- 2.a SSH/SFTP
The Fip SFTP is layered over OpenSSH (www.openssh.org) in a similar way to
'sftp'. So before using 'ipftp' in secure mode, openssh will need to be
installed by your or your system manager.
OpenSSH is available either as an installed option on most current Linux/Unix
builds or prebuilt binaries can be downloaded (for Solaris,
www.sunfreeware.com, Linux redhat etc, AIX www.bull.de).
For Win2k - watch this space - the implementation is not tested or tuned yet.
Once OpenSSH is installed, use the 'sftp' program to check it works with the
remote site.
Then to use secure ftp in 'ipftp', add the parameter
use-ssh:yes
Note that the commands for 'ftpbefore'. 'ftpbeffile' etc are restricted to
those available in SSH/sftp and NOT the full FTP set.
There is also an extra parameter for buffering file sends
sftp-window: (number of buffers)
The number is from 0 to 64 and defaults to 2.
This allows for (number) more packets to be sent before they are each
acknowledged - which can greatly increase the speed of transmission as a whole
on slower or long distance connections
SFTP comes in a number of versions. Fip only allows the 2 significant ones
- 3 all commands that Fip needs
- 6 same as 3 but with more options (which are usually ignored) and better
folder listings
The initial connection is normally negociated between the 2 programs
But sometimes it is easier to force it - normally to v3
sftp-version:3
---- 2.b SSH/Sending to a new system using sshpass
On *nix, most modern systems use 'sshpass' (it can be downloaded from
sourceforge.net) - beware it will not work on old versions of some OS as it
needs a certain minimum version of OpenSSL. This layers the sshpass in the
parameter file eg :
sshpass-program:/usr/local/bin/sshpass
The passwd is added from the 'passwd' parameter.
See the doc on sshpass for which parameter to use - and change the paths for
both sshpass and ssh to what they are on your system.
sshpass-program:/fip/z/gnu/sshpass-1.04/sshpass
- fills in password etc from parameter file
- then runs the ssh in ssh-program (or default)
or add the remote password after the '-p' :
ssh-program:/fip/z/gnu/sshpass-1.04/sshpass -p 'xxx' /usr/bin/ssh
- runs the whole thing
++ NOTE YOU MUST connect MANUALLY beforehand and accept any prompts such as:
The authenticity of host ‘some_hostname (192.168…..)’ can’t be
established
Are you sure you want to continue connecting (yes/no)?
++ IF YOU do NOT do this, ipftp will hang on the unanswered question as
nothing will reply 'yes' ++
---- 2.c SSH/Sending on/to a Remote server in the cloud which is a clone of
another
Sometimes the remote hostname may have multiple ip addresses - run a 'dig' to
check :
dig ftp.remotehost.com
.. and check the ANSWER section, eg :
;; ANSWER SECTION:
ftp.remotehost.com. 10 IN A 209.73.214.14
ftp.remotehost.com. 10 IN A 209.73.214.15
The problem will be that in the 'known_hosts' file, ftp.remotehost.com can only
be a single entry. So ipftp may/will halt when the 2nd address kicks in.
You will get unpleasant key mismatch errors such as :
WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
To work around this, add these parameters to ssh-options :
-oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no
eg
ssh-options:-oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no
-oForwardX11=no -oForwardAgent=no -oProtocol=2
---- 2.d SSH/Sending to a new system using shared keys
If sshpass does NOT exist (and cannot be loaded for some reason) to make
'ipftp' work using SSH, the remote site needs to let you in without prompting
for a password.
This entails generating a pub key on one system and installing it on the other
- and then generation a public key on the other and copying it to the first.
You should read (and digest) the man pages for the ssh-keygen to work out what
data needs to be added to which files.
For any reasonable recent Linux distro, the commands are :
1. generate the keys - public and private
ssh-keygen -t ed25519
just hit return to all prompts unless your site requires a passphrase
if this errors (esp if it says that 'ed25519' does not exist) try
ssh-keygen -t rsa -b 4096
just hit return to all prompts unless your site requires a passphrase
2. copy it over to the other system(s)
ssh-copy-id fip@fgbserver9010
this logs on to the remote
If you get an error msg :
.ssh/authorized_keys: NO such file
logon to the remote and
mkdir -p ~/.ssh
touch ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 644 ~/.ssh/authorized_keys
then retry ssh-copy-id
3. connect manually to check it works !
BEFORE running ipftp, 'ssh' must be able to connect and logon WITHOUT a logon.
So if it does not, go back to the doc for you system as it may differ slightly
to that described above.
If you are unsure whether the ssh is working correctly, ssh debugging can be
turned on by adding '-vvv' to the command line.
ssh -vvv pif@remotesys
Similarly ipftp can be put in a similar mode using the 'ssh-debug:yes'
parameter and 'ipftp' and 'ssh' both spit out reams of meaningful information.
If it does not work (!), it could be that 'ssh' is in a different folder ipftp
looks in /usr/bin first and then /usr/local/bin - and if not found gives up
with an error. Use 'ssh-program:/your/path/tossh/ssh' to state where it really
is.
If you need to use a non-standard ssh program or set of options, the defaults
can be changed. The defaults are below - note the use of single quotes around
each option.
ssh-program:/usr/bin/ssh
ssh-options:-q -oForwardX11=no -oForwardAgent=no -oProtocol=2
Plus ...
if the number of connection-retries is NOT 5 -oConnectionAttempts=(number)
if the remote port is NOT 22 -oPort=(number)
if the timeout is NOT 120 seconds -oConnectTimeout=(number)
if you are connection to newr systems, try -oBatchMode=yes
For proxy connection, try '-oProxyCommand=nc -v -x(proxy host):(proxy port)
%h %p'
(Note the single quotes around the WHOLE parameter - not just the value -
where there are embedded spaces)
(pre 18j59 these options defaulted to :
ssh-options:-q '-oForwardX11 no' '-oForwardAgent no' '-oProtocol 2'
If you are having problems with the SSH link - bad dns, flaky network etc - you
can specify a log filename for just the SSH messages.
THIS FILE CAN GET VERY large - so pls remember to turn off (and also use FipSeq
to make day/hour files)
This file is in addtion to the normal fip 'logfile'
ssh-debug:yes
ssh-debug-logfile:/fip/log/ftp/SSH_NAME_\$e\$y\%i\$d.log
There is also a ssh-debug-loghead: for adding more information at the start of
exach transmission.
--- Running parallel streams and an error stream
Use ipwheel, ippost, iprobin or just the USERS file to route data into any of
the 4 (or so) main ftp queues, and use -W, -Y and -O to handle errors.
ftp1 ipftp -i 2ftp1 -O ftp_error -W
ftp2 ipftp -i 2ftp2 -O ftp_error -W
ftp3 ipftp -i 2ftp3 -O ftp_error -W
ftp4 ipftp -i 2ftp4 -O ftp_error -W
ftperr ipftp -i ftp_error -Y
--- Troubleshooting a problematic connection
To find where the problem lies .. our checklist is
- Try it manually first to check it is ok - INCLUDING sending a file to make
sure permissions are ok
- Always try it manually from the Fip server to make sure the Firewall is ok
too.
- Use the log file parameter and see what it brings up.
- check with both active and passive and check timings - which works better.
- Check both servers are running FULL-duplex
- if it still failing test at the TCP level with snoop/tcpdump/wireshark (which
can lead to total obfuscation as you get too much info)
The log file does give you everything sent and received on the Control Port
plus the first chunk of data on the Data Port
- so you can tell :
- if the FipHdr has been stripped off by looking at the first part of the
file sent part.
or - if the ftpbefore has worked - as the CD or whatever will be noted with
the reply (positive or neg)
The FULL-duplex one is quite a tricky one as most network programs work of in
half-duplex, including the FTP control port. The symptom is that small files
are FTPed with no problem but larger ones (generally over 200K) always bomb
out.
With most servers being auto-sensing, all it takes is a network glitch for them
to drop down to a slower speed or half-duplex - and they dont always readjust
upwards when it has cleared. (On a sparc, /sbin/mii-tool is a useful thing to
hack with)
For the Active/Passive thing, if a firewall is eating the data connection in in
one direction, it is quite hard to diagnose - except for any 425 error messages
(which are always generated on the side which is starting the data connection,
and so may not be ipftp).
So swopping between passive and active is definitely one thing to try.
The difficulty is when FTPing thru both your firewall and the remote client's
to get to the remote server.
One of the Firewalls MUST be setup to allow either the High port connections or
traffic on a specified fixed-data-port.
Note - active means the remote server will start the connection to the client
(ipftp in this case) for any data (LS or FILE)
- passive means ipftp will start the connect to the remote server for any data
(LS or FILE)
---- Sending all files in a folder .. and stopping
Use -A for where you want all the files from a folder to be sent - and then
ipftp stops/exits.
The wrinkle is - what happens if the connection fails; we want ipftp to keep
pounding away until all the files have gone.
It gets used for things like sending all 900 files to Amazon for a Kindle
edition for example - where the chances of a network blip in the early hours of
the morning are quite high, and we need ipftp to reesablish the connection and
continue.
Try it with no offline queue : (where QD is a date like QD:20130126)
ipftp -A -i /fip/data/ebooks/\QD_kindle -h'WD:\QD' -Z -z goa_kindle -y 10 -Y
------------------------------------------------------------------------------
Version control
;19a36 26jul18 re-added socks proxy ;2 bytesTX for SFTP/SCP was zero ;3
savtimeout and drain-thsTotData added
;4 save sshOptions ;5 better failure tracking for sftp
;6-8 nat-pasv-address - woops LINUX is normally i86 so needs reversing
;9-12 12apr19 redid ssh with sending fiphdr Plus added ssl-display
;13-16 15jul19 allow logon for ssh (careful) and -? for prompt between files
and -u owner for rc.fip
;18 18feb20 bug in multiple files
;19 24apr20 new version of fipssf ;20-21 ssl-session ;22 WINNT ;23 10nov20
buglette SSH + ftpbefore:cd /xxx ;24 cosmetic
;25 21jul21 added use-socks and made all socks* fipseq ;26 loop ssf_init
;27-28 9mar22 bugette in get*-except
;29 28feb23 added public-address-from-aws-meta
;30-31 7mar23 added max-get-sessions
;32a added timeout to thru the proxy
;33a 25may23 added verbose-logging
;34-35 16oct23 better NLST handling and POLLIT and RETRtimeout for connections
which will not drop (-V = disp and -U = passive)
;36 8jul24 redid send-multiples
;18j97 29dec09 ;1-3 added TLS/SSL and fipstddir ;4 bugette for balance and
programname ;5 \%q is time taken
;6 08mar10 added tracker-script ;7-8 added ConnectionAttempts an
ConnectTimeout to ssh/sftp
;10-11 16apr10 only read pram file once for GETS
;12-17 7jun10 added error message in drain to describe what ! (15 added
linger-on-close) ; 16 better wilds with SSH ;17 TLS'C'
;18 25jul10 bugette with -o not adding /fip/spool
;19 10aug10 added rename-prefix and rename-newname for SFTP
;20 29oct10 bugette with false error msg for gets
;21-23 10nov10 GET-skips fudge for W2k FTPD bug ;22 \%c bytes RCV or TX
;23 added -A onePass and doneque is now FipSeq and fixed GENERIC_GET bugette
;24 added send-external-fiphdr:yes/no
;25-31 redid keepConnectionOpen so it did NOT logon each time /
do_connect-OpenData
;32-33 6dec11 woops remport NOT working for ssh and added remote-folder-exists
;34 12*425 is an error ;35 sshpass now works
;36 added list-folder-only plus added an error if Unknown keyword in parameter
file
;37 15feb12 added max-data-port and min-data-port (min is same as
start-data-port)
;38 20feb12 woops TLS/SSL only worked 'onefile:'
;39 \$c and CTRL bugette
;40 6jul12 added hidden 'CD ~' if 2 sequential files have different Parameter
files but the same host/logon
;41 4sep12 added max-single-fiphdr-size
;42 21sep12 added fiphdr ZO original filename
;43-45 10oct12 added timeout alarm for sshpass plus added sshSftpVersionNumber
;46-47 17jan13 added send-multiples ;48 added getIsHash for Renames ; 49
cleanup ; 50 outque is now parseable
;51 added get-copy-file ; 52-53 tuned
fipdelete/fipduplicate/cd/fipallowfile/fipblockfile for spaces in filename
;54 5dec13 fixed max-files for GETS too ;55 added file_trace ;56-57 minor
logfile additions
;58 8aug14 passive-force-address added
;59-60 5sep14 added ssh-debug-logfile plus sshpass quotes to " not ' ** needs
fipssl 01d
;61 3oct14 passive-connections:(FipSeq)
;62 11dec14 added local-filter:yes ;63 CWD~ should be CDUP ;64 added
hash-in-fiphdr ! ;65-66 log msgs and newE2 for queue in logging
;67 24sep15 default linger 3 -> 20 ;68 back to CWD ~ not CDUP
;69-73 better check for IPaddress in place of hostname ; 74 added speedy for
GET files
;75-77 added fiphdrMD5 chksum on GET files
;78-81 20jun16 added check-compare-file timestamp
;82 5jan17 added ssh-window ;83 added merge-fiphdr-on-getfiles
;85-86 18apr17 bugette in LARGE skipdetails
;87-88 5jul17 ssf buglette (in fipssf not ipftp) and recode added
;89-91 3nov17 added SCP send
;92 1feb18 use-ssh, use-tls, ssh-debug, ssh-window, tls-auth now parseable
;93 6feb18 onefile, max-files, append, uniquename, inc-seqno, max-seqno,
min-seqno, ignore-timeouts, throttle-speed, max-attempts now parseable
;94 9feb18 bugette - sshSftpVersionNumber getting reset EVERY time we read the
param file - should be 'if no connection'
;95 15feb18 added ssl-verify an ssl-ciphers
;96 19mar18 added nat-pasv-address
;018i35 12dec06 added skip-purge-after
;b 18dec06 PASV and 425
;c 09feb07 ssh wait cleanup
;d-g 23mar07 added -x default max files and added fipallowfile
;h 20sep07 small internal chg to link with StingRay FTP Server - no PORT/PASV
on logon
;i3 24oct07 added 'fipcd' and fipdelay can now have a noOfSecs after
;i4 1feb08 redunBalanced
;i5-10 19feb08 added send-external-file-name ;i6 max skip files upped
1000->20000
;10-11 27may08 bug in fipdelay (from i3) and some remote servers do NOT allow
skipdetails
;12-17 5jun08 added rename-prefix, rename-newname and BUG in PASV ; 17 for
log-level:
;18 -justskip added
;19-21 added log-skip-details:(and stackoverflow doc)
;22-25 note_balance_action
;26-27 added -H for cluster host name and allow ZZ to be used in
get-extra-fiphdr
;28-35 14sep09 check if proxy for failure and added minimum-poll-interval ;
30 HE+HR in GET FipHdr
;017z 24jul05 made H1-3, DR, DG, DT available to script, added -k and -y and
script for GET
;b-d 12aug05 cleaned up all those extra PORT/PASV on skipped files
;e 15sep05 keep-connection-open added for sending
;f 14oct05 STOU/uniquename need tuning
;g 21oct05 Reworked ZeroLen GET files and ignore - now we do NOT add to skip
notr zap/move etc
;h 23nov05 -W and -O mean that the offline queu is that specified and '2ftpX'
is not appended
;j-k 15dec05 added shadow skip file - possibly on another drive. and added -K
for
;l-o 09jan06 added add-md5-signature and cleanup of getfiles
;p 23jan06 added fixed-data-port
;q-r 23feb06 added put-empty-files (allow format: same as formatname)
;s-t 28jul06 added remote_trace style timings to send file done msg as H4
;u 10aug06 wait3 64 bit and added -X and timing-stats
;v-w 07sep06 added priority-sends plus bugette with fip-onerror
plus for GET, must have a dest:, so added a default.
;x-y 04oct06 new DestRedun
;z 06dec06 added start-data-port
;016z 10aug04 add skips to old too
;b 24sep04 speedy
;c-h 27oct04 added ssf-secure shell stuff
(d,h 19nov04 hash-in-filename added)
;i-j 31jan05 better Win2K Gets
;k-m 07feb05 'dests' can be FipSeq and added outque:
;n-p 23apr05 remote-done-que is now FipSeq
;r-v 13may05 redid old skip list for WINNT and added -B for skip-balance-group
;w-x 15jun05 added -Y there is no offline queue
;y-z 30jun05 only go PASV if no Data port is open
;015z 06jun02 better control over filename.....
;a/b 18jun02 WINNT get was broken
plus AS400 gives a funny file ended message.
;c/d 28jul02 added ignore-all-preparation
;e 12sep02 added creating ResForks on the remote system
;f 15sep02 bugette for GET with many entries - may miss the last ones
;g/h 10oct02 bugette with zero length files plus added
check-primary-server-for-getfiles: and balance-skip-files
;i 16oct02 to strip the './' at the begining of WarFTP files.
;j 23jan03 added inc/min/max Seqno
;k 30jun03 bugette - skip file not zapped when nothing there.
;l-m 12sep03 redid timeout on connect
;n 30sep03 added logon to checking host
;o 06oct03 quiet on DataPorts.....
;p 12dec03 added -h for extra-fiphdr on GETS
;q 06jan04 default formatname for Gets
;r 19feb04 small mod - comments only at start of line
;s 26feb04 better skiplist
;t-w 25mar04 added FTP_LIST_FILE: or -F and fipdir
;y 29jun04 added -O offline queue
;z 02jul04 bugette in PASV/passive sends
;014z 08sep00 added minimum-log
;a 29nov00 improved logeachmsg
;b 20dec00 bugette for the case of maxattempts exceeded and no errque
;c-d 10jan01 bugette in fiphdr:
;e-i 30jan01 cleanups
;j 27mar01 better exit codes for single-shot and errors
;k 10jun01 added FTP_EXTERNAL_FILE: and FTP_ZAP_EXTERNAL:
;l-m 12jul01 do NOT normally walk the remote folders for getfiles.
;n-o 25jul01 added no-fiphdr-on-getfiles
;p 09aug01 PASV/passive-connection tuned. and -V added
;r-u 17aug01 added 'except's
;v 01oct01 fipdelete mods for 'fipon-error'
;w 16jan02 added loglasterrfile/log-last-error-file
;x-y 14mar02 added remote-trace and zap DO on attempts from filename
;z 30apr02 added script plus FTP_FILE/BINARY_BEFORE/AFTER
;013b 15aug00 added skip-file plus possible buggette in GetFiles and lousy
networks
;a 19aug00 bugette in get-extra-fiphdr
;b 05sep00 added log of skip files
;012e 03apr00 bugette in LOG file - could get confused with many files in
offline all withh different log files.
;a 26apr00 chkmsg seqno not right
;b 01may00 NT getfiles was buggy
added fipon-error/on-error and maxattempts and slowdown
;c 17may00 added maxfiles:
;d 21jun00 better handling of the Done Queue
;e 09aug00 added 'zapresforks'
;011g 30nov99 added ftpproxy
;a 06dec99 mod to drain_it - occasionally missing a result code.
;b-c 10dec99 v occasionally, may miss change of seqno
;d-g 14jan00 allow space in NT filenames - redo seqno
(copyright) 2024 and previous years FingerPost Ltd.