ipftp This program loops around a queue and sends or gets or lists files to/from a Remote Server via FTP. As FTP does hang on (regular) occasion, no response for a timeout period will abort that file or transmission and the program attempt to restart where appropiate. By default, the program can be spooled against a queue or kicked off for a single file or queue from a script using the '-1' input switch, which we call 'single-shot'. For single shot, the files are NOT deleted unless the keyword 'del-files' is specified. Files are only deleted if sent correctly or, for getting files, if a reasonable response was received like 'File Not Found'. The FipHdr of each file to be sent is checked for the 'DF' field which, as per normal for Fippies, is the name of a parameter file in tables/ftp. This parameter file, tables/ftp/FORMAT by default, is used to describe the transmission. The Remote Host Name can be a Fip Pseudo-Host (ie with an entry in 'tables/sys/DEST_REDUN' ). In this case, the primary host is checked for being down and, if so, data sent to the secondary. If that too is down, files are not sent. File for remote systems that are offline or un-pingable are send to 'spool/ftp-offline' which will be rescanned on a regular basis. When such a system recovers, the program will automatically send the collected data files form 'ftp_offline'. Files for retrieving can be left on the remote server untouched, deleted or moved to a specified Done folder. There is a Poll input switch which will just look once during the period for any files and get them accordingly. A skip file may be created and maintained automatically for GET files which we do NOT need to re-get. This is used where a 'get-and-zap' or 'get-and'move' cannot be used. ipftp can also be used to merely list the files on the remote server using the -D input switch or the 'list-folder-only:' parameter. Syntax for the parameter file are : ; comment remhost: Hostname on remote system This overrides the '-s' Input switch. defaults to that specified in the '-s' Input switch This can be in FipSeq or just fixed text. remport: Port on remote system This overrides the '-p' Input switch. defaults to that specified in the '-p' Input switch or failing that, defaults to 21 This can be in FipSeq or just fixed text. logon: Logon for remote system default: anonymous This can be in FipSeq or just fixed text. password: Password for remote system default: none This can be in FipSeq or just fixed text. nopassword: The remote system does not need a password To send a password of nothing (which is different to NO pwd), specify the keyword with no parameter : eg password: Normally a remote system wants a password even if it is blank. remhost,remport,logon,password,nopassword may be repeated another 9 times for roundrobin which is enabled by the -R input switch. remhost:mainsys.bigdom.com logon:main password:passo remhost1:backup1.bigdom.com logon1:back password1:passo remhost2:backup1.bigdom.com logon2:backo password2:passo ftpbefore: FTP command to execute after logon default: none ftpbeffile: FTP command to execute before each file. default: none ftpaftfile: FTP command to execute after each file. default: none ftpafter: FTP command to execute before closing up default: none Run some raw Ftp commands.... There can be several commands for each section, each on a separate line. Note that the actual commands are NOT those from the user interface, but the raw ones as used by the FTP protocol. For weirdos you might want to check that the remote system will actually allow you to do these ! Valid commands include : dele file delete ** See notes below cwd change directory mkd make queue rmd remove queue pwd get current queue list long queue list - like an 'ls -l' or 'dir' nlst short queue list - like an 'ls' or 'dir/w' nlst -lrt : long list, sorted by time. For lists, always 'cwd' into the queue beforehand. type 'a' for ascii or 'i' for image or binary rnfr rename file : these two work in tandem .. rnto rnfr=old name, rnto=new name. eg: ; Do a MKDIR (who cares if it fails) and a Chg Dir before anything ftpbefore:mkd /pub/pporange ftpbefore:cwd /pubmp/pporange ; before each file, make sure we delete an old version before ftpbeffile:dele \SN ; at the end do a List to see what is really there. ; the list will be in the log file ftpafter:nlst -lt logfile:/fip/log/ftp/FTPCLIENTS.\$d-\$m-\$e\$y Note also there are some Fip-enhanced keywords : fiptouch fipdelay fipblockfile fipallowfile fipdelete fipduplicate fipon-error fipdir fiplongdir fipcd see below for explanations on what these do ! ftpproxy: FTP command to execute after logon default: none to logon to yet-another FTP server. Please see the notes below ignore-all-preparation: ignore ALL the before/after/beffile/aftfiles and just send that RAW file out. Use this for big binary files that need NO alterations or additions. before: FipSeq string added before data of each file. default: none after: FipSeq string added after data of each file. default: none beffile: FipSeq file added before data of each file. default: none Beffile is added after any 'before' string and before normal data. aftfile: FipSeq file added after data of each file. default: none Aftfile is added after normal data and before any 'after' string binary-beffile: binary file added before data of each file. def: none Beffile is added after any 'before' string and before normal data. binary-aftfile: binary file added after data of each file. def: none Aftfile is added after normal data and before any 'after' string first: Name of optional file in tables/ftp that is sent on startup and on each scan of the queue when a new file has been found. The text of the file is in FipSeq. default:none last: Name of optional file in tables/ftp that is sent when all the files in the queue have been sent and before we rescan the queue. The text of the file is in FipSeq. default:none check-compare-file: For Grabbing, Compare the contents of this file BEFORE checking for new/changes default: none Use this for timestamp file which is changed on the remote when there are new or corrected data files available. hostname: The hostname for the system 'ipftp' is running. This defaults to the hostname the system was booted with. However for systems with multiple ethernet addresses, this is used to tell the remote system the IP address to use. (Note for versions 06c+, this is no longer required). del-files: Delete the files once sent. This is for -1 single file only. default: for -1 send one file, do NOT delete input file default: for spooled folder, delete ALL input files once sent correctly. timeout: Timeout period when sending for no response default:120 secs The default timeout can be modified by the -T input switch. If you are sending to a host which is only across the computer room, you should take this timeout down to 10 or 15 secs. connection-retries: No of retries if the first connection fails. The default is 5. Which means a remote system which has disappeared can hold up the next traffic by up to (5) * (120 secs timeout) = 10 mins !. passive-connection: (yes/no) This causes the remote server to enter PASV mode (if it can). The default is NO for 'active' if the command is NOT specified The default is YES for 'passive' if the command is specified with no options. From version 18j61, NO or YES can be in a FipHdr field in the file being sent in the FipHdr : PC:yes eg passive-connection:\PC passive-force-address:yes/no default: yes Force the IP address for a Passive Data connection to be the same as the Control Sometimes the remote server has a funny NAT address - use this to 'clean' it up linger-on-close: Timeout in secs for the data to be sent. For slow connections, increase this, default: 20 (secs) to allow up to 20 secs max before cutting the call outque: For get files only, this is the output folder for incoming files This overrides the '-o' input switch and the default is 'spool/2go'. doneque: Queue to move original files once sent. default: none balance-done-queue: (Balance group) Balance the done queue items errorque: Queue to move original files if NOT sent. default: none This is used ONLY where systems do NOT have a valid IP address because there is no entry in the host file, or it has been missplet or, more likely, DNS has flipped. If the remote host is out there but the Logon is refused or, once logged on, we were unable to write the file, the files are stuffed in 'spool/ftp_offline' or if the input queue is NOT the default, spool/ftp_(inputque)_offline onefile: Send one file at a time to this host. After each file, 'ipftp' disconnects, waits and then reconnects before sending the next. default: send all files in the queue. maxfiles: Send (or Get, from version 18j54) this number of files at a time to/from this host. After so many files, 'ipftp' disconnects, waits and then reconnects before sending/getting the next bunch. Use this to 'throttle' a connection where too many files (in or out) might cause a system overload. default: send/get all files logfile: Log file name default: none This should be the full path and filename of the the log file. The contents of the Log file are preserved and appended to. Use this to debug new feeds. You must make sure the queue exists for the log file as 'ipftp' will not create it. logfile-dest: Destination for the log file default: none This is the Fip DU destination for a log of the transmission. Use this to debug new feeds. Only specify EITHER 'logfile' or 'logfile-dest' (or neither) but not both. minimum-log: Only log starts. stops and errors in the 'logfile' Normally the whole session is logged. dest: Fip Destination for any Directory Listing or default: none Retreive files. This should be a valid DU in the tables/sys/USERS file. send-no-data: Do NOT send any data from this file default: send data Use this option to send just a headline with a before or beffile or to place a marker in a directory in the remote host. fiphdr: Send Fip Hdr if part of the file. default: send data only Normally old the data part is sent and the FipHdr stripped off. hash-in-fiphdr: (FipSeq chr) default:none A hash/pound (#) in a FipHdr field is mapped to this chr Normally hashes are end-of-field in a FipHdr. So it needs to be mapped to something else and we changed it here on outbound. newname: Newname for the file(s) in FipSeq default: as original newname:\SN.\$Z The default is the same as 'newname:\SN'. Note that the 'first' and 'last' files are NOT renamed. uniquename: Force the remote server to store the file default: no with a new name if it is a duplicate. Some systems allow you to give a filename, others do not. append: Force the remote server to either append to default: no an existing file or create a new the file Some systems allow you to give a filename, others do not. This is not available on all remote systems ! forcename: force the filename on the remote system to be upper or lowercase forcename:upper default: no change formatname: Make sure the format of the filename on the remote system is correct for that system options are raw, unix, mac, nt, alphanumeric,pc default: unix on *nix and nt on Win2k YOU WILL NEARLY ALWAYS want to format the filename as different systems only allow a subset of characters. Note that Win2K boxes do not like trailing dots so 'formatname:nt' also strips them. To preserve the filename - such as if you are using ZO - use formatname:raw for no changes at all. sizename: Maximum size of the filename on the remote system eg sizename:32 default: no limit Note that normally the max for Windows/NTFS is 127, Unix is 255 and Mac is 31 The maximum for sizename is therefore 255. send-external-file-name: Name on the remote system for any external files Use this parameter to send the external file separately to this name default - if there is an external file it is sent in the one file. send-external-fiphdr:yes/no if there is an external file containing the data (pointed to in the FTP_EXTERNAL_FILE: fiphdr field), it may (or may not) have a FipHdr. Use this parameter to NO=strip it (or YES=leave it on) when sending. default is yes to send the fiphdr on the external file rename-prefix: (FipSeq) prefix for files that will be moved in get-and-move: If the remote server is a Windows server, permissions may prohibit a straight move of duplicate filenames. Use this to make the filename unique. It is a Prefix as often, the extension needs to be kept! Eg rename-prefix:\$e\$y\$i\$d\$h\$n\$b_ If the original file is POPE.JPG, this will prefix the date and time to the moved file. rename-newname: (FipSeq) newname for files that will be moved in get-and-move: If the remote server is a Windows server, permissions may prohibit a straight move of duplicate filenames. Use this to make the filename unique. Remember FipHdr E1 will have the filename Eg rename-newname:\E1.\$z send-empty-files:put/send send-empty-files:put/send or put-empty-files:put/send put-empty-files:ignore For SEND, if the file is zero length or empty this parameter will allow you to : put or send - send the file and process as normal ignore - skip to next file default is send -- Listing Folders instead of sending ..... (note the default is to SEND) list-folder-only: Only one parameter file is used - specify with the '-z' input switch. Also specify '-G 0' for a one-off grab or '-G 99' for a grab every 99 secs This MUST have at least one of the following in the parameter file: ftpbefore:fipdir (for short listing of just the filename) or ftpbefore:fiplongdir (for a long list in the remote servers own style) or ftpbefore:fipstddir (for a long list in the generic style - using MLSD) -- Getting files instead of sending ...... (note the default is to SEND) getallfiles: Get ALL files from the remote server. getallfiles:199* Get ALL files starting '199' from the remote. getfile: Get a file with this name from the remote. get-and-move: get-and-moveall: get-and-zap: get-and-zapall: getallfiles-except:(mask) get-and-moveall-except:(mask) get-and-zapall-except:(mask) In this case no data is sent, as only the FipHdr is used to find the filename. There can be several lines of 'getfile'. Watch out for case sensitive remote systems !! No wild cards are allowed for single file gets but you can use FipSeq : eg ; get a file from the remote system with the same name .. ; .. as the input file on the local system. getfile:\SN ; always get the readme getfile:README ; get todays file which has a 8 digit day extension style:QD $D,%.08d getfile:News.\QD You should also define the Fip destination DU using 'dest:', if not all the files all files will be sent to a DU of 'woops'. Use the 'except' versions where the mask is used to IGNORE files. eg get-and-zapall-except:.pl Get all the files EXCEPT those with '.pl' in the filename NOTE the except string is just a simple compare and CANNOT include wildstrings/chrs This is slightly inconsistent eg get-and-zapall-except:*.pl will look for a filename INCLUDING the 4 chrs '*.pl' BUT it can be FipSeq eg get-and-zapall-except:\$e\$y\$i\$d Get all files EXCEPT those with todays date in format YYYYMMDD like 20081231 Commands 'get-and-zap', 'get-and-zapall' and 'get-and-zapall-except' : get then zap each file. Commands 'get-and-move', 'get-and-moveall' and 'get-and-moveall-except' : get then move each file to a done folder on the remote box setup by the 'remote-done-folder' keyword. PLEASE see below for comments on getmove and getzap. Note that if the remote FTP server is a Win2K box, the name of the file MAY have a '/' in it (or a Unix FTP server may have a file with a '\' in the name). Normally 'ipftp' ignores such files as it assumes they are sub-folders which should not be scavenged. Use the parameter 'walk-remote-folder-tree:' to grab these - see below! get-filter-case-sensitive:no FOR SSH connections only, you can force the filter to be case INsensitive using this parametrer. default is YES it is case sensitive use-client-side-filter:yes/no or local-filter:yes/no Filter in ipftp NOT on the remote server - use this for use with slightly-odd FTP sites like smartfile.com who do NOT allow wildcard gets - try the remote manually using 'ftp', check that files exists and do a 'mget *'. If it returns '550 No files to get' when it should get something, set this flag to YES and try with ipftp. remote-done-folder: Relative path on the remote box for the data files after they have been GET'ted. eg remote-done-folder:../done remote-done-exists: Action if the file already exists in the remote-done-folder Options are 'replace' - replace existing with this one - (default) or 'add-ext' - add the new version with a date_time extention or 'ignore-new' - just delete the NEW version once the transmission has been completed successfully eg remote-done-exists:ignore-new get-extra-fiphdr: extra FipHdr to attach to incoming, GET'ted files eg get-extra-fiphdr:#ZI:#SU:REMOTE This will tell 'ipwheel' to archive the incoming data under the archive log REMOTE. no-fiphdr-on-getfiles: do NOT add a FipHdr to the GET'ted files. default is to add one with Date and Time fields etc. add-fiphdr-on-getfiles: add a FipHdr to the GET'ted files. merge-fiphdr-on-getfiles: add a FipHdr to the GET'ted files. default is to add one with Date and Time fields etc. NOTE that if there IS a FipHdr on the GET'ted files, the default is to merge it with normal ipftp ones standalone-filename: (for Get files, do NOT put a Fip-style filename but use the original filename (suitably modified for the actual system - ie strip '/' if on UNIX, ':' for Win2k) The default is the normal Fip filename/fiphdr malarky. eg standalone-filename:\SN_\$e\$y\$i\$d.fip get-copy-file: (full path name in FipSeq) default:no Make a copy of each incoming file as this folder/name eg get-copy-file:/fip/data/raw.data/\$e\$y\$i\$d_\DF/\ZZ_\$h\$n\$b_\$z this leaves a copy in raw.data/(date)_(format)/(filename)_(time)_(seqno) hash-in-filename: (FipSeq chr) default:\235 A hash/pound (#) in a GET filename is mapped to this chr Normally hashes are end-of-field in a FipHdr. So it needs to be mapped to something else in order to preserve it. get-empty-files:get get-empty-files:ignore For GET, if the file at the other end is zero length or empty this parameter will allow you to : get the file and process as normal ignore skip to next file default is ignore skip-files: Name of a file in /fip/fix/ftp for holding the names of files brought over; so that only new files are received. It is remade from the LIST on every Get ALL. Note for unix/linix systems, the name is cases-SENSitive. eg skip-files:fromFTP default: none check-skip-details: Yes/no The default is only to check if the file-to-get exists or not. But sometimes you need to track files which are updated - ie the name remains the same but the contents differ. Set this option for tracking changes too. Note it is slower and does take a bit more system resource as well as an extra skip file in /fip/fix/skip to hold the existing file times etc. default: no skip-purge-after: (hours) Number of hours to keep the skip entry default is 1. You might want to tune this : make bigger if sites add/take off old material reduce the time if the same link is used for differnet data skip-balance-group: name of a balance group (in tables/sys/BALANCE) to distribute the skip file when changed (see doc on 'ipbalan') This is oftern used where more than one system is GETTING the same remote system/folder (usually with 'check-primary-server-for-getfiles'). shadow-skip-file: (full path/filename) Name of a duplicate of the skip-file - probably on a network drive. Whichever file is newest is used. minimum-poll-interval: (secs) minumum delay between polls for gets default: 10 secs log-skip-details: Log why a file is being retrieved - new file, size change etc nat-pasv-address:(IP address) if using a proxy or firewall, which does NOT automatically NAT internal to external addresses, this is the external address of THIS host to the Extranet world. It is used in the PORT command sent to the remote server. fixed-data-port: (number) if the number is >= 20, fix the data channel port number to this number and do NOT change it. min-data-port: (number) max-data-port: (number) if the number is >= 20, the data channel port number will be in the range of min <= number <= max defaults are min-32800, max 60000 check-primary-server-for-getfiles: pseudo-host name that is specified in tables/sys/DEST_REDUN that is used whether the current host should be getting the files or not. ie in the ftp parameter file REMOTEGET is check-primary-server-for-getfiles:remotewire and in the DEST_REDUN is ; psuedohost primary secondary remotewire fip1 fip2 and in the SYSTEM file for both fip1 AND fip2 there is a line rem1 local ipftp -G 600 -Z -z REMOTEGET Then if fip1 is up, the ipftp on fip1 will always get while on fip2 it will just check/loop. if fip1 is down, the 'ipftp' on fip2 will start getting. log-each-file: (dest) or logeachfile:(dest) Send a Success/failed msg to this destination for each file. There is no default. This log file is just a FipHdr with the following extra fields : DR-File Sent OK DR:ok or DR:error DG-Will Retry later DG:retrying, DG:stopped DT-Some message text DT:No connection default: no log created. The text for the DR and DG can be in FipSeq and so can contain FipHdr and other variables. As they are FipHdr fields, please do NOT put NL, CR etc in the fields. Note that System Variable \$q holds the time taken for transmission. log-last-error-file: (dest) or loglasterrfile:(dest) Same as for 'logeachfile' but it is sent ONLY after the last failed attempt where 'maxattempts' is specified. Only log-EACH-file or log-LAST-ERROR-file can be specified - not both. log-max-size: If logging is ON and a file is bigger than this size DRgood:(text) Message for the FipHdr field DR on a successful tx default: ok DRbad: (text) Message for the FipHdr field DR on a unsuccessful tx default: error DGcont:(text) Message for the FipHdr field DG if, after an unsuccessful tz, another attempt will be made. default: retrying DGstop:(text) Message for the FipHdr field DG if no further attempts will be made as the file was sent successfully or the maximum no of attempts has been tried. default: stopped fiphdr-for-logeachfile: (FipSeq) or msgeachfile:(FipSeq) Additional information to add to the FipHdr of the 'logeachfile' or 'loglasterrfile' msg. This should be in FipHdr format and be in FipSeq. It can be used to pass FipHdr fields in the outgoing file into the log file. eg msgeachfile: DF:logdial\nSS:\SS\n default: nothing added stop-on-error: Stop if you get a '500' series error in those commands you have specified in the 'ftpbefore', 'ftpafter' etc keywords. Normally these are ignored .. .. which is what you want if you have something like : ftpbefore:dele \SN to delete a similar file beforehand and the file does not exist : 550 no such file or directory is the message received and can be ignored .. but then the same message is also given for 'cwd' that does not work ! Where it is important, use 'stop-on-error' or 'on-error'. This is overriden by the 'on-error' keyword - see below. fip-syn-log: add a Item Log line to the Fip Syndication Log file log/SYN. normally only the normal Fip log is kept up to date. newEN: A 2 letter FipHdr field which will be the filename of the output file or, for GET, the name of the file on the remote system. This defaults to EN but if you are already using EN, map it to another field. For GetFiles this can be the full pathname (especially with Lists), so use E1 (below) for the filename only. newE1: A 2 letter FipHdr field which will be the name of the file to GET This defaults to E1 but if you are already using E1, map it to another field. This field does NOT have any '/' or '\\' chrs. (for the original name on the remote server, use the fiphdr ZO and not E1) newEQ: A 2 letter FipHdr field which will be the name of the folder on the remote system for SENDing. Default is none This is usable only for the messaging back newE2: A 2 letter FipHdr field which will be the name of the input folder for SENDs for logging only. default: E2 max-single-fiphdr-size: size that a single fiphdr field can be. default is 8196-sep2012 (was 2024) the minimum is 1024 and maximum is 31000 log-max-size: If logging is ON and a file is bigger than this size then the first 64 chrs of each block are NOT stuffed in the log. This is because the log file can get massive ! default is 'log-max-size:30000' check-message: FipSeq string to replace default check message if the -C input switch has been enabled. The default string is "Check \$d-\$m-\$e\$y \$h:\$n\n" failover: Used with the Round Robin switch (-R) and multiple remhost/logon/password, this keyword will NOT round robin but will always try the highest named 'remhost's first and, if not there, will failover to the second etc. offline: If Offline : wait in secs between attempts default: 60 secs For the first couple of attempts the program will wait about 4 seconds or so but all attempts after that will be at least 60 seconds apart. ascii: The end-of-line (whether CR, NL or CRNL) will be converted by the remote to whatever it requires. 'ipftp' converts to CRNL and ignores NULs before sending. default: binary (ie files received are the same as sent) remote-wants-crnl: default: file is sent unchanged Line endings are converted to CR NL and the file is sent 'binary' remote-wants-nl: default: file is sent unchanged Line endings are converted to NL only and the file is sent 'binary' remote-wants-cr: default: file is sent unchanged Line endings are converted to CR only and the file is sent 'binary' locale: use a different 'locale' ( look at the man pages for locale) Most computers are set to US English and never changed. Use this parameter to customise any date/time or Chr translation s The parameter MUST be a valid locale on your system! To find out what valid locale exist, type 'locale -a' on unix. Eg ; Set for for Brazil, portugese locale:pt_BR balance-seqno: Send the Sequence number to this Balance Group (see 'ipbalance') Use this to make sure the sequence number is always updated on any companion systems. balance-delete: Send a note for ipbalan/ipsvrd to delete the mirrored file on any remote server. balance-done-queue: (Balance group) Balance the done queue items balance-queue: Folder under /fip/spool to leave balance files for a copy of ipbalan. default 2balan ce redun-balance: balance group for redundant balance. Note you should only use balance or redun-balance but not both. ignore-timeouts: do NOT message if the sender timed out. Use this with caution! We recommend you use it ONLY for GET-polling where there may be times when the remote system dies but you do not care to know OR you have other tools to check for the problem. The actual message which is ignored is : Thu Nov 4 16:49:33 ipftp !x : **Error - FROMZZ - 220.127.116.11 - NULL : ** Timed Out - no response from remote 0 on-error: ignore/abort If we have an error from an 'ftpbefore'/'ftpbefile' etc, should the program stop processing the file or continue ? This sets up the default for that file. ** See also 'fipon-error' below. The default is 'ignore' Ths overrides the 'stop-on-error' keyword as it is more flexible log-line: Extra information in FipSeq for the Item Log when sending log-level:(number) adjust the amount of logging in the ALL log More is less ! so the higher the number, the less you get.. log-level:99 is the same as input switch '-q' ie minimum logging log-level:9 does not report every file, but does report end of tx. default is -1 for all logging log-get-every: No of seconds to log Get Accesses where nothing new was found this reduces the amount of repetitive log messages. maxattempts: No of attempts to send this file before stopping default is unlimited attempts, files is error are sent to woops maxattempts is only checked if the we are unable to send the file. slow-down: No of secs to pause between commands (between 1 and 5) If the remote host is old or overloaded, use this to slow down the link. inc-seqno: or increment-seqno: max-seqno: or maximum-seqno min-seqno: or minimum-seqno Set the Max and Min sequence numbers for this service Default is min of 1 and max of 99999 and increment is 1 Use FipSeq \%Z to pull out Note that \$z and \$s will always produce a 4 or 3 digit sequence number. Eg min-seqno:100 max-seqno:1000 inc-seqno:20 ; Start at 100 and by jumps to 20 until and including 1000 zapresforks: Path to Mac ResourceForks to zap those too (sending only) This is the path to the resource fork from the data folder. It assumes the filename is appended. This is NOT valid for '-1' single shot items, only spooled. default is to NOT zap eg for Helios zapresforks:.rsrc/ eg for Ushare zapresforks:% script: Optional Script to run AFTER the file has been sent successfully. eg script:/fip/local/FTP_NEXT_STAGE \YI-\YS tracker-script: Optional Tracking script - run AFTER script (if any) (Note FipHdr fields E7 (log-line message) and E8 (result code 0=success, -ve=neg) default: none resfork-type: ethershare/ushare Add a resource fork on the file at the remote site (sending only) resfork-creator: 4 letter creator type eg 'ZILA' default:FIPO resfork-filetype: 4 letter filetype eg '8BIM' default:TEXT resfork-template: template ResFork to copy resfork-path: Path to resfork eg for ethershare this is '.rsrc/' use-ssh:yes/no The commands are for a sftp on the remote server - see below Both ipftp and ipftpssl may be used for SSH/SFTP or SCP default is NO use-tls:yes/implicit/explicit/auth/no The commands are for a ftp running over SSL/TLS on the remote server NOTE - ipftpssl and NOT ipftp must be used for SSL/TLS default is NO no - normal, standard FTP on (normally) port 21 for the control yes or explicit - connect (normally) on port 21 in clear then use SSL for USER, PASS and data auth - connect (normally) on port 21 in clear, use SSL for USER, PASS then return to clear for non-data commands - but use SSL for all data implicit - connect (normally) on port 990: use SSL for all control and data tls-auth: (XXX) AUTH type for TLS/SSL default: TLS Valid entries are TLS, SSL, TLS-C (whatever that is !) and something starting 'X-' which will be something homegrown ! NOTE that for all versions of SSL the method string is "SSL" (this string is case sensitive according to the RFC) eg tls-auth:SSL ssl-method: tls tls1 tls1.1 tls1.2 sslv2 sslv3 sslv2and3 Version number to use for TLS/SSL default: 999 for current default (2 or 3) (only the digits are significant, so add other text to make it readable) For 'modern' connection, pls do NOT use sslv2 ! as it is deemed insecure If default it will check the available list and pick the highest. The default is currently 23 which on a modern server is sslv3 and tls1_2 !) ssl-password: (password) ssl-passwd: (password) default: none Optional password if the handshake requires a shared secret ssl-cert: (name of a PEM certificate file) default: none ssl-root-cert: (name of a root PEM certificate file) defaunt: none Optional certificates - held in tables/ssl ssl-verify: yes/no verify certificates default: yes ssl-ciphers: (list) acceptable ciphers (use 'openssl ciphers' to list) default: "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS" throttle-speed: (no of KILOBITS per second maximum) If the TCP pipe being used is only 64kbps, then large files may timeout. ie a small 1k file should take under a second, but a 1 mb file will take about 60 seconds to send normally. So if you have set the 'timeout' value at 60 secs, they it might abort before the data could reasonably be send. Use this command to pace the link better. Note the parameter is in KBITS/s which is the normal way of describing a link and NOT BYTES which is the normal way of describing a file. This does NOT stop/slow the data being sent - just delays the timeout for big files. eg for an E1 (2 megabit) line throttle-speed:2048 T1 (1.54 mps) throttle-speed:1544 56kb line throttle-speed:56 Line you know is overloaded - ADSL perhaps - throttle-speed:33 keep-connection-open: (secs) For sending files, do NOT close the connection after each series of files and keep the connection open. This means any AFTER commands (ftpafter:) are ONLY done on error. The parameter is the number of seconds between accesses (which is just a 'PWD') to make sure the line has not been closed prematurely by the remote end or a network device. add-md5-signature: (FipHdr field) Create an MD5 signature for the outbound files (Data Part only) and put it in a FipHdr field. This FipHdr can then be used as a 'ftpbeffile' or 'ftpaftfile'. default: none send-multiple-file:yes Sending multiple files that are related (eg XML companion file to a JPEG) These files are located in a separate folder and MUST exist before sending NOTE - the best way is to zip all the files up and send the single zip ! Otherwise you can get all sorts of problems when the transmission breaks (which it will do for big files on occasion) half way thru sending. send-multiple-que: (name of folder) specify a folder where the files are - this is a must send-multiple-key: (FipSeq) As the folder MAY have lots of files, specify a KEY or STUBname which can be used to find all related files (ie if you do an ls for this key in this folder you ONLY get the files you want) Often the key is a part of the incoming filename, or the contents of a FipHdr field You must also specify ONE of these send-multiple-all:yes send everything matching the key send-multiple-ext:(fileextension) only send files with this extension - this can be a FipHdr field And thes remaining parameters are all optional send-multiple-fiphdr: yes/no depending on whether you need to sent the FipHdr too send-multiple-zap: yes/no zap after successful send ? send-multiple-remote-name:\$e\$y\$i\$d_\$h\$n\$b_\E1.\$z name for remote if different - might want to add a unique string - date/time for example Where sections of FipHdr fields are required or changes to the output style, use keywords : fixed, partial, combie, optional, repeat, newdate and/or style. (see The SysAdmin manual for more information). They are normally specified : fixed:QZ 1234543 partial:QT ST,3,2,U,<,> combie:QY ep|na,(0000000)a option:QE ep,11,7,s repeat:QK XK,-,3 or repeat:QP PK,,4,#X style:QS XN,%.03d replace:QN NN abc=DEF def=GHI newdate:QT hours-3 "\ZD-\ZM" unique:QU XC The input file can optionally be just a FipHdr file with pointers to where the data resides. This can be useful where large PDFs or JPEGs or other binary files need to be ftp'ed but you do NOT want to move/copy them inside the system. To do this, use FipHdr fields FTP_EXTERNAL_FILE: (full path name to file) A single file before and binary file before (or after) may be specified by : FTP_BEFORE: syntax - same as 'before' FTP_AFTER: syntax - same as 'after' FTP_FILE_BEFORE: syntax - same as 'beffile' FTP_FILE_AFTER: syntax - same as 'aftfile' FTP_BINARY_BEFORE: syntax - same as 'binary-beffile' FTP_BINARY_AFTER: syntax - same as 'binary-aftfile' Another FipHdr field can also have bearing on this : FTP_ZAP_EXTERNAL: which deletes the external file if transmitted with no errors (default is to leave the external file alone). FTP_LIST_FILE: (filename or path/filname in FipSeq) The filename is used for a new file holding a LIST of the current remote folder. (same as -F) Input Parameters are : (all Optional) -i : queue to scan for input default: spool/2ftp This can be specified as a queue under /fip/spool or if it starts with a '/', the complete pathname. OR -1 : name of file to send where we are sending a single file. def:none This can be specified as a queue under /fip/spool or if it starts with a '/', the complete path and filename. The input file is NOT deleted in this case. OR -G : poll remote queue for files to Get default: local spool This parameter sets the number of seconds to wait between Polls. Only one parameter file is used - specify with the '-z' input switch. This MUST have at least one 'getfile', 'getallfiles', 'get-and-zap' or 'get-and-zapall' line (or be used with the -D switch) If the parameter is zero - '0' - then it is polled once only If > 0, the minimum time is 10 seconds unless modified by 'minimum-poll-interval:' OR -D : get the directory listing Only one parameter file is used - specify with the '-z' input switch. Also specify '-G 0' for a one-off grab or '-G 99' for a grab every 99 secs This MUST have at least one of the following in the parameter file: ftpbefore:fipdir (for short listing of just the filename) or ftpbefore:fiplongdir (for a long list in the remote servers own style) or ftpbefore:fipstddir (for a long list in the generic style - using MLSD) -z : default parameter file default: tables/ftp/FORMAT Less often used parameters ... -4 : use socks4 to skip thru a proxy default: no proxy -5 : use socks5 to skip thru a proxy default: no proxy -A : spool a folder - then stop when it is empty default: keep spooling or (-1) do single normally you will -Y for no offlines for this see below -b : generate bandwidth statistics default: no -B : default balance group for skip files default: none (see skip-balance-group parameter) -c : list of hosts to track for check messages. default: none -C : Interval in secs for Check Messages default: none If no data file has been sent for a certain number of seconds then a Check Message may be sent. Change the text of a Check Message with the 'check-message' keyword. -d : done queue for sent files default: none Normally files are deleted after sending. -F : force all files to be JUST the List File. default: no -h : extra FipHdr info default: none Used for cases where external information is not in the FipHdr of the file - system variables for example This is generally more use for GET files rather than SEND. It can be used for extra logging information or to fill in parameters from a script. EG a parameter file GENERIC.GET might have an line 'remhost:\BN' while the script has ipftp -z generic.get -h '\#BN:10.10.20.31' etc etc -H : alternate host name default: none use this for checking primary/secondary for clustered systems where the actual hostname differs from the cluster name ie if the host name may be fipcluster-a or fipcluster-b but there is a cluster hostname of fipcluster -k : block size to send data default: 32 for 32k This can be any number from 1 to 32 (in kilo bytes). For clients with bad connections, smaller blocks sometimes work better. -K : ignore the skip file and skipdetails files. default: use if specified -l : do NOT log files sent/received default: log -m : Use main Sequence number for Check Messages default: use different -M : File to replace "check-message" keyword default: none -o : output queue for get files default: spool/2go -O : offline queue for send files default: spool/ftp_offline_(-i name) where (-i name) is the name of the input spooler (normally '2ftp') See also -Y do NOT move un-sent files to the offline queue Note that if -W (do NOT watch offline q) is specified, the Offline queue is just that specified and the Input queuename is NOT appended. -p : Default control port number (for remote) default: 21 -P : Minimum data port number default: 32800 -q : quiet mode - do not message if we find directories or other non-files are found in the input queue - just ignore them. default: log message if directory found For get files this will NOT message if no file(s) are found or the file is to be skipped. -R : round robin if more than one host specified default: no -s : Default Remote host name default: none -S : log the trace of each transaction default: do not This generates a one line log of each file sent is stored in log/remote_trace with a name of 'date_(DF)'. -t : network file wait for files arriving default: no wait from across a network - using NFS perhaps -T : default timeout for messages back from default: 120 secs the remote system. Shorten this for quick or internal networks to 20-30 secs or shorter. This MUST be between 10 and 3000 seconds. -V : make all connections Passive / PASV default: no This should be used with care as not all FTPd's can handle PASV connections It is better practice to use the parameter 'passive-connection:' for those sessions that must be PASV than to use -V for all. -W : do NOT scan the offline queue for sends default: do Use this flag where a 2nd 'ipftp' (or other program) is scanning the offline queue of this ftp. -x : default no of files to send (or get) before closing the connection def. all This is overridden in the parameter file by 'onefile' or 'maxfiles' -y : (secs) wait this time if an error occurs AND there is no offline queue at all default: oh yes there is On error, wait this number of seconds but do not ignore any files Ie keep going, but pause after an error. -Y : there is no offline queue at all default: oh yes there is and on error, handle files in the normal files (ie ignore all files for the same address for 60 secs) -Z : force all traffic to use the default parameter file (ie either FORMAT or the '-z' switch) default: use DF -v : print version no and exit ------------------------------------------- A Checklist for a new Grab or Push using ipftp ... Things that you can check (you probably have, but here is a small list) 1. Which FTP ? - Is it normal FTP or normal FTP over secure link (FTPS) or Secure FTP over SSH (SFTP or SCP )? ipftp can handle all three on Linux or Solaris but you have to add extra middleware (openssl and openssh) for the secure ones on Windoze. Note that for the middle option FTP over a secure link, you must use ipftpssl not ipftp. 2. Connectivity - Test it manually using FTP (from the Fip Server - NOT from your desktop or your home) to check if your/their firewalls have been setup correctly. It also takes Fip out of the equation - if the manual tests do not work, dont expect Fip to do any better ! 3. Active/Passive - If the connection passes through more than one firewall, one will have to be set correctly to allow 'High-Port FTP access' So when testing manually, look for any message with the word 'Passive' in. You can (normally - ie if not the dreadful Windows FTP program) toggle active/passive using the PASS command. Try doing a 'dir' of the remote folder - if it hangs, you have the wrong setting. Try doing a 'get of a file' if it hangs, either you have the wrong setting or permissions do not allow you to get. In the ipftp parameter file, add the line 'passive-connection:yes' if you want to make the link passive. To make the link active, comment the line out. 4. Sub-folders - if there are any subfolders you need to check, use ftpbefore:cd (subfoldername) 5. Permissions on the remote server - Do you have permission to delete file (if using get-and-zapall) or rename (if using get-and-moveall) ? Test it manually. So many sites lock things down and it is common for the remote administrators to lock it down TOO much - especially if the data is in a subfolder ! ------------------------------------------- Notes ... -- For a single shot GET - getallfiles or whatever - try ipftp -G 0 -z jerry where 'jerry' is the name of your parameter file -- When Sending - Watch out for remote machines which cannot handle long filenames. Use 'newname' to clean that up. -- When Sending - Watch out for remote machines which cannot handle certain chrs in filenames. Use 'formatname' to clean that up. Especially 'formatname:nt' for Win2K boxes (which HATE ':' etc) or older Macs (which needed the filename trimmed to 31 chrs) -- When Sending - Watch out for duplicate file names on the remote box. Often you may want to delete the file of the same name first or use 'uniquename' to force the remote server to accept it. eg ftpbeffile:dele \SN or uniquename: -- When Sending - if using 'uniquename' (which is an FTP STOU command) and something goes wrong, some remote hosts will leave an empty file with another seqno extention in the same queue. -- Any commands specified by 'ftpbefore' are done once only before the first file of a series of files all with the same Parameter file. -- Note that in the standard FTP on older versions of Win2K, there is a 'feature' that says you HAVE to wait a few milliseconds after a 'delete', so there is an extra FIP ftp command called 'fipdelay' which will wait a second before continuing. ftpbeffile:dele /tmp/pporange/oinky ftpbeffile:fipdelay ftpbeffile:fipdelay ftpbeffile:fipdelay Note from version 18i2 'fipdelay can have a number after for the number of seconds to wait which is in the range 1 to 60. eg to wait 25 seconds : ftpbeffile:fipdelay 25 This is also useful where the remote system is slow(ish) and the files to grab are large and may take more than a second to build - Audio, Video and some pix/jpegs for example - so add a second of two to the 'ftpbeffile'.. just in case. ftpbeffile:fipdelay 2 -- For Get files, Parameter 'check-compare-file' will GET the contents of this file and compare it to the previous contents. ONLY if the contents diff, will any new files be grabbed. eg check-compare-file:.timestamp If there is a file called '.timestamp' on the remote, the CONTENTS are checked and if different, the folder is scanned for the changes. - On old Windows2K servers, there is also another feature where the Server loses all the permissions on a drive. Which means you can only create zero-length files. Microsoft says the problem does not reappear if you reboot! -- For GETs and SENDs FipHdr field 'EN' should be the filename on the remote system. This can be changed using the "newEN" parameter. For GETs a second field 'E1' (and 'ZZ' if required) will be just the filename with no path information. Note that both these fields have been cleanedup of funny chrs which are bad for the local server. So for GETs, another field 'ZO' will hold the name as it was on the remote. (For SENDS, field E2 will have the input folder name for Logging purposes only) -- There is also a means to add a flag file - a unix 'touch' - using an extra FIP ftp command called 'fiptouch' which will create a zero length file. ftpafter:fiptouch /trigger/incomingFile/\EN -- An extra FIP Ftp command 'fipblockfile' will check to see if a file of this name is on the remote server. If it is there, the file to send is NOT sent. eg ftpbefore:fipblockfile stop.all If there is a file called 'stop.all' on the remote, NO files are to be sent. -- An extra FIP Ftp command 'fipallowfile' will check to see if a file of this name is on the remote server. ONLY if it is there, will the file be sent. eg ftpbefore:fipallowfile ok.txt If there is a file called 'ok.txt' on the remote, files will be sent. -- An extra FIP ftp command 'fipdelete' will delete ALL files in the current folder of the remote server. Use with Care ! Make sure you are in the right area first. In fact you may wish to use a mask to make sure you are only deleting certain files: eg ftpbefore:fipdelete *.txt -- For continuous feeds where data is dribbled to a/several remote hosts, Check Messages allow you to notify the remote site that everyting is still working. -- For Check Messages to be sent, an interval (-C) must be specified. In each parameter file the 'check-message' string can have the message to send OR the default will be used OR the contents of the (-M) check message file is sent. - Using 'get-and-zap', 'get-and-zapall' and 'get-and-zapall-except'. These three commands will get a file(s) and delete off the remote server once it has been retrieved. BUT you MUST be very careful that you are in the right folder in order to do this - NEVER use 'root' or 'administrator' as the logon! or you risk pulling files from the root/top folder and totally damage the remote box !! Usually a specific logon with very restricted access is used for these commands. - Note that if there are multiple 'get-and-zap's and 'getfile'/'getallfiles' then it is the last specified which determines whether ALL are zapped or not. - Basically the message is do NOT mix Zaps with Non-zaps in the same parameter file. -- An extra FIP ftp command 'fipduplicate' will copy a file a second time with a different filename. It can ONLY be used for Send and in 'ftpaftfile' eg ftpaftfile:fipduplicate \EN.copy This differs from 'fiptouch' which creates a zero-length file as the WHOLE data is resent. -- Individual commands may be considered more important - such as : ftpbefore:cwd /sgt/bilko If the folder does not exist, you normally wish to STOP. But if you have a 'mkd' beforehand, it will work the first time but not thereafter So the fip builtin 'fipon-error' can be used BEFORE the line to set how to process : ftpbefore: fipon-error:ignore ftpbefore: mkd /sgt/bilko ftpbefore: fipon-error:abort ftpbefore: cwd /sgt/bilko or from version 18i, these can be replaced by the fipcd builtin which will mkidr if the folder does not exists and the cd to it ftpbefore: fipcd /sgt/bilko - Always rename files (ie RNTO/RNFR combinations) using the 'ftpaftfile' keyword and NOT 'ftpafter' because by the time 'ipftp' gets to the 'ftpafter' statements it is not at all bothered whether any files were sent or not! For ordinary FTP - and FTP over SSL use : newname:\SN.tmp ftpaftfile:rnfr \SN.tmp ftpaftfile:rnto \SN.gogogo For SFTP (over SSH) you can (only) use the builtin 'REN (oldname) (newname)' use-ssh:yes newname:RP\$z.tmp ftpaftfile:REN RP\$z.tmp \SN.pdf - Directory lists of the remote folder ftpbefore: cd /pub/secret/\EQ ftpbefore: fipdir ftpbefore: fiplongdir /fip/web/pages/junk/SHOWME.TXT ftpbefore: fipstddir /fip/web/pages/junk/STDDIR.TXT 'fipdir' gives a short and 'fiplongdir' a long list in whatever format the remote system gives ! There can optionally be a filename after the 'fipdir' to create/replace a file. Also 'fipbefore:fipstddir will do the same but in the system-independent format as used by MLSD. Note this will only work on servers that can process the enhanced MLSD command ! -- GET - There is a peculiar quirk of some FTPs - often on Linux or Win2k - that will display a complete tree of all files and sub-folders under a given folders. 'ipftp' assumes you do NOT want to drill down this list and, by default, will only pick up files from the top folder specified. If you do need to drill down, use parameter 'walk-remote-folder-tree:'. USE WITH CARE - you could cwd/cd to root on such a box and start pulling over EVERY FILE ON THE DISK !!! -- For Mac resource forks, 4 parameters specify : - where a template is to use as the basis - what software is being used - nfshare, ethershare, ushare - the file type - the path for the res fork from the data file eg resfork-path: .rsrc/ resfork-type:ethershare resfork-filetype:JPEG resfork-creator:FIPO resfork-template:rsrc_template The 'resfork-type' can be 'ethershare', 'ushare' or 'nfsshare'. Note that, except for resfork-type, none of these parameters are converted so case is important. The FileType and Creator are NORMALLY 4 uppercase letters/numbers and default to TEXT and FIPO respectively. -- Proxy There are several types of Proxy servers - http - ftp - socks4 and socks5 You will need to check with the network team what type it is. For Socks 4/5 - use these parameters to control socks-host: (hostname of the socks proxy) no default socks-port: (port number of the socks proxy) default: 1080 socks-user: (user name for the socks proxy) no default if nothing specified, assumed that there is none socks-pwd: (password for the socks proxy) no default It can also be forced to ALWAYS use socks4 or socks5 with the -4 or -5 input switches To use ipftp thru a proxy server, there is an 'ftpproxy:' parameter to add the extra information. This uses the same syntax as the 'ftpbefore' etc above. The flow is that you connect and logon to the local proxy server first.. .. then put something in to tell it where the remote FTP server is. Generally there are two type of Proxies Case 1 - where only USER and PASS are needed. Case 2 - where ACCT is needed. Case 1 - 'ftpproxy' is used to state the actual IP address or hostname of the proper remote host is added to the USER after an '@' sign. USER and PASS then give the logon and password for the remote. - 'remhost', 'logon' and 'password' should be the proxy server hostname/logon/password - 'ftpproxy' holds any logon and password of the remote eg ftpproxy:USER firstname.lastname@example.org ftpproxy:PASS eyeAteFigs Case 2 - the word 'proxy_server_user' is added as an extra word on the logon - 'remhost' is the proxy server - 'logon' is the REMOTE logon '@ REMOTE host - 'password' is the REMOTE password - 'ftpproxy' uses ACCT to hold the password of the PROXY server eg remhost:proxy_host logon:remote_logon@remote_host proxy_server_user password:remote_password ftpproxy:ACCT proxy_password ; use log to check what is REALLY happening !! logfile:/fip/log/ftp/FIPTEST.\$D and the log file gives ftp> open proxy_server Connected to proxy_server. 220 Blue Coat FTP Service Name (proxy_server:fip): remote_ftp_server_user@remote_ftp_server proxy_server_user 331 Enter password. Password:(remote_ftp_server_password) 332 Enter proxy password. Account:(proxy_server_password) 230 User xxxx logged in. Access restrictions apply. Remote system type is UNIX. Using binary mode to transfer files. ... etc -- SCP --------------------------------------------- Note that SCP is pretty lightweight - so while the transmission may be quicker, the lack of error checking (at the remote end) the lack of a final ack to say the file has been received is really an issue ! from Version 18j91 on FOR SEND ONLY ; logon password and host as normal remhost:10.3.3.32 logon:fip ; flag we need ssh use-ssh: ; onefile only onefile: ; getting the folder right ; AQ is in the FipHdr of the incoming file ; BE CAREFUL - make sure you get the right folder that you have permissions for !! combie:QQ AQ,tmp ; -t for push/put/send ; -p for preserve times ssh-scp-name:"scp -vpt /\QQ" ; check where your sshpass is !! and add password ssh-program:/fip/3rdparty/sshpass -pXXX /usr/bin/ssh ; possibly play with these settings ;;ssh-options:-oForwardAgent=no -oPermitLocalCommand=no -oClearAllForwardings=yes possible changes to these ;;ssh-scp-response-wait:30 ; permissions of files on the remote server - be very careful; can be FipSeq; make octal ! ;;ssh-scp-permissions:0644 -- Secure FTP -------------------------------------- Confusingly there are two - completely different - 'Secure' FTPs plus a defunct company called SecoueFTP etc etc 1. a more secure version of ordinary FTP which uses SSL/TLS in the same way http and httpS work for secure web sites. 2. a file copy layer which sits ontop of SSH - which has nothing at all to do with ordinary FTP The extra confusion is that SSH uses SSL - so dont mix the two up ! How do you know which is the one you want ? - What is the port number on the remote server ? port 21 - it is BOTH normal FTP and the SSL/TLS version (port 21 is the same as normal FTP) test with ordinary 'ftp' client port 990 - it is ONLY SSL/TLS version test with 'telnet' to (remhost) 990 and cut the connection once you are satisfied it connects port 22 - it is ONLY the sftp on top of SSH test with 'ssh' or 'sftp' --- 1. SSL/TLS This is the easy one.. YOU MUST USE ipftpssl for any/all ssl/tls traffic as ipftp blocks these commands. Prerequsite is the SSL layer which nowadays comes as standard on most platforms. Otherwise it can be downloaded from the installation kit or the website of the OS - Sun for Solaris, RedHat, Suse etc http://www.openssl.org For *nix, if you have gcc installed, it is usually easier (!) to compile from the latest sources at http://www.openssl.org/source/ For Win2k, there are precompiled versions at http://www.openssl.org/related/binaries.html which points at http://www.slproweb.com/products/Win32OpenSSL.html NOTE you generally have to also add the 'MicroSoft Visual C++ 2008 Redistributables' (vcredit) Pick the 32bit Light version unless you are running very very high volume stuff. To get ipftp to use SSL/TLS, all you need to do is add the keyword use-tls:(param) where param is no - normal, standard FTP on (normally) port 21 for the control (ie DO NOT use SSL/TLS) yes or explicit - connect (normally) on port 21 in clear then use SSL/TLS for USER, PASS and data auth - connect (normally) on port 21 in clear, use SSL/TLS for USER, PASS then return to clear (but use SSL/TLS for all data) This is normally the only version which works if using a Proxy server or a really nasty Firewall implicit - connect (normally) on port 990: use SSL/TLS for all control and data If passwords and/or certicates are required, use the ssl-cert etc to add --- 2.a SSH/SFTP The Fip SFTP is layered over OpenSSH (www.openssh.org) in a similar way to 'sftp'. So before using 'ipftp' in secure mode, openssh will need to be installed by your or your system manager. OpenSSH is available either as an installed option on most current Linux/Unix builds or prebuilt binaries can be downloaded (for Solaris, www.sunfreeware.com, Linux redhat etc, AIX www.bull.de). For Win2k - watch this space - the implementation is not tested or tuned yet. Once OpenSSH is installed, use the 'sftp' program to check it works with the remote site. Then to use secure ftp in 'ipftp', add the parameter use-ssh:yes Note that the commands for 'ftpbefore'. 'ftpbeffile' etc are restricted to those available in SSH/sftp There is also an extra parameter for buffering file sends ssh-window: (number of buffers) The number is from 0 to 64 and defaults to 2. This allows for (number) more packets to be sent before they are each acknowledged - which can greatly increase the speed of transmission as a whole on slower or long distance connections ---- 2.b SSH/Sending to a new system using sshpass On *nix, most modern systems use 'sshpass' (it can be downloaded from sourceforge.net) - beware it will not work on old versions of some OS as it needs a certain minimum version of OpenSSL. This layers the sshpass in the parameter file eg : sshpass-program:/usr/local/bin/sshpass The passwd is added from the 'passwd' parameter. See the doc on sshpass for which parameter to use - and change the paths for both sshpass and ssh to what they are on your system. sshpass-program:/fip/z/gnu/sshpass-1.04/sshpass - fills in password etc from parameter file - then runs the ssh in ssh-program (or default) or add the remote password after the '-p' : ssh-program:/fip/z/gnu/sshpass-1.04/sshpass -p 'xxx' /usr/bin/ssh - runs the whole thing ++ NOTE YOU MUST connect MANUALLY beforehand and accept any prompts such as: The authenticity of host ‘some_hostname (192.168…..)’ can’t be established Are you sure you want to continue connecting (yes/no)? ++ IF YOU do NOT do this, ipftp will hang on the unanswered question as nothing will reply 'yes' ++ ---- 2.c SSH/Sending on/to a Remote server in the cloud which is a clone of another Sometimes the remote hostname may have multiple ip addresses - run a 'dig' to check : dig ftp.remotehost.com .. and check the ANSWER section, eg : ;; ANSWER SECTION: ftp.remotehost.com. 10 IN A 18.104.22.168 ftp.remotehost.com. 10 IN A 22.214.171.124 The problem will be that in the 'known_hosts' file, ftp.remotehost.com can only be a single entry. So ipftp may/will halt when the 2nd address kicks in. You will get unpleasant key mismatch errors such as : WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! To work around this, add these parameters to ssh-options : -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no eg ssh-options:-oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no -oForwardX11=no -oForwardAgent=no -oProtocol=2 ---- 2.d SSH/Sending to a new system using shared keys If sshpass does NOT exist (and cannot be loaded for some reason) to make 'ipftp' work using SSH, the remote site needs to let you in without prompting for a password. This entails generating a pub key on one system and installing it on the other - and then generation a public key on the other and copying it to the first. You should read (and digest) the man pages for the ssh-keygen to work out what data needs to be added to which files. But generally for Unix/Linux, the default enc in 'rsa' so : the public key is in ~/.ssh/id_rsa.pub The contents of this file needs to be added (without changing, adding NL, spaces etc) to the file on the remote system called : $HOME/.ssh/authorized_keys and in some cases $HOME/.ssh/authorized_keys2 Email or somehow send it to the remote site but do not Note the key is normally too long to be edited, so email the whole file rather than view,cut,paste.email as that tends to mangle lineends. The '2' version should really have just the 'dsa' and 'rsa' but it does not seem to matter. Example - assume the hostname of the Fip sending system is 'sendersys' (logon fip) and the remote host is 'remotesys' (logon pif) 1. On sendersys system, logon with the same logon as Fip runs under - in this case 'fip' Generate the 3 different keys using ssh-keygen, cat them together and send them as a single file to the remote (some systems do not use the -t switch for ssh-keygen, so 'man' or 'howto' to find what it should be) ssh-keygen -t dsa ssh-keygen -t rsa ssh-keygen -t rsa1 cd cd .ssh cat *.pub > sender.k scp sender.k remotesys: use scp or sftp for this as it will add the remotesys into the 'known-hosts' file too. 2. On remotesys, logon with the same logon as Fip - in this case 'pif' ssh-keygen -t dsa ssh-keygen -t rsa ssh-keygen -t rsa1 cd cd .ssh cat *.pub > remote.k scp remote.k sendersys: 3. On sendersys system, create or add to the 'authorized_keys' file cd cd .ssh ls just to see what is there cat ../remote.k >> authorized_keys to add to an existing keys file or cat ../remote.k > authorized_keys if it does not exist cp authorized_keys authorized_keys2 some older versions need this chmod 644 authorized_keys* sort out permissions or the keys files chmod 755 . .. and the .ssh folder 4. On remotesys system, create or add to the 'authorized_keys' file cd cd .ssh ls just to see what is there cat ../sender.k >> authorized_keys to add to an existing keys file or cat ../sender.k > authorized_keys if it does not exist cp authorized_keys authorized_keys2 some older versions need this chmod 644 authorized_keys* sort out permissions or the keys files chmod 755 . .. and the .ssh folder 5. try it using ssh on the sendersys - first using just the hostname ssh pif@remotesys if you get a 'Add to known hosts message' - type 'yes' - then once logged on, exit and retry - second with host.full.internet.com ssh email@example.com if you get a 'Add to known hosts message' - type 'yes' - then once logged on, exit and retry Do you get prompted for a password - NO=it worked, YES=back to the drawing board. Some versions of openssh do not work automatically if the main folder for that logon does NOT have the correct permissions 'drwxr-xr-x' (chmod 755 .) or is owned by another logon like 'root'. If you are the only user of that folder, just change it - if not you will have to 'negociate' with the other users.... BEFORE running ipftp, 'ssh' must be able to connect and logon WITHOUT a logon. So if it does not, go back to the doc for you system as it may differ slightly to that described above. If you are unsure whether the ssh is working correctly, ssh debugging can be turned on by adding '-v -v -v' to the command line. ssh -v -v -v pif@remotesys Similarly ipftp can be put in a similar mode using the 'ssh-debug:yes' parameter and 'ipftp' and 'ssh' both spit out reams of meaningful information. If it does not work (!), it could be that 'ssh' is in a different folder ipftp looks in /usr/bin first and then /usr/local/bin - and if not found gives up with an error. Use 'ssh-program:/your/path/tossh/ssh' to state where it really is. If you need to use a non-standard ssh program or set of options, the defaults can be changed. The defaults are below - note the use of single quotes around each option. ssh-program:/usr/bin/ssh ssh-options:-q -oForwardX11=no -oForwardAgent=no -oProtocol=2 Plus ... if the number of connection-retries is NOT 5 -oConnectionAttempts=(number) if the remote port is NOT 22 -oPort=(number) if the timeout is NOT 120 seconds -oConnectTimeout=(number) if you are connection to newr systems, try -oBatchMode=yes For proxy connection, try '-oProxyCommand=nc -v -x(proxy host):(proxy port) %h %p' (Note the single quotes around the WHOLE parameter - not just the value - where there are embedded spaces) (pre 18j59 these options defaulted to : ssh-options:-q '-oForwardX11 no' '-oForwardAgent no' '-oProtocol 2' If you are having problems with the SSH link - bad dns, flaky network etc - you can specify a log filename for just the SSH messages. THIS FILE CAN GET VERY large - so pls remember to turn off (and also use FipSeq to make day/hour files) This file is in addtion to the normal fip 'logfile' ssh-debug:yes ssh-debug-logfile:/fip/log/ftp/SSH_NAME_\$e\$y\%i\$d.log There is also a ssh-debug-loghead: for adding more information at the start of exach transmission. --- Running parallel streams and an error stream Use ipwheel, ippost, iprobin or just the USERS file to route data into any of the 4 (or so) main ftp queues, and use -W, -Y and -O to handle errors. ftp1 ipftp -i 2ftp1 -O ftp_error -W ftp2 ipftp -i 2ftp2 -O ftp_error -W ftp3 ipftp -i 2ftp3 -O ftp_error -W ftp4 ipftp -i 2ftp4 -O ftp_error -W ftperr ipftp -i ftp_error -Y --- Troubleshooting a problematic connection To find where the problem lies .. our checklist is - Try it manually first to check it is ok - INCLUDING sending a file to make sure permissions are ok - Always try it manually from the Fip server to make sure the Firewall is ok too. - Use the log file parameter and see what it brings up. - check with both active and passive and check timings - which works better. - Check both servers are running FULL-duplex - if it still failing test at the TCP level with snoop/tcpdump/wireshark (which can lead to total obfuscation as you get too much info) The log file does give you everything sent and received on the Control Port plus the first chunk of data on the Data Port - so you can tell : - if the FipHdr has been stripped off by looking at the first part of the file sent part. or - if the ftpbefore has worked - as the CD or whatever will be noted with the reply (positive or neg) The FULL-duplex one is quite a tricky one as most network programs work of in half-duplex, including the FTP control port. The symptom is that small files are FTPed with no problem but larger ones (generally over 200K) always bomb out. With most servers being auto-sensing, all it takes is a network glitch for them to drop down to a slower speed or half-duplex - and they dont always readjust upwards when it has cleared. (On a sparc, /sbin/mii-tool is a useful thing to hack with) For the Active/Passive thing, if a firewall is eating the data connection in in one direction, it is quite hard to diagnose - except for any 425 error messages (which are always generated on the side which is starting the data connection, and so may not be ipftp). So swopping between passive and active is definitely one thing to try. The difficulty is when FTPing thru both your firewall and the remote client's to get to the remote server. One of the Firewalls MUST be setup to allow either the High port connections or traffic on a specified fixed-data-port. Note - active means the remote server will start the connection to the client (ipftp in this case) for any data (LS or FILE) - passive means ipftp will start the connect to the remote server for any data (LS or FILE) ---- Sending all files in a folder .. and stopping Use -A for where you want all the files from a folder to be sent - and then ipftp stops/exits. The wrinkle is - what happens if the connection fails; we want ipftp to keep pounding away until all the files have gone. It gets used for things like sending all 900 files to Amazon for a Kindle edition for example - where the chances of a network blip in the early hours of the morning are quite high, and we need ipftp to reesablish the connection and continue. Try it with no offline queue : (where QD is a date like QD:20130126) ipftp -A -i /fip/data/ebooks/\QD_kindle -h'WD:\QD' -Z -z goa_kindle -y 10 -Y ------------------------------------------------------------------------------ Version control ;19a06 26jul18 re-added socks proxy ;2 bytesTX for SFTP/SCP was zero ;3 savtimeout and drain-thsTotData added ;4 save sshOptions ;5 better failure tracking for sftp ;6 nat-pasv-address - woops LINUX is normally i86 so needs reversing ;18j97 29dec09 ;1-3 added TLS/SSL and fipstddir ;4 bugette for balance and programname ;5 \%q is time taken ;6 08mar10 added tracker-script ;7-8 added ConnectionAttempts an ConnectTimeout to ssh/sftp ;10-11 16apr10 only read pram file once for GETS ;12-17 7jun10 added error message in drain to describe what ! (15 added linger-on-close) ; 16 better wilds with SSH ;17 TLS'C' ;18 25jul10 bugette with -o not adding /fip/spool ;19 10aug10 added rename-prefix and rename-newname for SFTP ;20 29oct10 bugette with false error msg for gets ;21-23 10nov10 GET-skips fudge for W2k FTPD bug ;22 \%c bytes RCV or TX ;23 added -A onePass and doneque is now FipSeq and fixed GENERIC_GET bugette ;24 added send-external-fiphdr:yes/no ;25-31 redid keepConnectionOpen so it did NOT logon each time / do_connect-OpenData ;32-33 6dec11 woops remport NOT working for ssh and added remote-folder-exists ;34 12*425 is an error ;35 sshpass now works ;36 added list-folder-only plus added an error if Unknown keyword in parameter file ;37 15feb12 added max-data-port and min-data-port (min is same as start-data-port) ;38 20feb12 woops TLS/SSL only worked 'onefile:' ;39 \$c and CTRL bugette ;40 6jul12 added hidden 'CD ~' if 2 sequential files have different Parameter files but the same host/logon ;41 4sep12 added max-single-fiphdr-size ;42 21sep12 added fiphdr ZO original filename ;43-45 10oct12 added timeout alarm for sshpass plus added sshSftpVersionNumber ;46-47 17jan13 added send-multiples ;48 added getIsHash for Renames ; 49 cleanup ; 50 outque is now parseable ;51 added get-copy-file ; 52-53 tuned fipdelete/fipduplicate/cd/fipallowfile/fipblockfile for spaces in filename ;54 5dec13 fixed max-files for GETS too ;55 added file_trace ;56-57 minor logfile additions ;58 8aug14 passive-force-address added ;59-60 5sep14 added ssh-debug-logfile plus sshpass quotes to " not ' ** needs fipssl 01d ;61 3oct14 passive-connections:(FipSeq) ;62 11dec14 added local-filter:yes ;63 CWD~ should be CDUP ;64 added hash-in-fiphdr ! ;65-66 log msgs and newE2 for queue in logging ;67 24sep15 default linger 3 -> 20 ;68 back to CWD ~ not CDUP ;69-73 better check for IPaddress in place of hostname ; 74 added speedy for GET files ;75-77 added fiphdrMD5 chksum on GET files ;78-81 20jun16 added check-compare-file timestamp ;82 5jan17 added ssh-window ;83 added merge-fiphdr-on-getfiles ;85-86 18apr17 bugette in LARGE skipdetails ;87-88 5jul17 ssf buglette (in fipssf not ipftp) and recode added ;89-91 3nov17 added SCP send ;92 1feb18 use-ssh, use-tls, ssh-debug, ssh-window, tls-auth now parseable ;93 6feb18 onefile, max-files, append, uniquename, inc-seqno, max-seqno, min-seqno, ignore-timeouts, throttle-speed, max-attempts now parseable ;94 9feb18 bugette - sshSftpVersionNumber getting reset EVERY time we read the param file - should be 'if no connection' ;95 15feb18 added ssl-verify an ssl-ciphers ;96 19mar18 added nat-pasv-address ;018i35 12dec06 added skip-purge-after ;b 18dec06 PASV and 425 ;c 09feb07 ssh wait cleanup ;d-g 23mar07 added -x default max files and added fipallowfile ;h 20sep07 small internal chg to link with StingRay FTP Server - no PORT/PASV on logon ;i3 24oct07 added 'fipcd' and fipdelay can now have a noOfSecs after ;i4 1feb08 redunBalanced ;i5-10 19feb08 added send-external-file-name ;i6 max skip files upped 1000->20000 ;10-11 27may08 bug in fipdelay (from i3) and some remote servers do NOT allow skipdetails ;12-17 5jun08 added rename-prefix, rename-newname and BUG in PASV ; 17 for log-level: ;18 -justskip added ;19-21 added log-skip-details:(and stackoverflow doc) ;22-25 note_balance_action ;26-27 added -H for cluster host name and allow ZZ to be used in get-extra-fiphdr ;28-35 14sep09 check if proxy for failure and added minimum-poll-interval ; 30 HE+HR in GET FipHdr ;017z 24jul05 made H1-3, DR, DG, DT available to script, added -k and -y and script for GET ;b-d 12aug05 cleaned up all those extra PORT/PASV on skipped files ;e 15sep05 keep-connection-open added for sending ;f 14oct05 STOU/uniquename need tuning ;g 21oct05 Reworked ZeroLen GET files and ignore - now we do NOT add to skip notr zap/move etc ;h 23nov05 -W and -O mean that the offline queu is that specified and '2ftpX' is not appended ;j-k 15dec05 added shadow skip file - possibly on another drive. and added -K for ;l-o 09jan06 added add-md5-signature and cleanup of getfiles ;p 23jan06 added fixed-data-port ;q-r 23feb06 added put-empty-files (allow format: same as formatname) ;s-t 28jul06 added remote_trace style timings to send file done msg as H4 ;u 10aug06 wait3 64 bit and added -X and timing-stats ;v-w 07sep06 added priority-sends plus bugette with fip-onerror plus for GET, must have a dest:, so added a default. ;x-y 04oct06 new DestRedun ;z 06dec06 added start-data-port ;016z 10aug04 add skips to old too ;b 24sep04 speedy ;c-h 27oct04 added ssf-secure shell stuff (d,h 19nov04 hash-in-filename added) ;i-j 31jan05 better Win2K Gets ;k-m 07feb05 'dests' can be FipSeq and added outque: ;n-p 23apr05 remote-done-que is now FipSeq ;r-v 13may05 redid old skip list for WINNT and added -B for skip-balance-group ;w-x 15jun05 added -Y there is no offline queue ;y-z 30jun05 only go PASV if no Data port is open ;015z 06jun02 better control over filename..... ;a/b 18jun02 WINNT get was broken plus AS400 gives a funny file ended message. ;c/d 28jul02 added ignore-all-preparation ;e 12sep02 added creating ResForks on the remote system ;f 15sep02 bugette for GET with many entries - may miss the last ones ;g/h 10oct02 bugette with zero length files plus added check-primary-server-for-getfiles: and balance-skip-files ;i 16oct02 to strip the './' at the begining of WarFTP files. ;j 23jan03 added inc/min/max Seqno ;k 30jun03 bugette - skip file not zapped when nothing there. ;l-m 12sep03 redid timeout on connect ;n 30sep03 added logon to checking host ;o 06oct03 quiet on DataPorts..... ;p 12dec03 added -h for extra-fiphdr on GETS ;q 06jan04 default formatname for Gets ;r 19feb04 small mod - comments only at start of line ;s 26feb04 better skiplist ;t-w 25mar04 added FTP_LIST_FILE: or -F and fipdir ;y 29jun04 added -O offline queue ;z 02jul04 bugette in PASV/passive sends ;014z 08sep00 added minimum-log ;a 29nov00 improved logeachmsg ;b 20dec00 bugette for the case of maxattempts exceeded and no errque ;c-d 10jan01 bugette in fiphdr: ;e-i 30jan01 cleanups ;j 27mar01 better exit codes for single-shot and errors ;k 10jun01 added FTP_EXTERNAL_FILE: and FTP_ZAP_EXTERNAL: ;l-m 12jul01 do NOT normally walk the remote folders for getfiles. ;n-o 25jul01 added no-fiphdr-on-getfiles ;p 09aug01 PASV/passive-connection tuned. and -V added ;r-u 17aug01 added 'except's ;v 01oct01 fipdelete mods for 'fipon-error' ;w 16jan02 added loglasterrfile/log-last-error-file ;x-y 14mar02 added remote-trace and zap DO on attempts from filename ;z 30apr02 added script plus FTP_FILE/BINARY_BEFORE/AFTER ;013b 15aug00 added skip-file plus possible buggette in GetFiles and lousy networks ;a 19aug00 bugette in get-extra-fiphdr ;b 05sep00 added log of skip files ;012e 03apr00 bugette in LOG file - could get confused with many files in offline all withh different log files. ;a 26apr00 chkmsg seqno not right ;b 01may00 NT getfiles was buggy added fipon-error/on-error and maxattempts and slowdown ;c 17may00 added maxfiles: ;d 21jun00 better handling of the Done Queue ;e 09aug00 added 'zapresforks' ;011g 30nov99 added ftpproxy ;a 06dec99 mod to drain_it - occasionally missing a result code. ;b-c 10dec99 v occasionally, may miss change of seqno ;d-g 14jan00 allow space in NT filenames - redo seqno (copyright) 2019 and previous years FingerPost Ltd.