This program loops around a queue and sends or gets or lists files to/from a
Remote Server via FTP.

As FTP does hang on (regular) occasion, no response for a timeout period will
abort that file or transmission and the program attempt to restart where

By default, the program can be spooled against a queue or kicked off for a
single file or queue from a script using the '-1' input switch, which we call

For single shot, the files are NOT deleted unless the keyword 'del-files' is

Files are only deleted if sent correctly or, for getting files, if a reasonable
response was received like 'File Not Found'.

The FipHdr of each file to be sent is checked for the 'DF' field which, as per
normal for Fippies, is the name of a parameter file in tables/ftp. This
parameter file, tables/ftp/FORMAT by default, is used to describe the

The Remote Host Name can be a Fip Pseudo-Host (ie with an entry in
'tables/sys/DEST_REDUN' ). In this case, the primary host is checked for being
down and, if so, data sent to the secondary. If that too is down, files are not

File for remote systems that are offline or un-pingable are send to
'spool/ftp-offline' which will be rescanned on a regular basis. When such a
system recovers, the program will automatically send the collected data files
form 'ftp_offline'.

Files for retrieving can be left on the remote server untouched, deleted or
moved to a specified Done folder. There is a Poll input switch which will just
look once during the period for any files and get them accordingly.

A skip file may be created and maintained automatically for GET files which we
do NOT need to re-get. This is used where a 'get-and-zap' or 'get-and'move'
cannot be used.

ipftp can also be used to merely list the files on the remote server using the
-D input switch or the 'list-folder-only:' parameter.

Syntax for the parameter file are :
    ; comment
    remhost:    Hostname on remote system
        This overrides the '-s' Input switch.
        defaults to that specified in the '-s' Input switch
        This can be in FipSeq or just fixed text.
    remport:    Port on remote system
        This overrides the '-p' Input switch.
        defaults to that specified in the '-p' Input switch
        or failing that, defaults to 21
        This can be in FipSeq or just fixed text.
    logon:      Logon for remote system     default: anonymous
        This can be in FipSeq or just fixed text.
    password:   Password for remote system      default: none
        This can be in FipSeq or just fixed text.
    nopassword: The remote system does not need a password
        To send a password of nothing (which is different to NO pwd),
        specify the keyword with no parameter :
        eg  password:
        Normally a remote system wants a password even if it is blank.
    remhost,remport,logon,password,nopassword may be repeated another 9
        times for roundrobin which is enabled by the -R input switch.
    ftpbefore:  FTP command to execute after logon      default: none
    ftpbeffile: FTP command to execute before each file.    default: none
    ftpaftfile: FTP command to execute after  each file.    default: none
    ftpafter:   FTP command to execute before closing up    default: none
        Run some raw Ftp commands....
        There can be several commands for each section, each on a separate line.
        Note that the actual commands are NOT those from the user interface,
        but the raw ones as used by the FTP protocol. For weirdos you might
        want to check that the remote system will actually allow you to do these !
        Valid commands include :
            dele    file delete ** See notes below
            cwd change directory
            mkd make queue
            rmd remove queue
            pwd get current queue
            list    long  queue list - like an 'ls -l' or 'dir'
            nlst    short queue list - like an 'ls' or 'dir/w'
            nlst -lrt : long list, sorted by time.
                For lists, always 'cwd' into the queue beforehand.
            type    'a' for ascii or 'i' for image or binary
            rnfr    rename file : these two work in tandem ..
            rnto        rnfr=old name, rnto=new name.
            ; Do a MKDIR (who cares if it fails) and a Chg Dir before anything
            ftpbefore:mkd /pub/pporange
            ftpbefore:cwd /pubmp/pporange

            ; before each file, make sure we delete an old version before
            ftpbeffile:dele \SN

            ; at the end do a List to see what is really there.
            ; the list will be in the log file
            ftpafter:nlst -lt
        Note also there are some Fip-enhanced keywords :
        see below for explanations on what these do !

    ftpproxy:   FTP command to execute after logon      default: none
        to logon to yet-another FTP server.
        Please see the notes below

    ignore-all-preparation: ignore ALL the before/after/beffile/aftfiles
        and just send that RAW file out. Use this for big binary files that
        need NO alterations or additions.
    before: FipSeq string added before data of each file.   default: none
    after:  FipSeq string added after  data of each file.   default: none
    beffile: FipSeq file added before data of each file.    default: none
        Beffile is added after any 'before' string and before normal data.
    aftfile: FipSeq file added after  data of each file.    default: none
        Aftfile is added after normal data and before any 'after' string
    binary-beffile: binary file added before data of each file.  def: none
        Beffile is added after any 'before' string and before normal data.
    binary-aftfile: binary file added after  data of each file.  def: none
        Aftfile is added after normal data and before any 'after' string
    first:  Name of optional file in tables/ftp that is sent
        on startup and on each scan of the queue when a new file has
        been found.  The text of the file is in FipSeq. default:none
    last:   Name of optional file in tables/ftp that is sent when all the
        files in the queue have been sent and before we rescan the
        queue.  The text of the file is in FipSeq.  default:none

    check-compare-file: For Grabbing, Compare the contents of this file BEFORE
checking for new/changes    default: none
        Use this for timestamp file which is changed on the remote when there are new
or corrected data files available.

    hostname: The hostname for the system 'ipftp' is running.
        This defaults to the hostname the system was booted with.
        However for systems with multiple ethernet addresses,
        this is used to tell the remote system the IP address to use.
        (Note for versions 06c+, this is no longer required).
    del-files: Delete the files once sent. This is for -1 single file only.
            default: for -1 send one file, do NOT delete input file
            default: for spooled folder, delete ALL input files
                once sent correctly.
    timeout: Timeout period when sending for no response    default:120 secs
        The default timeout can be modified by the -T input switch.
        If you are sending to a host which is only across the computer
        room, you should take this timeout down to 10 or 15 secs.
    connection-retries: No of retries if the first connection
        fails. The default is 5. Which means a remote system
        which has disappeared can hold up the next traffic by up
        to (5) * (120 secs timeout) = 10 mins !.
    passive-connection: (yes/no) This causes the remote server to enter PASV mode
(if it can).
        The default is NO for 'active' if the command is NOT specified
        The default is YES for 'passive' if the command is specified with no
        From version 18j61, NO or YES can be in a FipHdr field in the file being sent
            in the FipHdr : PC:yes
            eg passive-connection:\PC
    passive-force-address:yes/no                    default: yes
        Force the IP address for a Passive Data connection to be the same as the
        Sometimes the remote server has a funny NAT address - use this to 'clean' it
    linger-on-close: Timeout in secs for the data to be sent.
        For slow connections, increase this,
        default: 20 (secs) to allow up to 20 secs max before cutting the call

    doneque: Queue to move original files once sent.    default: none
    errorque: Queue to move original files if NOT sent. default: none
        This is used ONLY where systems do NOT have a valid IP address
        because there  is no entry in the host file, or it has been
        missplet or, more likely, DNS has flipped.
        If the remote host is out there but the Logon is refused or,
        once logged on, we were unable to write the file, the files
        are stuffed in 'spool/ftp_offline' or if the input queue
        is NOT the default, spool/ftp_(inputque)_offline
    outque: For get files only, this is the output folder for incoming files
        This overrides the '-o' input switch and the default is 'spool/2go'.

    onefile: Send one file at a time to this host.
        After each file, 'ipftp' disconnects, waits and then reconnects
        before sending the next.  default: send all files in the queue.
    maxfiles: Send (or Get, from version 18j54) this number of files at a time
to/from this host.
        After so many files, 'ipftp' disconnects, waits and then reconnects before
sending/getting the next bunch.
        Use this to 'throttle' a connection where too many files (in or out) might
cause a system overload.
                                default: send/get all files
    logfile: Log file name                  default: none
        This should be the full path and filename of the the log file.
        The contents of the Log file are preserved and appended to.
        Use this to debug new feeds.
        You must make sure the queue exists for the log file as 'ipftp' will not
create it.
    logfile-dest: Destination for the log file      default: none
        This is the Fip DU destination for a log of the transmission.
        Use this to debug new feeds.
        Only specify EITHER 'logfile' or 'logfile-dest' (or neither)
        but not both.
    minimum-log:    Only log starts. stops and errors in the 'logfile'
        Normally the whole session is logged.
    dest:   Fip Destination for any Directory Listing or    default: none
        Retreive files. This should be a valid DU in the tables/sys/USERS file.
    send-no-data: Do NOT send any data from this file   default: send data
        Use this option to send just a headline with a before or beffile
        or to place a marker in a directory in the remote host.
    fiphdr: Send Fip Hdr if part of the file.       default: send data only
        Normally old the data part is sent and the FipHdr stripped off.
    hash-in-fiphdr: (FipSeq chr)            default:none
        A hash/pound (#) in a FipHdr field is mapped to this chr
        Normally hashes are end-of-field in a FipHdr. So it needs to be mapped to
something else and we changed it here on outbound.
    newname: Newname for the file(s) in FipSeq      default: as original
        The default is the same as 'newname:\SN'.
        Note that the 'first' and 'last' files are NOT renamed.
    uniquename: Force the remote server to store the file   default: no
        with a new name if it is a duplicate.
        Some systems allow you to give a filename, others do not.
    append: Force the remote server to either append to default: no
        an existing file or create a new the file
        Some systems allow you to give a filename, others do not.
        This is not available on all remote systems !
    forcename: force the filename on the remote system to be upper or lowercase
        forcename:upper                 default: no change
    formatname: Make sure the format of the filename on the remote system
        is correct for that system
        options are raw, unix, mac, nt, alphanumeric,pc
        default: unix on *nix and nt on Win2k
        YOU WILL NEARLY ALWAYS want to format the filename as different systems only
allow a subset of characters.
        Note that Win2K boxes do not like trailing dots so 'formatname:nt' also
strips them.
        To preserve the filename - such as if you are using ZO - use formatname:raw
for no changes at all.
    sizename: Maximum size of the filename on the remote system
        eg sizename:32                  default: no limit
        Note that normally the max for Windows/NTFS is 127, Unix is 255 and Mac is 31
        The maximum for sizename is therefore 255.
    send-external-file-name: Name on the remote system for any external files
        Use this parameter to send the external file separately to this name
        default - if there is an external file it is sent in the one file.
        if there is an external file containing the data (pointed to in the
FTP_EXTERNAL_FILE: fiphdr field), it may (or may not) have a FipHdr.
        Use this parameter to NO=strip it (or YES=leave it on) when sending.
        default is yes to send the fiphdr on the external file
    rename-prefix: (FipSeq) prefix for files that will be moved in get-and-move:
        If the remote server is a Windows server, permissions may prohibit a straight
move of duplicate filenames.
        Use this to make the filename unique.
        It is a Prefix as often, the extension needs to be kept!
        Eg  rename-prefix:\$e\$y\$i\$d\$h\$n\$b_
        If the original file is POPE.JPG, this will prefix the date and time to the
moved file.
    rename-newname: (FipSeq) newname for files that will be moved in get-and-move:
        If the remote server is a Windows server, permissions may prohibit a straight
move of duplicate filenames.
        Use this to make the filename unique. Remember FipHdr E1 will have the
        Eg  rename-newname:\E1.\$z
or  put-empty-files:put/send
        For SEND, if the file is zero length or empty this parameter will allow you
to :
            put or send - send the file and process as normal
            ignore      - skip to next file
        default is send

-- Listing Folders instead of sending ..... (note the default is to SEND)

        Only one parameter file is used - specify with the '-z' input switch.
        Also specify '-G 0' for a one-off grab or '-G 99' for a grab every 99 secs
        This MUST have at least one of the following in the parameter file:
            ftpbefore:fipdir    (for short listing of just the filename)
        or  ftpbefore:fiplongdir    (for a long list in the remote servers own style)
        or  ftpbefore:fipstddir (for a long list in the generic style - using MLSD)

-- Getting files instead of sending ......  (note the default is to SEND)
    getallfiles:        Get ALL files from the remote server.
    getallfiles:199*    Get ALL files starting '199' from the remote.
    getfile:        Get a file with this name from the remote.
        In this case no data is sent, as only the FipHdr is used to find
        the filename. There can be several lines of 'getfile'.
        Watch out for case sensitive remote systems !!
        No wild cards are allowed for single file gets but you can use FipSeq :
        eg  ; get a file from the remote system with the same name ..
            ; .. as the input file on the local system.
            ; always get the readme
            ; get todays file which has a 8 digit day extension
            style:QD    $D,%.08d
        You should also define the Fip destination DU using 'dest:',
        if not all the files all files will be sent to a DU of 'woops'.

        Use the 'except' versions where the mask is used to IGNORE files.
        eg  get-and-zapall-except:.pl
            Get all the files EXCEPT those with '.pl' in the filename
        NOTE the except string is just a simple compare and CANNOT include
        This is slightly inconsistent
        eg  get-and-zapall-except:*.pl
            will look for a filename INCLUDING the 4 chrs '*.pl'
        BUT it can be FipSeq
        eg  get-and-zapall-except:\$e\$y\$i\$d
            Get all files EXCEPT those with todays date in format YYYYMMDD like 20081231

        Commands 'get-and-zap', 'get-and-zapall' and 'get-and-zapall-except' :
            get then zap each file.
        Commands 'get-and-move', 'get-and-moveall' and 'get-and-moveall-except' :
            get then move each file to a done folder on the remote
            box setup by the 'remote-done-folder' keyword.
            PLEASE see below for comments on getmove and getzap.

        Note that if the remote FTP server is a Win2K box, the name of the
        file MAY have a '/' in it (or a Unix FTP server may have a file with a
        '\' in the name). Normally 'ipftp' ignores such files as it assumes they
        are sub-folders which should not be scavenged.
        Use the parameter 'walk-remote-folder-tree:' to grab these - see below!
    get-filter-case-sensitive:no FOR SSH connections only, you can force the
filter to be case INsensitive using this parametrer.
        default is YES it is case sensitive
    or local-filter:yes/no  Filter in ipftp NOT on the remote server - use this for
use with slightly-odd FTP sites like smartfile.com who do NOT allow wildcard
gets - try the remote manually using 'ftp', check that files exists and do a
'mget *'. If it returns '550 No files to get' when it should get something, set
this flag to YES and try with ipftp.

    remote-done-folder: Relative path on the remote box for the data files
        after they have been GET'ted.
        eg  remote-done-folder:../done
    remote-done-exists: Action if the file already exists in the
        Options are 'replace'   - replace existing with this one - (default)
            or 'add-ext'    - add the new version with a date_time extention
            or 'ignore-new' - just delete the NEW version once the transmission has been
completed successfully
        eg  remote-done-exists:ignore-new
    get-extra-fiphdr: extra FipHdr to attach to incoming, GET'ted files
        eg  get-extra-fiphdr:#ZI:#SU:REMOTE
        This will tell 'ipwheel' to archive the incoming data under
        the archive log REMOTE.
    no-fiphdr-on-getfiles: do NOT add a FipHdr to the GET'ted files.
        default is to add one which Date and Time fields etc.
    standalone-filename: (for Get files, do NOT put a Fip-style filename but use
the original filename (suitably modified for the actual system - ie strip '/'
if on UNIX, ':' for Win2k)
        The default is the normal Fip filename/fiphdr malarky.
        eg  standalone-filename:\SN_\$e\$y\$i\$d.fip
    get-copy-file: (full path name in FipSeq)   default:no
        Make a copy of each incoming file as this folder/name
        eg get-copy-file:/fip/data/raw.data/\$e\$y\$i\$d_\DF/\ZZ_\$h\$n\$b_\$z
        this leaves a copy in raw.data/(date)_(format)/(filename)_(time)_(seqno)
    hash-in-filename: (FipSeq chr)          default:\235
        A hash/pound (#) in a GET filename is mapped to this chr
        Normally hashes are end-of-field in a FipHdr. So it needs to be mapped to
something else in order to preserve it.
        For GET, if the file at the other end is zero length or empty
        this parameter will allow you to :
            get the file and process as normal
            ignore  skip to next file
        default is ignore
    skip-files: Name of a file in /fip/fix/ftp for holding the names of files
            brought over; so that only new files are received. It is
            remade from the LIST on every Get ALL.
            Note for unix/linix systems, the name is cases-SENSitive.
        eg  skip-files:fromFTP
        default: none
    check-skip-details: Yes/no
        The default is only to check if the file-to-get exists or not. But sometimes
        you need to track files which are updated - ie the name remains the same
        but the contents differ. Set this option for tracking changes too.
        Note it is slower and does take a bit more system resource as well
        as an extra skip file in /fip/fix/skip to hold the existing file times etc.
        default: no
    skip-purge-after: (hours) Number of hours to keep the skip entry
        default is 1.  You might want to tune this :
            make bigger if sites add/take off old material
            reduce the time if the same link is used for differnet data
    skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
        the skip file when changed (see doc on 'ipbalan')
        This is oftern used where more than one system is GETTING the same remote
        system/folder (usually with 'check-primary-server-for-getfiles').
    shadow-skip-file: (full path/filename)
        Name of a duplicate of the skip-file - probably on a network drive.
        Whichever file is newest is used.
    minimum-poll-interval: (secs)
        minumum delay between polls for gets    default: 10 secs
        Log why a file is being retrieved - new file, size change etc
    fixed-data-port: (number)
        if the number is >= 20, fix the data channel port number to this number and
do NOT change it.
    min-data-port: (number)
    max-data-port: (number)
        if the number is >= 20, the data channel port number will be in the range of
min <= number <= max
        defaults are min-32800, max 60000
    check-primary-server-for-getfiles: pseudo-host name that is specified in
        that is used whether the current host should be getting the files or not.
        ie in the ftp parameter file REMOTEGET is
        and in the DEST_REDUN is
            ; psuedohost    primary secondary
            remotewire  fip1    fip2
        and in the SYSTEM file for both fip1 AND fip2 there is a line
            rem1    local   ipftp -G 600 -Z -z REMOTEGET
        Then if fip1 is up, the ipftp on fip1 will always get while on fip2 it will
just check/loop.
            if fip1 is down, the 'ipftp' on fip2 will start getting.

    log-each-file: (dest) or
    logeachfile:(dest) Send a Success/failed msg to this destination
            for each file. There is no default. This log file is
            just a FipHdr with the following extra fields :
                DR-File Sent OK     DR:ok or DR:error
                DG-Will Retry later DG:retrying, DG:stopped
                DT-Some message text    DT:No connection
            default: no log created.
        The text for the DR and DG can be in FipSeq and so can contain
        FipHdr and other variables. As they are FipHdr fields, please
        do NOT put NL, CR etc in the fields.
        Note that System Variable \$q holds the time taken for transmission.
    log-last-error-file: (dest) or
    loglasterrfile:(dest) Same as for 'logeachfile' but it is sent ONLY
        after the last failed attempt where 'maxattempts' is specified.
        Only log-EACH-file or log-LAST-ERROR-file can be specified - not both.
    log-max-size: If logging is ON and a file is bigger than this size
    DRgood:(text)   Message for the FipHdr field DR on a   successful tx
            default: ok
    DRbad: (text)   Message for the FipHdr field DR on a unsuccessful tx
            default: error
    DGcont:(text)   Message for the FipHdr field DG if, after an
            unsuccessful tz, another attempt will be made.
            default: retrying
    DGstop:(text)   Message for the FipHdr field DG if no further
            attempts will be made as the file was sent successfully
            or the maximum no of attempts has been tried.
            default: stopped
    fiphdr-for-logeachfile: (FipSeq) or
    msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
            'logeachfile' or 'loglasterrfile' msg. This should be in FipHdr
            format and be in FipSeq. It can be used to pass FipHdr fields
            in the outgoing file into the log file.
            eg  msgeachfile:    DF:logdial\nSS:\SS\n
            default: nothing added
    stop-on-error:  Stop if you get a '500' series error in those commands
        you have specified in the 'ftpbefore', 'ftpafter' etc keywords.
        Normally these are ignored ..
        .. which is what you want if you have something like :
            ftpbefore:dele \SN
        to delete a similar file beforehand and the file does not exist :
            550 no such file or directory
        is the message received and can be ignored ..
        but then the same message is also given for 'cwd' that does not work !
        Where it is important, use 'stop-on-error' or 'on-error'.
        This is overriden by the 'on-error' keyword - see below.
    fip-syn-log:    add a Item Log line to the Fip Syndication Log file log/SYN.
            normally only the normal Fip log is kept up to date.
    newEN:  A 2 letter FipHdr field which will be the filename of the output file
or, for GET, the name of the file on the remote system.
        This defaults to EN but if you are already using EN, map it to another field.
        For GetFiles this can be the full pathname (especially with Lists),
        so use E1 (below) for the filename only.
    newE1:  A 2 letter FipHdr field which will be the name of the file to GET
        This defaults to E1 but if you are already using E1, map it to another field.
This field does NOT have any '/' or '\\' chrs.
        (for the original name on the remote server, use the fiphdr ZO and not E1)
    newEQ:  A 2 letter FipHdr field which will be the name of the folder on the
remote system for SENDing. Default is none
        This is usable only for the messaging back
    newE2:  A 2 letter FipHdr field which will be the name of the input folder for
SENDs for logging only. default: E2
    max-single-fiphdr-size: size that a single fiphdr field can be.
        default is 8196-sep2012 (was 2024)
        the minimum is 1024 and maximum is 31000

    log-max-size: If logging is ON and a file is bigger than this size
        then the first 64 chrs of each block are NOT stuffed in the log.
        This is because the log file can get massive !
        default is 'log-max-size:30000'
    check-message: FipSeq string to replace default check message if the -C
        input switch has been enabled.
        The default string is "Check \$d-\$m-\$e\$y \$h:\$n\n"
    failover: Used with the Round Robin switch (-R) and multiple
        remhost/logon/password, this keyword will NOT round robin
        but will always try the highest named 'remhost's first
        and, if not there, will failover to the second etc.
    offline: If Offline : wait in secs between attempts default: 60 secs
        For the first couple of attempts the program will wait about
        4 seconds or so but all attempts after that will be at least
        60 seconds apart.
    ascii:  The end-of-line (whether CR, NL or CRNL) will be converted
        by the remote to whatever it requires. 'ipftp' converts to
        CRNL and ignores NULs before sending.
        default: binary (ie files received are the same as sent)
    remote-wants-crnl:          default: file is sent  unchanged
        Line endings are converted to CR NL and the file is sent 'binary'
    remote-wants-nl:            default: file is sent  unchanged
        Line endings are converted to NL only and the file is sent 'binary'
    remote-wants-cr:            default: file is sent unchanged
        Line endings are converted to CR only and the file is sent 'binary'
    locale: use a different 'locale' ( look at the man pages for locale)
        Most computers are set to US English and never changed.
        Use this parameter to customise any date/time or Chr translation
        The parameter MUST be a valid locale on your system!
        To find out what valid locale exist, type 'locale -a' on unix.
        Eg  ; Set for for Brazil, portugese
    balance-seqno:  Send the Sequence number to this Balance Group
        (see 'ipbalance') Use this to make sure the sequence number
        is always updated on any companion systems.
    balance-delete: Send a note for ipbalan/ipsvrd to delete the mirrored
        file on any remote server.
    redun-balance: balance group for redundant balance.
        Note you should only use balance or redun-balance but not both.
    ignore-timeouts: do NOT message if the sender timed out. Use this with
        caution! We recommend you use it ONLY for GET-polling where
        there may be times when the remote system dies but you do not
        care to know OR you have other tools to check for the problem.
        The actual message which is ignored is :
Thu Nov  4 16:49:33 ipftp !x : **Error - FROMZZ - - NULL : **
Timed Out - no response from remote   0
    on-error:   ignore/abort
        If we have an error from an 'ftpbefore'/'ftpbefile' etc,
        should the program stop processing the file or continue ?
        This sets up the default for that file.
        ** See also 'fipon-error' below.
        The default is 'ignore'
        Ths overrides the 'stop-on-error' keyword as it is more flexible
    log-line: Extra information in FipSeq for the Item Log when sending
    log-level:(number)  adjust the amount of logging in the ALL log
        More is less ! so the higher the number, the less you get..
        log-level:99 is the same as input switch '-q' ie minimum logging
        log-level:9 does not report every file, but does report end of tx.
        default is -1 for all logging
    log-get-every: No of seconds to log Get Accesses where nothing new was found
        this reduces the amount of repetitive log messages.
    maxattempts: No of attempts to send this file before stopping
        default is unlimited attempts, files is error are sent to woops
        maxattempts is only checked if the we are unable to send the file.
    slow-down: No of secs to pause between commands (between 1 and 5)
        If the remote host is old or overloaded, use this to slow down the link.
    inc-seqno: or increment-seqno:
    max-seqno: or maximum-seqno
    min-seqno: or minimum-seqno
        Set the Max and Min sequence numbers for this service
        Default is min of 1 and max of 99999 and increment is 1
        Use FipSeq \%Z to pull out
        Note that \$z and \$s will always produce a 4 or 3 digit sequence number.
        Eg  min-seqno:100
            ; Start at 100 and by jumps to 20 until and including 1000
    zapresforks: Path to Mac ResourceForks to zap those too (sending only)
        This is the path to the resource fork from the data folder.
        It assumes the filename is appended.
        This is NOT valid for '-1' single shot items, only spooled.
            default is to NOT zap
        eg for Helios   zapresforks:.rsrc/
        eg for Ushare   zapresforks:%
    script: Optional Script to run AFTER the file has been sent successfully.
        eg  script:/fip/local/FTP_NEXT_STAGE \YI-\YS
    tracker-script: Optional Tracking script - run AFTER script (if any)
        (Note FipHdr fields E7 (log-line message) and E8 (result code 0=success,
        default: none

    resfork-type: ethershare/ushare
        Add a resource fork on the file at the remote site (sending only)
    resfork-creator: 4 letter creator type eg 'ZILA'    default:FIPO
    resfork-filetype: 4 letter filetype eg '8BIM'       default:TEXT
    resfork-template: template ResFork to copy
    resfork-path: Path to resfork
        eg for ethershare this is '.rsrc/'
        The commands are for a sftp on the remote server - see below
        Both ipftp and ipftpssl may be used for SSH/SFTP
        default is NO
        The commands are for a ftp running over SSL/TLS on the remote server
        NOTE - ipftpssl and NOT ipftp must be used for SSL/TLS
        default is NO
        no      - normal, standard FTP on (normally) port 21 for the control
        yes or explicit - connect (normally) on port 21 in clear then use SSL for
USER, PASS and data
        auth        - connect (normally) on port 21 in clear, use SSL for USER, PASS then
return to clear for non-data commands - but use SSL for all data
        implicit    - connect (normally) on port 990: use SSL for all control and data
    tls-auth: (XXX)
        AUTH type for TLS/SSL           default: TLS
        Valid entries are TLS, SSL, TLS-C (whatever that is !) and something starting
'X-' which will be something homegrown !
        NOTE that for all versions of SSL the method string is "SSL" (this string is
case sensitive according to the RFC)
        eg tls-auth:SSL

    ssl-method: (1,2,3,23,999)
        Version number to use for TLS/SSL       default: 999 for current default (2 or 3)
    ssl-password: (password)
    ssl-passwd: (password)                default: none
        Optional password if the handshake requires a shared secret
    ssl-cert: (name of a PEM certificate file)      default: none
    ssl-root-cert: (name of a root PEM certificate file)    defaunt: none
        Optional certificates - held in tables/ssl

    throttle-speed: (no of KILOBITS per second maximum)
        If the TCP pipe being used is only 64kbps, then large files may timeout.
        ie a small 1k file should take under a second, but a 1 mb file will take
about 60 seconds to send normally. So if you have set the 'timeout' value at 60
secs, they it might abort before the data could reasonably be send.
        Use this command to pace the link better. Note the parameter is in KBITS/s
which is the normal way of describing a link and NOT BYTES which is the normal
way of describing a file.
        This does NOT stop/slow the data being sent - just delays the timeout for big
        eg for an E1 (2 megabit) line   throttle-speed:2048
        T1  (1.54 mps)      throttle-speed:1544
        56kb line           throttle-speed:56
        Line you know is overloaded - ADSL perhaps - throttle-speed:33

    keep-connection-open: (secs)
        For sending files, do NOT close the connection after each series of files and
keep the connection open. This means any AFTER commands (ftpafter:) are ONLY
done on error.
        The parameter is the number of seconds between accesses (which is just a
'PWD') to make sure the line has not been closed prematurely by the remote end
or a network device.

    add-md5-signature: (FipHdr field)
        Create an MD5 signature for the outbound files (Data Part only) and put it in
a FipHdr field.
        This FipHdr can then be used as a 'ftpbeffile' or 'ftpaftfile'.
        default: none

        Sending multiple files that are related (eg XML companion file to a JPEG)
        These files are located in a separate folder and MUST exist before sending
        NOTE - the best way is to zip all the files up and send the single zip !
Otherwise you can get all sorts of problems when the transmission breaks (which
it will do for big files on occasion) half way thru sending.
    send-multiple-que: (name of folder)
        specify a folder where the files are - this is a must
    send-multiple-key: (FipSeq)
        As the folder MAY have lots of files, specify a KEY or STUBname which can be
used to find all related files
        (ie if you do an ls for this key in this folder you ONLY get the files you
        Often the key is a part of the incoming filename, or the contents of a FipHdr
    You must also specify ONE of these
            send everything matching the key
            only send files with this extension - this can be a FipHdr field
    And thes remaining parameters are all optional
        send-multiple-fiphdr: yes/no
            depending on whether you need to sent the FipHdr too
        send-multiple-zap: yes/no
            zap after successful send ?
            name for remote if different - might want to add a unique string - date/time
for example

  Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).

    They are normally specified :
        fixed:QZ    1234543
        partial:QT  ST,3,2,U,<,>
        combie:QY   ep|na,(0000000)a
        option:QE   ep,11,7,s
        repeat:QK   XK,-,3
    or  repeat:QP   PK,,4,#X
        style:QS    XN,%.03d
        replace:QN  NN  abc=DEF def=GHI
        newdate:QT  hours-3 "\ZD-\ZM"
        unique:QU   XC

The input file can optionally be just a FipHdr file with pointers to where the
data resides. This can be useful where large PDFs or JPEGs or other binary
files need to be ftp'ed but you do NOT want to move/copy them inside the
system.  To do this, use FipHdr fields
    FTP_EXTERNAL_FILE: (full path name to file)
A single file before and binary file before (or after) may be specified by :
    FTP_BEFORE:     syntax - same as 'before'
    FTP_AFTER:      syntax - same as 'after'
    FTP_FILE_BEFORE:    syntax - same as 'beffile'
    FTP_FILE_AFTER:     syntax - same as 'aftfile'
    FTP_BINARY_BEFORE:  syntax - same as 'binary-beffile'
    FTP_BINARY_AFTER:   syntax - same as 'binary-aftfile'
Another FipHdr field can also have bearing on this :
    FTP_ZAP_EXTERNAL:   which deletes the external file if transmitted with no
errors (default is to leave the external file alone).
    FTP_LIST_FILE: (filename or path/filname in FipSeq)
        The filename is used for a new file holding a LIST of the current remote
folder.  (same as -F)

Input Parameters are : (all Optional)
    -i : queue to scan for input            default: spool/2ftp
        This can be specified as a queue under /fip/spool or
        if it starts with a '/', the complete pathname.
OR  -1 : name of file to send where we are sending a single file. def:none
        This can be specified as a queue under /fip/spool or
        if it starts with a '/', the complete path and filename.
        The input file is NOT deleted in this case.
OR  -G : poll remote queue for files to Get     default: local spool
        This parameter sets the number of seconds to wait between Polls.
        Only one parameter file is used - specify with the '-z' input switch.
        This MUST have at least one 'getfile', 'getallfiles', 'get-and-zap'
        or 'get-and-zapall' line (or be used with the -D switch)
        If the parameter is zero -  '0' - then it is polled once only
        If > 0, the minimum time is 10 seconds unless modified by
OR  -D : get the directory listing
        Only one parameter file is used - specify with the '-z' input switch.
        Also specify '-G 0' for a one-off grab or '-G 99' for a grab every 99 secs
        This MUST have at least one of the following in the parameter file:
            ftpbefore:fipdir    (for short listing of just the filename)
        or  ftpbefore:fiplongdir    (for a long list in the remote servers own style)
        or  ftpbefore:fipstddir (for a long list in the generic style - using MLSD)
    -z : default parameter file         default: tables/ftp/FORMAT

Less often used parameters ...
    -A : spool a folder - then stop when it is empty    default: keep spooling or
(-1) do single
        normally you will -Y for no offlines for this
        see below
    -b : generate bandwidth statistics          default: no
    -B : default balance group for skip files       default: none
        (see skip-balance-group parameter)
    -c : list of hosts to track for check messages. default: none
    -C : Interval in secs for Check Messages        default: none
        If no data file has been sent for a certain number of seconds
        then a Check Message may be sent. Change the text of a Check
        Message with the 'check-message' keyword.
    -d : done queue for sent files          default: none
        Normally files are deleted after sending.
    -F : force all files to be JUST the List File.      default: no
    -h : extra FipHdr info                  default: none
        Used for cases where external information is not in the
        FipHdr of the file - system variables for example
        This is generally more use for GET files rather than SEND.
        It can be used for extra logging information or to fill in parameters from a
        EG a parameter file GENERIC.GET might have an line 'remhost:\BN'
        while the script has ipftp -z generic.get -h '\#BN:' etc etc
    -H : alternate host name                default: none
        use this for checking primary/secondary for clustered systems where the
actual hostname differs from the cluster name
        ie if the host name may be fipcluster-a or fipcluster-b but there is a
cluster hostname of fipcluster

    -k : block size to send data                default: 32 for 32k
        This can be any number from 1 to 32 (in kilo bytes).
        For clients with bad connections, smaller blocks sometimes work better.
    -K : ignore the skip file and skipdetails files.    default: use if specified
    -l : do NOT log files sent/received         default: log
    -m : Use main Sequence number for Check Messages    default: use different
    -M : File to replace "check-message" keyword      default: none
    -o : output queue for get files         default: spool/2go
    -O : offline queue for send files       default: spool/ftp_offline_(-i name)
        where (-i name) is the name of the input spooler (normally '2ftp')
        See also -Y do NOT move un-sent files to the offline queue
        Note that if -W (do NOT watch offline q) is specified, the Offline queue
        is just that specified and the Input queuename is NOT appended.
    -p : Default control port number (for remote)   default: 21
    -P : Minimum data port number           default: 32800
    -q : quiet mode - do not message if we find directories or other
        non-files are found in the input queue - just ignore them.
                    default: log message if directory found
        For get files this will NOT message if no file(s) are found
        or the file is to be skipped.
    -R : round robin if more than one host specified    default: no
    -s : Default Remote host name           default: none
    -S : log the trace of each transaction      default: do not
        This generates a one line log of each file sent is stored in
        log/remote_trace with a name of 'date_(DF)'.
    -t : network file wait for files arriving   default: no wait
        from across a network - using NFS perhaps
    -T : default timeout for messages back from default: 120 secs
        the remote system. Shorten this for quick or
        internal networks to 20-30 secs or shorter.
        This MUST be between 10 and 3000 seconds.
    -V : make all connections Passive / PASV        default: no
        This should be used with care as not all FTPd's can handle PASV connections
        It is better practice to use the parameter 'passive-connection:' for
        those sessions that must be PASV than to use -V for all.
    -W : do NOT scan the offline queue for sends        default: do
        Use this flag where a 2nd 'ipftp' (or other program) is scanning the
        offline queue of this ftp.
    -x : default no of files to send (or get) before closing the connection def.
        This is overridden in the parameter file by 'onefile' or 'maxfiles'
    -y : (secs) wait this time if an error occurs
        AND there is no offline queue at all        default: oh yes there is
        On error, wait this number of seconds but do not ignore any files
        Ie keep going, but pause after an error.
    -Y : there is no offline queue at all           default: oh yes there is
        and on error, handle files in the normal files (ie ignore all files for the
same address for 60 secs)
    -Z : force all traffic to use the default parameter file
        (ie either FORMAT or the '-z' switch)   default: use DF
    -v : print version no and exit

A Checklist for a new Grab or Push using ipftp ...

Things that you can check (you probably have, but here is a small list)

1. Which FTP ? - Is it normal FTP or normal FTP over secure link (FTPS) or
Secure FTP over SSH (SFTP)?
ipftp can handle all three on Linux or Solaris but you have to add extra
middleware (openssl and openssh) for the secure ones on Windoze. Note that for
the middle option FTP over a secure link, you must use ipftpssl not ipftp.

2. Connectivity - Test it manually using FTP (from the Fip Server - NOT from
your desktop or your home) to check if your/their firewalls have been setup
correctly. It also takes Fip out of the equation - if the manual tests do not
work, dont expect Fip to do any better !

3. Active/Passive - If the connection passes through more than one firewall,
one will have to be set correctly to allow 'High-Port FTP access'
So when testing manually, look for any message with the word 'Passive' in. You
can (normally - ie if not the dreadful Windows FTP program) toggle
active/passive using the PASS command.
Try doing a 'dir' of the remote folder - if it hangs, you have the wrong
Try doing a 'get of a file' if it hangs, either you have the wrong setting or
permissions do not allow you to get.

In the ipftp parameter file, add the line 'passive-connection:yes' if you want
to make the link passive. To make the link active, comment the line out.

4. Sub-folders - if there are any subfolders you need to check, use
ftpbefore:cd (subfoldername)

5. Permissions on the remote server - Do you have permission to delete file (if
using get-and-zapall) or rename (if using get-and-moveall) ? Test it manually.
So many sites lock things down and it is common for the remote administrators
to lock it down TOO much - especially if the data is in a subfolder !
Notes ...

-- For a single shot GET - getallfiles or whatever - try
    ipftp -G 0 -z jerry
where 'jerry' is the name of your parameter file

-- When Sending - Watch out for remote machines which cannot handle long
filenames. Use 'newname' to clean that up.

-- When Sending - Watch out for remote machines which cannot handle certain
chrs in filenames. Use 'formatname' to clean that up.
Especially 'formatname:nt' for Win2K boxes (which HATE ':' etc) or older Macs
(which needed the filename trimmed to 31 chrs)

-- When Sending - Watch out for duplicate file names on the remote box. Often
you may want to delete the file of the same name first or use 'uniquename' to
force the remote server to accept it.
    eg  ftpbeffile:dele \SN
    or  uniquename:

-- When Sending - if using 'uniquename' (which is an FTP STOU command) and
something goes wrong, some remote hosts will leave an empty file with another
seqno extention in the same queue.

-- Any commands specified by 'ftpbefore' are done once only before the first
file of a series of files all with the same Parameter file.

-- Note that in the standard FTP on older versions of Win2K, there is a
'feature' that says you HAVE to wait a few milliseconds after a 'delete', so
there is an extra FIP ftp command called 'fipdelay' which will wait a second
before continuing.
    ftpbeffile:dele /tmp/pporange/oinky
Note from version 18i2 'fipdelay can have a number after for the number of
seconds to wait which is in the range 1 to 60. eg to wait 25 seconds :
    ftpbeffile:fipdelay 25
This is also useful where the remote system is slow(ish) and the files to grab
are large and may take more than a second to build - Audio, Video and some
pix/jpegs for example - so add a second of two to the 'ftpbeffile'.. just in
    ftpbeffile:fipdelay 2

-- For Get files, Parameter 'check-compare-file' will GET the contents of this
file and compare it to the previous contents.
    ONLY if the contents diff, will any new files be grabbed.
    eg  check-compare-file:.timestamp
    If there is a file called '.timestamp' on the remote, the CONTENTS are checked
and if different, the folder is scanned for the changes.

- On old Windows2K servers, there is also another feature where the Server
loses all the permissions on a drive. Which means you can only create
zero-length files. Microsoft says the problem does not reappear if you reboot!

-- For GETs and SENDs FipHdr field 'EN' should be the filename on the remote
system. This can be changed using the "newEN" parameter.
For GETs a second field 'E1' (and 'ZZ' if required) will be just the filename
with no path information.
Note that both these fields have been cleanedup of funny chrs which are bad for
the local server.
So for GETs, another field 'ZO' will hold the name as it was on the remote.
(For SENDS, field E2 will have the input folder name for Logging purposes only)

-- There is also a means to add a flag file - a unix 'touch' - using an extra
FIP ftp command called 'fiptouch' which will create a zero length file.
    ftpafter:fiptouch /trigger/incomingFile/\EN

-- An extra FIP Ftp command 'fipblockfile' will check to see if a file of this
name is on the remote server. If it is there, the file to send is NOT sent.
    eg  ftpbefore:fipblockfile stop.all
    If there is a file called 'stop.all' on the remote, NO files are to be sent.

-- An extra FIP Ftp command 'fipallowfile' will check to see if a file of this
name is on the remote server. ONLY if it is there, will the file be sent.
    eg  ftpbefore:fipallowfile ok.txt
    If there is a file called 'ok.txt' on the remote, files will be sent.

-- An extra FIP ftp command 'fipdelete' will delete ALL files in the current
folder of the remote server. Use with Care ! Make sure you are in the right
area first. In fact you may wish to use a mask to make sure you are only
deleting certain files:
    eg  ftpbefore:fipdelete *.txt

-- For continuous feeds where data is dribbled to a/several remote hosts, Check
Messages allow you to notify the remote site that everyting is still working.

-- For Check Messages to be sent, an interval (-C) must be specified. In each
parameter file the 'check-message' string can have the message to send OR the
default will be used OR the contents of the (-M) check message file is sent.

- Using 'get-and-zap', 'get-and-zapall'  and 'get-and-zapall-except'. These
three commands will get a file(s) and delete off the remote server once it has
been retrieved. BUT you MUST be very careful that you are in the right folder
in order to do this - NEVER use 'root' or 'administrator' as the logon! or you
risk pulling files from the root/top folder and totally damage the remote box
!! Usually a specific logon with very restricted access is used for these

- Note that if there are multiple 'get-and-zap's and 'getfile'/'getallfiles'
then it is the last specified which determines whether ALL are zapped or not. -
Basically the message is do NOT mix Zaps with Non-zaps in the same parameter

-- An extra FIP ftp command 'fipduplicate' will copy a file a second time with
a different filename. It can ONLY be used for Send and in 'ftpaftfile'
    eg  ftpaftfile:fipduplicate \EN.copy
This differs from 'fiptouch' which creates a zero-length file as the WHOLE data
is resent.

-- Individual commands may be considered more important - such as :
    ftpbefore:cwd /sgt/bilko
If the folder does not exist, you normally wish to STOP.  But if you have a
'mkd' beforehand, it will work the first time but not thereafter
So the fip builtin 'fipon-error' can be used BEFORE the line to set how to
process :
    ftpbefore: fipon-error:ignore
    ftpbefore: mkd /sgt/bilko
    ftpbefore: fipon-error:abort
    ftpbefore: cwd /sgt/bilko
or from version 18i, these can be replaced by the fipcd builtin which will
mkidr if the folder does not exists and the cd to it
    ftpbefore: fipcd /sgt/bilko

- Always rename files (ie RNTO/RNFR combinations) using the 'ftpaftfile'
keyword  and NOT 'ftpafter' because by the time 'ipftp' gets to the 'ftpafter'
statements it is not at all bothered whether any files were sent or not!
For ordinary FTP - and FTP over SSL use :
    ftpaftfile:rnfr \SN.tmp
    ftpaftfile:rnto \SN.gogogo
For SFTP (over SSH) you can (only) use the builtin 'REN (oldname) (newname)'
    ftpaftfile:REN RP\$z.tmp \SN.pdf

- Directory lists of the remote folder
    ftpbefore: cd /pub/secret/\EQ
    ftpbefore: fipdir
    ftpbefore: fiplongdir /fip/web/pages/junk/SHOWME.TXT
    ftpbefore: fipstddir /fip/web/pages/junk/STDDIR.TXT
'fipdir' gives a short and 'fiplongdir' a long list in whatever format the
remote system gives !
There can optionally be a filename after the 'fipdir' to create/replace a file.
Also 'fipbefore:fipstddir will do the same but in the system-independent format
as used by MLSD. Note this will only work on servers that can process the
enhanced MLSD command !

-- GET - There is a peculiar quirk of some FTPs - often on Linux or Win2k -
that will display a complete tree of all files and sub-folders under a given
folders. 'ipftp' assumes you do NOT want to drill down this list and, by
default, will only pick up files from the top folder specified. If you do need
to drill down, use parameter 'walk-remote-folder-tree:'.
    USE WITH CARE - you could cwd/cd to root on such a box and start pulling over

-- For Mac resource forks, 4 parameters specify :
    - where a template is to use as the basis
    - what software is being used - nfshare, ethershare, ushare
    - the file type
    - the path for the res fork from the data file
    resfork-path: .rsrc/

The 'resfork-type' can be 'ethershare', 'ushare' or 'nfsshare'.

Note that, except for resfork-type,  none of these parameters are converted so
case is important.

The FileType and Creator are NORMALLY 4 uppercase letters/numbers and default
to TEXT and FIPO respectively.

-- Proxy

To use ipftp thru a proxy server, there is an 'ftpproxy:' parameter to add the
extra information.

This uses the same syntax as the 'ftpbefore' etc above.

The flow is that you connect and logon to the local proxy server first..
.. then put something in to tell it where the remote FTP server is.

Generally there are two type of Proxies
    Case 1 - where only USER and PASS are needed.
    Case 2 - where ACCT is needed.

Case 1 - 'ftpproxy' is used to state the actual IP address or hostname of the
proper remote host is added to the USER after an '@' sign.  USER and PASS then
give the logon and password for the remote.
- 'remhost', 'logon' and 'password' should be the proxy server
- 'ftpproxy' holds any logon and password of the remote
    eg  ftpproxy:USER abcde@
        ftpproxy:PASS eyeAteFigs

Case 2 - the word 'proxy_server_user' is added as an extra word on the logon
- 'remhost' is the proxy server
- 'logon'   is the REMOTE logon '@ REMOTE host
- 'password'    is the REMOTE password
- 'ftpproxy' uses ACCT to hold the password of the PROXY server

logon:remote_logon@remote_host proxy_server_user
ftpproxy:ACCT proxy_password
; use log to check what is REALLY happening !!

and the log file gives
ftp> open proxy_server

Connected to proxy_server.
220 Blue Coat FTP Service

Name (proxy_server:fip): remote_ftp_server_user@remote_ftp_server

331 Enter password.

332 Enter proxy password.

230 User xxxx logged in.  Access restrictions apply.
Remote system type is UNIX.
Using binary mode to transfer files.

... etc

-- Secure FTP --------------------------------------

Confusingly there are two - completely different - 'Secure' FTPs plus a defunct
company called SecoueFTP etc etc
    1. a more secure version of ordinary FTP which uses SSL/TLS in the same way
http and httpS work for secure web sites.
    2. a file copy layer which sits ontop of SSH - which has nothing at all to do
with ordinary FTP

The extra confusion is that SSH uses SSL - so dont mix the two up !

How do you know which is the one you want ?

- What is the port number on the remote server ?
    port 21     - it is BOTH normal FTP and the SSL/TLS version (port 21 is the same
as normal FTP)
                test with ordinary 'ftp' client
    port 990    - it is ONLY SSL/TLS version
                test with 'telnet' to (remhost) 990 and cut the connection once you are
satisfied it connects
    port 22     - it is ONLY the sftp on top of SSH
                test with 'ssh' or 'sftp'

--- 1. SSL/TLS
    This is the easy one..
Prerequsite is the SSL layer which nowadays comes as standard on most
platforms. Otherwise it can be downloaded from the installation kit or the
website of the OS - Sun for Solaris, RedHat, Suse etc
For *nix, if you have gcc installed, it is usually easier (!) to compile from
the latest sources at http://www.openssl.org/source/
For Win2k, there are precompiled versions at
which points at
NOTE you generally have to also add the 'MicroSoft Visual C++ 2008
Redistributables' (vcredit)
Pick the 32bit Light version unless you are running very very high volume

To get ipftp to use SSL/TLS, all you need to do is add the keyword
where param is
        no      - normal, standard FTP on (normally) port 21 for the control (ie DO NOT
use SSL/TLS)
        yes or explicit - connect (normally) on port 21 in clear then use SSL/TLS for
USER, PASS and data
        auth        - connect (normally) on port 21 in clear, use SSL/TLS for USER, PASS
then return to clear
                 (but use SSL/TLS for all data) This is normally the only version which
works if using a Proxy server or a really nasty Firewall
        implicit    - connect (normally) on port 990: use SSL/TLS for all control and

If passwords and/or certicates are required, use the ssl-cert etc to add

--- 2.a SSH/SFTP

The Fip SFTP is layered over OpenSSH (www.openssh.org) in a similar way to
'sftp'. So before using 'ipftp' in secure mode, openssh will need to be
installed by your or your system manager.

OpenSSH is available either as an installed option on most current Linux/Unix
builds or prebuilt binaries can be downloaded (for Solaris,
www.sunfreeware.com, Linux redhat etc, AIX www.bull.de).
For Win2k - watch this space - the implementation is not tested or tuned yet.

Once OpenSSH is installed, use the 'sftp' program to check it works with the
remote site.

Then to use secure ftp in 'ipftp', add the parameter

Note that the commands for 'ftpbefore'. 'ftpbeffile' etc are restricted to
those available in SSH/sftp

---- 2.b SSH/Sending to a new system using sshpass

On *nix, most modern systems use 'sshpass' (it can be downloaded from
sourceforge.net) - beware it will not work on old versions of some OS as it
needs a certain minimum version of OpenSSL. This layers the sshpass in the
parameter file eg :

The passwd is added from the 'passwd' parameter.

See the doc on sshpass for which parameter to use - and change the paths for
both sshpass and ssh to what they are on your system.

        - fills in password etc from parameter file
        - then runs the ssh in ssh-program (or default)

or add the remote password after the '-p' :
    ssh-program:/fip/z/gnu/sshpass-1.04/sshpass -p 'xxx' /usr/bin/ssh
        - runs the whole thing

You may want to try it manually beforehand and accept any prompts such as:
The authenticity of host ‘some_hostname (192.168…..)’ can’t be
Are you sure you want to continue connecting (yes/no)?

Otherwise if you are too bored to do that or have many hosts, add ' -o
StrictHostKeyChecking=no ' to the sshpass string.
(Note just to make you irritated, some versions of sshpass do NOT have the '-o'
input switch - so you HAVE to try it manually first.

---- 2.c SSH/Sending to a new system using shared keys

If sshpass does NOT exist (and cannot be loaded for some reason) to make
'ipftp' work using SSH, the remote site needs to let you in without prompting
for a password.

This entails generating a pub key on one system and installing it on the other
- and then generation a public key on the other and copying it to the first.

You should read (and digest) the man pages for the ssh-keygen to work out what
data needs to be added to which files. But generally for Unix/Linux, the
default enc in 'rsa' so :
     the public key is in ~/.ssh/id_rsa.pub
The contents of this file needs to be added (without changing, adding NL,
spaces etc) to the file on the remote system called :
and in some cases
Email or somehow send it to the remote site but do not  Note the key is
normally too long to be edited, so email the whole file rather than
view,cut,paste.email as that tends to mangle lineends. The '2' version should
really have just the 'dsa' and 'rsa' but it does not seem to matter.

Example - assume the hostname of the Fip sending system is 'sendersys' (logon
fip) and the remote host is 'remotesys' (logon pif)
    1. On sendersys system, logon with the same logon as Fip runs under - in this
case 'fip'
        Generate the 3 different keys using ssh-keygen, cat them together and send
them as a single file to the remote
        (some systems do not use the -t switch for ssh-keygen, so 'man' or 'howto' to
find what it should be)
            ssh-keygen -t dsa
            ssh-keygen -t rsa
            ssh-keygen -t rsa1
            cd .ssh
            cat *.pub > sender.k
            scp sender.k remotesys:
                use scp or sftp for this as it will add the remotesys into the
'known-hosts' file too.

    2. On remotesys, logon with the same logon as Fip  - in this case 'pif'
        ssh-keygen -t dsa
        ssh-keygen -t rsa
        ssh-keygen -t rsa1
        cd .ssh
        cat *.pub > remote.k
        scp remote.k sendersys:

    3. On sendersys system, create or add to the 'authorized_keys' file
        cd .ssh
        ls                      just to see what is there
        cat ../remote.k >> authorized_keys        to add to an existing keys file
    or  cat ../remote.k > authorized_keys        if it does not exist
        cp authorized_keys authorized_keys2     some older versions need this
        chmod 644 authorized_keys*          sort out permissions or the keys files
        chmod 755 .                 .. and the .ssh folder

    4. On remotesys system, create or add to the 'authorized_keys' file
        cd .ssh
        ls                      just to see what is there
        cat ../sender.k >> authorized_keys        to add to an existing keys file
    or  cat ../sender.k > authorized_keys        if it does not exist
        cp authorized_keys authorized_keys2     some older versions need this
        chmod 644 authorized_keys*          sort out permissions or the keys files
        chmod 755 .                 .. and the .ssh folder

    5. try it using ssh on the sendersys
        - first using just the hostname
            ssh pif@remotesys
            if you get a 'Add to known hosts message'
                - type 'yes'
                - then once logged on, exit and retry
        - second with host.full.internet.com
            ssh pif@remotesys.hoho.com
            if you get a 'Add to known hosts message'
                - type 'yes'
                - then once logged on, exit and retry
        Do you get prompted for a password - NO=it worked, YES=back to the drawing

Some versions of openssh do not work automatically if the main folder for that
logon does NOT have the correct permissions 'drwxr-xr-x' (chmod 755 .) or is
owned by another logon like 'root'. If you are the only user of that folder,
just change it - if not you will have to 'negociate' with the other users....

BEFORE running ipftp, 'ssh' must be able to connect and logon WITHOUT a logon.
So if it does not, go back to the doc for you system as it may differ slightly
to that described above.

If you are unsure whether the ssh is working correctly, ssh debugging can be
turned on by adding '-v -v -v' to the command line.
    ssh -v -v -v pif@remotesys

Similarly ipftp can be put in a similar mode using the 'ssh-debug:yes'
parameter and 'ipftp' and 'ssh' both spit out reams of meaningful information.

If it does not work (!), it could be that 'ssh' is in a different folder ipftp
looks in /usr/bin first and then /usr/local/bin - and if not found gives up 
with an error. Use 'ssh-program:/your/path/tossh/ssh' to state where it really

If you need to use a non-standard ssh program or set of options, the defaults
can be changed. The defaults are below - note the use of single quotes around
each option.
    ssh-options:-q -oForwardX11=no -oForwardAgent=no -oProtocol=2"
    Plus ...
    if the number of connection-retries is NOT 5    -oConnectionAttempts=(number)
    if the remote port is NOT 22            -oPort=(number)
    if the timeout is NOT 120 seconds       -oConnectTimeout=(number)
    if you are connection to newr systems, try  -oBatchMode=yes
    For proxy connection, try           -oProxyCommand='nc -v -x(proxy host):(proxy port)
%h %p'
(pre 18j59 these options defaulted to :
    ssh-options:-q '-oForwardX11 no' '-oForwardAgent no' '-oProtocol 2'

If you are having problems with the SSH link - bad dns, flaky network etc - you
can specify a log filename for just the SSH messages.
THIS FILE CAN GET VERY large - so pls remember to turn off (and also use FipSeq
to make day/hour files)
This file is in addtion to the normal fip 'logfile'
There is also a ssh-debug-loghead: for adding more information at the start of
exach transmission.

--- Running parallel streams and an error stream

Use ipwheel, ippost, iprobin or just the USERS file to route data into any of
the 4 (or so) main ftp queues, and use -W, -Y and -O to handle errors.

ftp1    ipftp -i 2ftp1 -O ftp_error -W
ftp2    ipftp -i 2ftp2 -O ftp_error -W
ftp3    ipftp -i 2ftp3 -O ftp_error -W
ftp4    ipftp -i 2ftp4 -O ftp_error -W
ftperr  ipftp -i ftp_error -Y

--- Troubleshooting a problematic connection

To find where the problem lies ..  our checklist is
- Try it manually first to check it is ok - INCLUDING sending a file to make
sure permissions are ok
- Always try it manually from the Fip server to make sure the Firewall is ok
- Use the log file parameter and see what it brings up.
- check with both active and passive and check timings - which works better.
- Check both servers are running FULL-duplex
- if it still failing test at the TCP level with snoop/tcpdump/wireshark (which
can lead to total obfuscation as you get too much info)

The log file does give you everything sent and received on the Control Port
plus the first chunk of data on the Data Port
- so you can tell :
    -  if the FipHdr has been stripped off by looking at the first part of the
file sent part.
or   - if the ftpbefore has worked - as the CD or whatever will be noted with
the reply (positive or neg)

The FULL-duplex one is quite a tricky one as most network programs work of in
half-duplex, including the FTP control port. The symptom is that small files
are FTPed with no problem but larger ones (generally over 200K) always bomb

With most servers being auto-sensing, all it takes is a network glitch for them
to drop down to a slower speed or half-duplex - and they dont always readjust
upwards when it has cleared. (On a sparc, /sbin/mii-tool is a useful thing to
hack with)

For the Active/Passive thing, if a firewall is eating the data connection in in
one direction, it is quite hard to diagnose - except for any 425 error messages
(which are always generated on the side which is starting the data connection,
and so may not be ipftp).
So swopping between passive and active is definitely one thing to try.
The difficulty is when FTPing thru both your firewall and the remote client's
to get to the remote server.
One of the Firewalls MUST be setup to allow either the High port connections or
traffic on a specified fixed-data-port.
Note    - active means the remote server will start the connection to the client
(ipftp in this case) for any data (LS or FILE)
    - passive means ipftp will start the connect to the remote server for any data
(LS or FILE)

---- Sending all files in a folder .. and stopping

Use -A for where you want all the files from a folder to be sent - and then
ipftp stops/exits.

The wrinkle is - what happens if the connection fails; we want ipftp to keep
pounding away until all the files have gone.

It gets used for things like sending all 900 files to Amazon for a Kindle
edition for example - where the chances of a network blip in the early hours of
the morning are quite high, and we need ipftp to reesablish the connection and

Try it with no offline queue : (where QD is a date like QD:20130126)

ipftp -A -i /fip/data/ebooks/\QD_kindle -h'WD:\QD' -Z -z goa_kindle -y 10 -Y

Version control
;18j81  29dec09 ;1-3 added TLS/SSL and fipstddir ;4 bugette for balance and
programname ;5 \%q is time taken
    ;6 08mar10 added tracker-script ;7-8 added ConnectionAttempts an
ConnectTimeout to ssh/sftp
    ;10-11 16apr10 only read pram file once for GETS
    ;12-17 7jun10 added error message in drain to describe what ! (15 added
linger-on-close) ; 16 better wilds with SSH ;17 TLS'C'
    ;18 25jul10 bugette with -o not adding /fip/spool
    ;19 10aug10 added rename-prefix and rename-newname for SFTP
    ;20 29oct10 bugette with false error msg for gets
    ;21-23 10nov10 GET-skips fudge for W2k FTPD bug ;22 \%c bytes RCV or TX
    ;23 added -A onePass and doneque is now FipSeq and fixed GENERIC_GET bugette
    ;24 added send-external-fiphdr:yes/no
    ;25-31 redid keepConnectionOpen so it did NOT logon each time /
    ;32-33 6dec11 woops remport NOT working for ssh and added remote-folder-exists
    ;34 12*425 is an error ;35 sshpass now works
    ;36 added list-folder-only plus added an error if Unknown keyword in parameter
    ;37 15feb12 added max-data-port and min-data-port (min is same as
    ;38 20feb12 woops TLS/SSL only worked 'onefile:'
    ;39 \$c and CTRL bugette
    ;40 6jul12 added hidden 'CD ~' if 2 sequential files have different Parameter
files but the same host/logon
    ;41 4sep12 added max-single-fiphdr-size
    ;42 21sep12 added fiphdr ZO original filename
    ;43-45 10oct12 added timeout alarm for sshpass plus added sshSftpVersionNumber
    ;46-47 17jan13 added send-multiples ;48 added getIsHash for Renames ; 49
cleanup ; 50 outque is now parseable
    ;51 added get-copy-file ; 52-53 tuned
fipdelete/fipduplicate/cd/fipallowfile/fipblockfile for spaces in filename
    ;54 5dec13 fixed max-files for GETS too ;55 added file_trace ;56-57 minor
logfile additions
    ;58 8aug14 passive-force-address added
    ;59-60 5sep14 added ssh-debug-logfile plus sshpass quotes to " not ' ** needs
fipssl 01d
    ;61 3oct14 passive-connections:(FipSeq)
    ;62 11dec14 added local-filter:yes ;63 CWD~ should be CDUP ;64 added
hash-in-fiphdr ! ;65-66 log msgs and newE2 for queue in logging
    ;67 24sep15 default linger 3 -> 20 ;68 back to CWD ~ not CDUP
    ;69-73 better check for IPaddress in place of hostname ; 74 added speedy for
GET files
    ;75-77 added fiphdrMD5 chksum on GET files
    ;78-81 20jun16 added check-compare-file timestamp
;018i35 12dec06 added skip-purge-after
    ;b 18dec06 PASV and 425
    ;c 09feb07 ssh wait cleanup
    ;d-g 23mar07 added -x default max files and added fipallowfile
    ;h 20sep07 small internal chg to link with StingRay FTP Server - no PORT/PASV
on logon
    ;i3 24oct07 added 'fipcd' and fipdelay can now have a noOfSecs after
    ;i4  1feb08 redunBalanced
    ;i5-10 19feb08 added send-external-file-name ;i6 max skip files upped
        ;10-11 27may08 bug in fipdelay (from i3) and some remote servers do NOT allow
        ;12-17 5jun08 added rename-prefix, rename-newname and BUG in PASV ; 17 for
        ;18 -justskip added
        ;19-21 added log-skip-details:(and stackoverflow doc)
        ;22-25 note_balance_action
        ;26-27 added -H for cluster host name and allow ZZ to be used in
        ;28-35 14sep09 check if proxy for failure and added minimum-poll-interval ;
30 HE+HR in GET FipHdr
;017z   24jul05 made H1-3, DR, DG, DT available to script, added -k and -y and
script for GET
    ;b-d 12aug05 cleaned up all those extra PORT/PASV on skipped files
    ;e 15sep05 keep-connection-open added for sending
    ;f 14oct05 STOU/uniquename need tuning
    ;g 21oct05 Reworked ZeroLen GET files and ignore - now we do NOT add to skip
notr zap/move etc
    ;h 23nov05 -W and -O mean that the offline queu is that specified and '2ftpX'
is not appended
    ;j-k 15dec05 added shadow skip file - possibly on another drive. and added -K
    ;l-o 09jan06 added add-md5-signature and cleanup of getfiles
    ;p 23jan06 added fixed-data-port
    ;q-r 23feb06 added put-empty-files (allow format: same as formatname)
    ;s-t 28jul06 added remote_trace style timings to send file done msg as H4
    ;u 10aug06 wait3 64 bit and added -X and timing-stats
    ;v-w 07sep06 added priority-sends plus bugette with fip-onerror
        plus for GET, must have a dest:, so added a default.
    ;x-y 04oct06 new DestRedun
    ;z 06dec06 added start-data-port
;016z   10aug04 add skips to old too
    ;b 24sep04 speedy
    ;c-h 27oct04 added ssf-secure shell stuff
        (d,h 19nov04 hash-in-filename added)
    ;i-j 31jan05 better Win2K Gets
    ;k-m 07feb05 'dests' can be FipSeq and added outque:
    ;n-p 23apr05 remote-done-que is now FipSeq
    ;r-v 13may05 redid old skip list for WINNT and added -B for skip-balance-group
    ;w-x 15jun05 added -Y there is no offline queue
    ;y-z 30jun05 only go PASV if no Data port is open
;015z   06jun02 better control over filename.....
    ;a/b 18jun02 WINNT get was broken
        plus AS400 gives a funny file ended message.
    ;c/d 28jul02 added ignore-all-preparation
    ;e 12sep02 added creating ResForks on the remote system
    ;f 15sep02 bugette for GET with many entries - may miss the last ones
    ;g/h 10oct02 bugette with zero length files plus added
        check-primary-server-for-getfiles: and balance-skip-files
    ;i 16oct02 to strip the './' at the begining of WarFTP files.
    ;j 23jan03 added inc/min/max Seqno
    ;k 30jun03 bugette - skip file not zapped when nothing there.
    ;l-m 12sep03 redid timeout on connect
    ;n 30sep03 added logon to checking host
    ;o 06oct03 quiet on DataPorts.....
    ;p 12dec03 added -h for extra-fiphdr on GETS
    ;q 06jan04 default formatname for Gets
    ;r 19feb04 small mod - comments only at start of line
    ;s 26feb04 better skiplist
    ;t-w 25mar04 added FTP_LIST_FILE: or -F and fipdir
    ;y 29jun04 added -O offline queue
    ;z 02jul04  bugette in PASV/passive sends
;014z   08sep00 added minimum-log
    ;a 29nov00 improved logeachmsg
    ;b 20dec00 bugette for the case of maxattempts exceeded and no errque
    ;c-d 10jan01 bugette in fiphdr:
    ;e-i 30jan01 cleanups
    ;j 27mar01 better exit codes for single-shot and errors
    ;k 10jun01 added FTP_EXTERNAL_FILE: and FTP_ZAP_EXTERNAL:
    ;l-m 12jul01 do NOT normally walk the remote folders for getfiles.
    ;n-o 25jul01 added no-fiphdr-on-getfiles
    ;p 09aug01 PASV/passive-connection tuned. and -V added
    ;r-u 17aug01 added 'except's
    ;v 01oct01 fipdelete mods for 'fipon-error'
    ;w 16jan02 added loglasterrfile/log-last-error-file
    ;x-y 14mar02 added remote-trace and zap DO on attempts from filename
    ;z 30apr02 added script plus FTP_FILE/BINARY_BEFORE/AFTER
;013b   15aug00 added skip-file plus possible buggette in GetFiles and lousy
    ;a 19aug00 bugette in get-extra-fiphdr
    ;b 05sep00 added log of skip files
;012e   03apr00 bugette in LOG file - could get confused with many files in
offline all withh different log files.
    ;a 26apr00 chkmsg seqno not right
    ;b 01may00 NT getfiles was buggy
        added fipon-error/on-error and maxattempts and slowdown
    ;c 17may00 added maxfiles:
    ;d 21jun00 better handling of the Done Queue
    ;e 09aug00 added 'zapresforks'
;011g   30nov99 added ftpproxy
    ;a 06dec99 mod to drain_it - occasionally missing a result code.
    ;b-c 10dec99 v occasionally, may miss change of seqno
    ;d-g 14jan00 allow space in NT filenames - redo seqno

(copyright) 2017 and previous years FingerPost Ltd.