sftpwire

sftpwire

    Fip version of Secure Shell SFTP

SFTPWIRE is a very simple SFTP daemon running (and started by) ssh.

Use 'ipftp' when you need to send data using FTP - or Grab data from a remote
FTPd site.
Use 'sftpwire' when a remote host needs to send to the Fip.

The file is slotted into the spool/2brouted queue for IPROUTE to process and
route.

Normally it is added as a SubSystem in /etc/ssh/sshd_config
    Subsystem fipsftp /fip/bin/sftpwire -V -n FTPWIRE_ELECTIONS

One useful builtin is merging FipHdrs - if the incoming file has a FipHdr, then
any 'extra-fiphdr' info, datetime FipHdrs, system FipHdrs are all merged.

-- Optional parameter file is under tables/wire and the '-n' input switch and
defaults to SFTPWIRE. A '.FIP' extension may be added.
NOTE This can be exactly the same as FTPWIRE - for each program, anything
totally specific is just ignored
    Syntax :
    ; comment line
    banner:             Replacement banner
    outque: (folder name in FipSeq) Output folder - this overrides the -O input
switch
    logon-file:(filename)       See below - default LOGON.(-n) or LOGON.FTPWIRE.FIP
    password-is-encoded: (yes/no)   Password in normal logon file or w4 lists is
encoded (default: no)

    use-w4-logon-list:(name eg FIPO.EXTERNAL)
                    Use this Fip w4 LogonList file for authentication and ignore all other
auth methods such as LOGON.FTPWIRE.FIP
    w4-logon-restrict-templates:(name(s) eg RADIUS_FTPUSER,RADIUS_FTPADMIN)
                    If using LogonList, only allow logons with this template(s)
    w4-logon-restrict-pubs:(pub name(s) eg PIF,BIG,DAILY)
                    If using LogonList, only allow logons with this pub code
    w4-logon-topq:
    w4-logon-outq:
    w4-logon-copyq:
    w4-logon-fiphdr:
    w4-logon-options:       if using LogonList, use these default values (see Logon
below for fuller explanation of each)

    use-etc-passwd: yes/no      (Linux/Unix only) - if YES, use the /etc/passwd file
for authentication and ignore all other auth methods such as LOGON.FTPWIRE.FIP
                    default: no
    external-path: (path)       for etc/passwd, Only allow logons with a home folder
starting with this path
                    eg  external-path:/home/ftp     expects all the home folders below /home/ftp
                    default: nothing specified for all folders

    extra-fiphdr: (fipseq)      Additional FipHdr info to be added to each incoming
file. default none
    extra-fiphdr-ext: (fipseq)  Additional FipHdr info to be added to each incoming
file if the file has this extension. default none
                    eg if a file is abc1.jpg extra-fiphdr-ext:jpg   SR:FTP_JPEGS
    replace-space: (fipseq)     Replace a space in the filename or folder with this
chr.        default: SPC
                    (see also allow-spaces:)
    replace-hash: (fipseq)      Replace hash chr in the filename or folder with this
chr.        default: '#'
    replace-unsafe: (fipseq)    Replace control and meta chrs in the filename or
folder with this.   default: '-'
                    Ftpwire will make a filename 'safe' for the system by cleaning meta chrs
'/'
                    FipHdr ZO contains the safe filename and SN is the 'given'
    no-archive:         do NOT archive the data in the daily archive files in log/data
(ZI: fiphdr field)
    display-log (yes/no/file/logon) (same as -D) Display all commands for each
connection      default: no
                    Use this to trace problem connections
                    Option 'file' will log the transactions in a dated file in log/ftp
                    Option 'logon' will log the transactions in a dated file for that logon in
log/ftp
    hourly-logs: (time) Remote Trace log files are normally daily
(/fip/log/remote_trace/(date)
            Use this to add a hour extension (only 60 for 60 mins is currently valid)
            (can be overriden by 'J' or '-J' in the LOGON file for each logon)
    force-folder-names:lower/upper/nochg    Force the case of any request for folder
names   default: nochange
    force-file-names:lower/upper/nochg  Force the case of any request for file
names   default: nochange
            Use these to handle Win2k <->Linux case issues - where a case-INsensitive
client is talking to a case-SENS servers
            One suggestion is to force all folders and files lowercase and set both of
these ..:lower

    balance-group; (Balance Group name) Balance group for balancing doneque items   
default: none / no balancing
        This group MUST be in sys/BALANCE
    balance-folder: (folder under spool) Balance queue for balancing doneque items  
default: 2balance
    move-on-read-group: (Balance Group name) Balance group for redundant
move-on-read    default: none / no balancing
            This needs a move-on-read-folder to be specified
    move-on-read-folder: (FipSeq name of a folder)  Name of folder for files once
read    default: none
    default-move-on-read: (yes/no)  Default for Move on Read                default: no

    linger-on-close: Timeout in secs for the data to be sent for Passive
connections
        For VERY slow connections, increase this already-enormous number,
        default: 20 (secs) to allow up to 20 secs max before cutting the call

    connection-retries: (number)    default 5
    send-timeout: (number)      default 60 secs
    recv-timeout: (number)      default 60 secs
    connection-timeout: (number)    default 20 secs
    session-timeout: (number)   default 1200 secs (20 mins) of no activity
    between-files-timeout: (number) default 1200 secs (20 mins) of no activity

??  allow-blank-pwd: (yes/no)   If there is a blank password in the logon file,
accept ANY password. default-no

    round-robin: (number)               default: none
    round-robin-fiphdr: (2 letter FipHdr field) default: none
    round-robin-offset-fiphdr: (2 letter FipHdr field)  default: none
        Round-Robin the output files and add the RR number to the fipHdr.
        Both parameters are required - the Number is the MAXimum.
        eg to leave the output in folder1 to folder9
            round-robin:9
            round-robin-fiphdr:RR
            round-robin-offset-fiphdr:RO
        and a suitable output folder might be   /fip/spool/xchg\RR
            (This can be in the LOGON file or the default input switch -O xchg\\R
(remember double backers)
        Note that the round-robin number is NOT added automatically to any output
folder - ie you MUST specify a FipHdr as in /fip/spool/2xml\RR
        The round-robin-offset-fiphdr allows the RR number to be offset by the -R
input switch which is the base offset - default 1
        So if '-R 8' and 'round-robin:10', the output will be in (folder)8 to
(folder)17
        So WITHOUT the -R switch both round-robin-fiphdr and
round-robin-offset-fiphdr will give the same number.

    allow: (IPaddress to allow)
    disallow: (IPaddress to block)
        use this for blacklist/whitelist certain addresses
        A '*' or '0' (star or zero) can be used to indicate a range eg 10.3.3.*
        An extra number with a preceeding space can be used to set a loglevel (see
below for values)
    disconnect-limit: (number)
        number of logon/password errors before connection is broken
        default: 30 unsuccessful attempts
    disable-limit: (number)
        number of logon/password errors before logon is blacklisted
        default: 30 unsuccessful attempts
    allow-site-fiphdr: (yes/no) see SITE FIPHDR below   default: no
    allow-ssh-fiphdr: (yes/no)              default: no

    timing-stats: (yes/no)      generate Timing stats (default is now YES)

    save-data-path: (pathname for data)
        This puts the data of the incoming data in a file in this folder and creates
a FipHdr file that contains 2 FipHdrs containing the full path/filename
            SX: and FTP_EXTERNAL_FILE:
        (ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE)
            eq  save-data-path:/fip/data/jpegs/\$e\$y\$i\$d/
        Use this for big files that you do not want to copy around the Fip Spool
area.
        ** if specified, ALL non-standalone files will be split like this **

    alert-email-address:  (one or more addresses separated by a comma)  default:
none
    alert-email-queue:  under spool                 default: 2smtp
    alert-email-extra:  (optional fipseq string to add to the FipHdr)   default:
none
    alert-email-top:    (optional fipseq string to add before any data) default: none
    alert-email-tail:   (optional fipseq string to add after any data)  default:
none
    alert-email-data:   (yes/no)                    default: no

    output-queue: (FipSeq)                  default: 2brouted or -o switch
    done-queue: (FipSeq)                    default: none or -d switch
    done-name-stub: (FipSeq) filename in done queue     default: safe name (ZO)

-- Logons and Passwords

There are 3 (main) types of Authentication.
    - default - using a Fip pipe delimited file called LOGON_FTPWIRE.FIP (or the
parameter of the '-n' input switch.
    - for Unix/Linux, using the normal /etc/passwd file
    - using the Fip w4 LogonList file

The following describes the parameter file syntax for the default. Please see
the relevant (external) doc for the others.

The types of logon/password are
    - anonymous logon
        use input switch -A to allow (disallowd by default)
        use parameters to optionally add more information
            anon-desc   - just a note for logging
            anon-fiphdr - extra fiphdr to add to each file
            anon-topq   - top folder for LIST and GETs
                        default is LISTs and GETS are not allowed
            anon-copyq  - folder holding a copy of any incoming file
            anon-curq   - under
    - full logon and password (normal running)
    - logon and allow any password - just leave the password blank and add -B

The Logon file is in tables/wir and is called LOGON.(name) where name is the
'-n' switch or FTPWIRE.FIP by default

fields in the Password file are pipe delimited and are :
field       0   Enabled or Disabled flag    E/D
        1   LogonName
        2   Password
        3   last mod time (used by the user interface only)
        4   Description/Comment
        5   optional home folder which is revealed as '/' to the remote
            if this is blank, then LS and GETs are returned as 'no such folder/file'
        6   Output folder for any incoming files. If it does NOT start with a '/', the
folder is under /fip/spool
            This can be in FipSeq
            If blank, the default output folder is used.
        7   optional Copy folder where an exact copy of the incoming file
                this can be the same as 5-home folder if the remote needs to see the file
        8   any optional ExtraFipHdr info
        9   Option single letter Flags
            (Negate by adding a dash/hyphen prefix - ie to make sure files are NEVER
deleted '-Z'
            S-standalone output file (original filename and no fiphdr)
            C-standalone copy file (original filename and no fiphdr)
            H-For Standalone copy, add a FipHdr
            D-display all commands for this client only (ie -D for this one client)
            F-allow extra FipHdr data to be added BEFORE the Store as a site command
                SITE FIPHDR #SU:ZIBBLE#CX:ZIBBLE2EDITO
            Z-allow delete of any Standalone Copy ('C') files
            T-allow sessionTimeout of 24 hours for this client (default is 20 mins or
the session-timeout parameter)
            U-allow Last File Timeout of 24 hours for this client (default is 20 mins or
the session-file-timeout parameter)
            M-allow client to MKDIR a sub folder
            R-allow client to RMDIR a sub folder
            G-allow client to GET a file (which is the default if 'S'tandalone or 'C'opy
is on - so to Disallow, use '-G')
            Q-on a CD/CWD, check the folder really does exist
            J-set hourly logs for the remote_trace log file
            W-if in standalone mode, overwrite files
            X-Send on Rename - Files are held in the 5-HomeFolder until a rename when
they are moved to the 6-Outputfolder
                2 further suboptions in () are optional for Pre strings and Post strings
                (pre=[string in FipSeq]) and (post=[FipSeq string]) where [FipSeq string]
is any parsable text !
                - if a source sends files preceeded by _^_(filename) then X (pre=_^_) will
detect these to be SendOnRename
                - if a source sends files with a '.tmp' extension and then renames them to
'.xml' or someother file type, use X (post=.tmp)
                NOTE that any files WITHOUT either the pre or post strings will be sent
immediately
                NOTE you cannot rename files if using V-virtual list
            V-Virtual list - use this to hold a list of files sent by the remote. LIST,
SIZE and MDTM commands will show the files
                Only files from the sender IN THIS SESSION are shown - none from previous
sessions or from any concurrent session.
                NOTE you cannot sendOnRename or rename files if using V-virtual list
            L-Logging options (sub options in following brackets)
                C - do NOT log connections/disc
                N - NewLogon
            A-Alert - send an alert email when a file arrives
                this option also requires an email address(es) in alert-email-address:...
eg

E|Pittlewire|zong|0|Mr Pittles Image
Agency||2edsys||#DF:PITTLEWIRE.FIP#EQ:pittle|
E|Brittle||0|Mr Brittle HardHat
Agency||xchg|#CX:B2FIP#DF:BRITTLE.FIP#EQ:brittle|

-- FipHdr fields added to each file UNLESS the Standalone option has been
flagged
    SP  IPaddress of the remote host
    SN  Filename given
    ZO  Safe filename
    SU  -n input switch or FTPWIRE
    SA  logon name
    SC  chrset - defaults to ASCII
    S1  client description for the logon file
    S2  wire id
    S3  session id
    S4  current PWD
    S5  any extra subfolder in a put command
    S6  current Client system details (if offered by remote)

Input Parameters :
Optional :
    -9 : do not use Speedy on a Speedy system
    -A : force the ApiKey to this               default: always need an apikey as 1st command
        This is for cases where the UNIX logon maps to a single Fip logon
    -d : done folder                    default: none
        This holds a copy of all incoming data files from every source
        The structure is
            (done folder) / (date)_(logon) eg 20110921_fip / (filename as written to the
output folder)
        It can be purged with an entry in maintenance (zapfiplog)
        eg if '-d raw.ftpwire' and we want the last 30 days data
            /fip/bin/ipdelque -q/fip/spool/raw.ftpwire -i1 -a30
    -E : maximum number of threads              default: 1
        up to a max of 200 (not Win2k)
        Note this is also a hardware limit in that small systems may not be able to
run more than 50 or so
    -F : allow SITE FIPHDR commands             default: no
    -I : wire id                        default: 0
        used to track which instance of a multi-ftpwire system a file arrived/logged
    -j : balance queue for balancing doneque items      default: 2balance
    -J : balance group for balancing doneque items      default: -none- no balancing
        This group MUST be in sys/BALANCE
    -l : log level
    -n : name of this wire                  default: FTPWIRE
    -O : Name of the output folder if not default       default: spool/2brouted
        This folder will be under /fip/spool
    -P : Port for control                   default: 9130
        -P 21 is normal
    -R : round-robin base - see above           default: 1
    -s : local hostname or ipaddress            default: all local addresses
        where a server has multiple ip address/hostnames, use '-s' to restrict
connections to a single address
    -S : default is standlone and not Fip           default: next folder is fip
        in this case do NOT add a Fiphdr and preserve exactly the incoming filename
        this can be overridden in the Logon file
    -T : log timing stats                   default: no
    -v : Print the version number and exit

-- Log levels for -l input switch are :
    default (-1)    errors only are logged
    0       connections/disconnections
    10      logons
    20      each file in or out

-- The -D input switch will display all cmds etc as they come in

eg

-- Secure FTP --------------------------------------

Confusingly there are two - completely different - 'Secure' FTPs plus a defunct
company called SecoueFTP etc etc
    1. a more secure version of ordinary FTP which uses SSL/TLS in the same way
http and httpS work for secure web sites.
    2. a file copy layer which sits ontop of SSH - which has nothing at all to do
with ordinary FTP

The extra confusion is that SSH uses SSL - so dont mix the two up !

((This is from the FileZilla website as they are the kings of ftp :

    .. TLS (FTPS) vs SSH (SFTP)
    FTPS (FTP encrypted with TLS) should not be confused with SFTP (SSH). The
latter is a completely different protocol.

    .. Explicit vs Implicit FTPS

    FTPS (FTP over TLS) is served up in two incompatible modes.
    If using explicit FTPS, the client connects to the normal FTP port and
explicitly switches into secure (TLS) mode with "AUTH TLS", whereas implicit
FTPS is an older style service that assumes TLS mode right from the start of
the connection (and normally listens on TCP port 990, rather than 21).
    In a FileZilla client this means prefixing the host with "FTPES://" to connect
an "explicit" FTPS server, or "FTPS://" for the legacy "implicit" server (for
which you will likely also need to set the port to 990).

Thank you FileZilla))

How do you know which is the one you want ?

- What is the port number on the remote server ?
    port 21  - it is BOTH normal FTP and the SSL/TLS version (port 21 is the same
as normal FTP)
                test with ordinary 'ftp' client
    port 990    - it is ONLY SSL/TLS version
                test with 'telnet' to (remhost) 990 and cut the connection once you are
satisfied it connects
    port 22  - it is ONLY the sftp on top of SSH
                test with 'ssh' or 'sftp'

--- 1. SSL/TLS
    YOU MUST USE ftpwiressl for any/all ssl/tls traffic as ftpwire blocks these
commands.

--- 2.a SSH/SFTP

    Ok - use sftpwire

.................
-- SFTPWIRE Installation

1. create a new single logon - example is 'fipdata'

As root , add the logon (make sure it has a shell and is NOT nologin
    useradd -m fipdata
    passwd fipdata
        (add a passwd)
- check it works
    su - fipdata

.................
2. as fip - make sure sftpwire is in /fip/bin

.................
3. as fip - check the name of the FTPWIRE parameter file in wire
    this wil be the '-n' parameter
    the following uses FTPWIRE_COMMON as an example

.................
4. as fip - check at least one logon in the logon file pointed to by
wire/FTPWIRE_COMMON
    and that the 4th field is a unique apikey (no spaces or | or utf8 chrs)
        E|(logon)|(pwd for ftpwire)|(apikey for sftpwire)|...

.................
5. as sudo/root - modify /etc/ssh/sshd_config

Here are the differences
DO THIS VERY CAREFULLY as you might lock existing users out !
Make a copy first !

root@violet:/etc/ssh# cp -p sshd_config sshd_config.asatDATE

5.a if you are restricting users with 'AllowUsers', add fipdata
    AllowUsers (previous entries...) fipdata

5.b if you want users to logon as well as shared keys,
# comment OUT 'PasswordAuthentication'
#11may17 PasswordAuthentication no
PermitEmptyPasswords no

5.c add this block at the bottom of the file - all 'Match'es must be the last
part
(NOTE use spaces, not tabs, for the indent :

# ------------------------------------
# these are the last few lines of the sshd_config file
# 11may17
Match User fipdata
    ForceCommand /fip/bin/sftpwire -n FTPWIRE_COMMON
    AllowTCPForwarding no
    X11Forwarding no
# ------------------------------------

or ...

# ------------------------------------
# ONLY allow fipsec : sftp on 9123

Match User fipsec LocalPort 22
    DenyUsers fipsec

Match User fipsec LocalPort 9122
    DenyUsers fipsec

# -A apikey (not Logon)
Match User fipsec LocalPort 9123
    ForceCommand /fip/bin/sftpwire -n FTPWIRE_SFTP -A PDS+JNMND88aj
    AllowTCPForwarding no
    X11Forwarding no

# ------------------------------------

5.y as sudo/root - restart sshd
systemctl restart sshd

5.z test the new settings have NOT mashed the existing settings
use ssh to connect and check you can STILL logon on to the server

6. File/Folder Permissions - make sure fip can read/write/delete the new files
created by the new logon

6.a groups - as sudo/root
    vi /etc/group
    find the 'fip' group
    add both fip and the new logon to the end of the line :
fip:x:1000:fip,fipdata

6.b chown/chmod
chown -R fip:fip /fip
chown -R fip:fip /fip/(star)
chmod -R 770 /fip/(star)
or, depending pn your preference
chmod -R 775 /fip/(star)

.................
7. check your firewall settings to make sure the remote can login !

eg for IPtables if you want to restrict to IP address range
(check range online at aomewhere like
https://mxtoolbox.com/subnetcalculator.aspx
-A INPUT -p tcp -m tcp -m state --state NEW --dport 22 -s
195.158.151.0/26,58.58.58.43/32 -j ACCEPT

.................
99. test !

...........................................................................
Version Control
;1a-f   28apr20 minor ;f added forceApiKey
;0a-v   12sep16 chj original version

(copyright) 2020 and previous years FingerPost Ltd.