sftpwire
sftpwire
Fip version of Secure Shell SFTP
SFTPWIRE is a very simple SFTP daemon running (and started by) ssh.
Use 'ipftp' when you need to send data using FTP - or Grab data from a remote
FTPd site.
Use 'sftpwire' when a remote host needs to send to the Fip.
The file is slotted into the spool/2brouted queue for IPROUTE to process and
route.
Normally it is added as a SubSystem in /etc/ssh/sshd_config
Subsystem fipsftp /fip/bin/sftpwire -n FTPWIRE_ELECTIONS
One useful builtin is merging FipHdrs - if the incoming file has a FipHdr, then
any 'extra-fiphdr' info, datetime FipHdrs, system FipHdrs are all merged.
-- Optional parameter file is under tables/wire and the '-n' input switch and
defaults to SFTPWIRE. A '.FIP' extension may be added.
NOTE This can be exactly the same as FTPWIRE - for each program, anything
totally specific is just ignored
Syntax :
; comment line
banner: Replacement banner
outque: (folder name in FipSeq) Output folder - this overrides the -O input
switch
logon-file:(filename) See below - default LOGON.(-n) or LOGON.FTPWIRE.FIP
password-is-encoded: (yes/no) Password in normal logon file or w4 lists is
encoded (default: no)
use-w4-logon-list:(name eg FIPO.EXTERNAL)
Use this Fip w4 LogonList file for authentication and ignore all other
auth methods such as LOGON.FTPWIRE.FIP
w4-logon-restrict-templates:(name(s) eg RADIUS_FTPUSER,RADIUS_FTPADMIN)
If using LogonList, only allow logons with this template(s)
w4-logon-restrict-pubs:(pub name(s) eg PIF,BIG,DAILY)
If using LogonList, only allow logons with this pub code
w4-logon-topq:
w4-logon-outq:
w4-logon-copyq:
w4-logon-fiphdr:
w4-logon-options: if using LogonList, use these default values (see Logon
below for fuller explanation of each)
use-etc-passwd: yes/no (Linux/Unix only) - if YES, use the /etc/passwd file
for authentication and ignore all other auth methods such as LOGON.FTPWIRE.FIP
default: no
external-path: (path) for etc/passwd, Only allow logons with a home folder
starting with this path
eg external-path:/home/ftp expects all the home folders below /home/ftp
default: nothing specified for all folders
extra-fiphdr: (fipseq) Additional FipHdr info to be added to each incoming
file. default none
extra-fiphdr-ext: (fipseq) Additional FipHdr info to be added to each incoming
file if the file has this extension. default none
eg if a file is abc1.jpg extra-fiphdr-ext:jpg SR:FTP_JPEGS
replace-space: (fipseq) Replace a space in the filename or folder with this
chr. default: SPC
(see also allow-spaces:)
replace-hash: (fipseq) Replace hash chr in the filename or folder with this
chr. default: '#'
replace-unsafe: (fipseq) Replace control and meta chrs in the filename or
folder with this. default: '-'
Ftpwire will make a filename 'safe' for the system by cleaning meta chrs
'/'
FipHdr ZO contains the safe filename and SN is the 'given'
no-archive: do NOT archive the data in the daily archive files in log/data
(ZI: fiphdr field)
display-log (yes/no/file/logon) (same as -D) Display all commands for each
connection default: no
Use this to trace problem connections
Option 'file' will log the transactions in a dated file in log/ftp
Option 'logon' will log the transactions in a dated file for that logon in
log/ftp
hourly-logs: (time) Remote Trace log files are normally daily
(/fip/log/remote_trace/(date)
Use this to add a hour extension (only 60 for 60 mins is currently valid)
(can be overriden by 'J' or '-J' in the LOGON file for each logon)
force-folder-names:lower/upper/nochg Force the case of any request for folder
names default: nochange
force-file-names:lower/upper/nochg Force the case of any request for file
names default: nochange
Use these to handle Win2k <->Linux case issues - where a case-INsensitive
client is talking to a case-SENS servers
One suggestion is to force all folders and files lowercase and set both of
these ..:lower
balance-group; (Balance Group name) Balance group for balancing doneque items
default: none / no balancing
This group MUST be in sys/BALANCE
balance-folder: (folder under spool) Balance queue for balancing doneque items
default: 2balance
move-on-read-group: (Balance Group name) Balance group for redundant
move-on-read default: none / no balancing
This needs a move-on-read-folder to be specified
move-on-read-folder: (FipSeq name of a folder) Name of folder for files once
read default: none
default-move-on-read: (yes/no) Default for Move on Read default: no
linger-on-close: Timeout in secs for the data to be sent for Passive
connections
For VERY slow connections, increase this already-enormous number,
default: 20 (secs) to allow up to 20 secs max before cutting the call
connection-retries: (number) default 5
send-timeout: (number) default 60 secs
recv-timeout: (number) default 60 secs
connection-timeout: (number) default 20 secs
session-timeout: (number) default 1200 secs (20 mins) of no activity
between-files-timeout: (number) default 1200 secs (20 mins) of no activity
?? allow-blank-pwd: (yes/no) If there is a blank password in the logon file,
accept ANY password. default-no
round-robin: (number) default: none
round-robin-fiphdr: (2 letter FipHdr field) default: none
round-robin-offset-fiphdr: (2 letter FipHdr field) default: none
Round-Robin the output files and add the RR number to the fipHdr.
Both parameters are required - the Number is the MAXimum.
eg to leave the output in folder1 to folder9
round-robin:9
round-robin-fiphdr:RR
round-robin-offset-fiphdr:RO
and a suitable output folder might be /fip/spool/xchg\RR
(This can be in the LOGON file or the default input switch -O xchg\\R
(remember double backers)
Note that the round-robin number is NOT added automatically to any output
folder - ie you MUST specify a FipHdr as in /fip/spool/2xml\RR
The round-robin-offset-fiphdr allows the RR number to be offset by the -R
input switch which is the base offset - default 1
So if '-R 8' and 'round-robin:10', the output will be in (folder)8 to
(folder)17
So WITHOUT the -R switch both round-robin-fiphdr and
round-robin-offset-fiphdr will give the same number.
allow: (IPaddress to allow)
disallow: (IPaddress to block)
use this for blacklist/whitelist certain addresses
A '*' or '0' (star or zero) can be used to indicate a range eg 10.3.3.*
An extra number with a preceeding space can be used to set a loglevel (see
below for values)
disconnect-limit: (number)
number of logon/password errors before connection is broken
default: 30 unsuccessful attempts
disable-limit: (number)
number of logon/password errors before logon is blacklisted
default: 30 unsuccessful attempts
allow-site-fiphdr: (yes/no) see SITE FIPHDR below default: no
allow-ssh-fiphdr: (yes/no) default: no
timing-stats: (yes/no) generate Timing stats (default is now YES)
save-data-path: (pathname for data)
This puts the data of the incoming data in a file in this folder and creates
a FipHdr file that contains 2 FipHdrs containing the full path/filename
SX: and FTP_EXTERNAL_FILE:
(ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE)
eq save-data-path:/fip/data/jpegs/\$e\$y\$i\$d/
Use this for big files that you do not want to copy around the Fip Spool
area.
** if specified, ALL non-standalone files will be split like this **
alert-email-address: (one or more addresses separated by a comma) default:
none
alert-email-queue: under spool default: 2smtp
alert-email-extra: (optional fipseq string to add to the FipHdr) default:
none
alert-email-top: (optional fipseq string to add before any data) default: none
alert-email-tail: (optional fipseq string to add after any data) default:
none
alert-email-data: (yes/no) default: no
output-queue: (FipSeq) default: 2brouted or -o switch
done-queue: (FipSeq) default: none or -d switch
done-name-stub: (FipSeq) filename in done queue default: safe name (ZO)
-- Logons and Passwords
There are 3 (main) types of Authentication.
- default - using a Fip pipe delimited file called LOGON_FTPWIRE.FIP (or the
parameter of the '-n' input switch.
- for Unix/Linux, using the normal /etc/passwd file
- using the Fip w4 LogonList file
The following describes the parameter file syntax for the default. Please see
the relevant (external) doc for the others.
The types of logon/password are
- anonymous logon
use input switch -A to allow (disallowd by default)
use parameters to optionally add more information
anon-desc - just a note for logging
anon-fiphdr - extra fiphdr to add to each file
anon-topq - top folder for LIST and GETs
default is LISTs and GETS are not allowed
anon-copyq - folder holding a copy of any incoming file
anon-curq - under
- full logon and password (normal running)
- logon and allow any password - just leave the password blank and add -B
The Logon file is in tables/wir and is called LOGON.(name) where name is the
'-n' switch or FTPWIRE.FIP by default
fields in the Password file are pipe delimited and are :
field 0 Enabled or Disabled flag E/D
1 LogonName
2 Password
3 last mod time (used by the user interface only)
4 Description/Comment
5 optional home folder which is revealed as '/' to the remote
if this is blank, then LS and GETs are returned as 'no such folder/file'
6 Output folder for any incoming files. If it does NOT start with a '/', the
folder is under /fip/spool
This can be in FipSeq
If blank, the default output folder is used.
7 optional Copy folder where an exact copy of the incoming file
this can be the same as 5-home folder if the remote needs to see the file
8 any optional ExtraFipHdr info
9 Option single letter Flags
(Negate by adding a dash/hyphen prefix - ie to make sure files are NEVER
deleted '-Z'
S-standalone output file (original filename and no fiphdr)
C-standalone copy file (original filename and no fiphdr)
H-For Standalone copy, add a FipHdr
D-display all commands for this client only (ie -D for this one client)
F-allow extra FipHdr data to be added BEFORE the Store as a site command
SITE FIPHDR #SU:ZIBBLE#CX:ZIBBLE2EDITO
Z-allow delete of any Standalone Copy ('C') files
T-allow sessionTimeout of 24 hours for this client (default is 20 mins or
the session-timeout parameter)
U-allow Last File Timeout of 24 hours for this client (default is 20 mins or
the session-file-timeout parameter)
M-allow client to MKDIR a sub folder
R-allow client to RMDIR a sub folder
G-allow client to GET a file (which is the default if 'S'tandalone or 'C'opy
is on - so to Disallow, use '-G')
Q-on a CD/CWD, check the folder really does exist
J-set hourly logs for the remote_trace log file
W-if in standalone mode, overwrite files
X-Send on Rename - Files are held in the 5-HomeFolder until a rename when
they are moved to the 6-Outputfolder
2 further suboptions in () are optional for Pre strings and Post strings
(pre=[string in FipSeq]) and (post=[FipSeq string]) where [FipSeq string]
is any parsable text !
- if a source sends files preceeded by _^_(filename) then X (pre=_^_) will
detect these to be SendOnRename
- if a source sends files with a '.tmp' extension and then renames them to
'.xml' or someother file type, use X (post=.tmp)
NOTE that any files WITHOUT either the pre or post strings will be sent
immediately
NOTE you cannot rename files if using V-virtual list
V-Virtual list - use this to hold a list of files sent by the remote. LIST,
SIZE and MDTM commands will show the files
Only files from the sender IN THIS SESSION are shown - none from previous
sessions or from any concurrent session.
NOTE you cannot sendOnRename or rename files if using V-virtual list
L-Logging options (sub options in following brackets)
C - do NOT log connections/disc
N - NewLogon
A-Alert - send an alert email when a file arrives
this option also requires an email address(es) in alert-email-address:...
eg
E|Pittlewire|zong|0|Mr Pittles Image
Agency||2edsys||#DF:PITTLEWIRE.FIP#EQ:pittle|
E|Brittle||0|Mr Brittle HardHat
Agency||xchg|#CX:B2FIP#DF:BRITTLE.FIP#EQ:brittle|
-- FipHdr fields added to each file UNLESS the Standalone option has been
flagged
SP IPaddress of the remote host
SN Filename given
ZO Safe filename
SU -n input switch or FTPWIRE
SA logon name
SC chrset - defaults to ASCII
S1 client description for the logon file
S2 wire id
S3 session id
S4 current PWD
S5 any extra subfolder in a put command
S6 current Client system details (if offered by remote)
Input Parameters :
Optional :
-9 : do not use Speedy on a Speedy system
-A : force the ApiKey to this default: always need an apikey as 1st command
This is for cases where the UNIX logon maps to a single Fip logon
-d : done folder default: none
This holds a copy of all incoming data files from every source
The structure is
(done folder) / (date)_(logon) eg 20110921_fip / (filename as written to the
output folder)
It can be purged with an entry in maintenance (zapfiplog)
eg if '-d raw.ftpwire' and we want the last 30 days data
/fip/bin/ipdelque -q/fip/spool/raw.ftpwire -i1 -a30
-E : maximum number of threads default: 1
up to a max of 200 (not Win2k)
Note this is also a hardware limit in that small systems may not be able to
run more than 50 or so
-F : allow SITE FIPHDR commands default: no
-I : wire id default: 0
used to track which instance of a multi-ftpwire system a file arrived/logged
-j : balance queue for balancing doneque items default: 2balance
-J : balance group for balancing doneque items default: -none- no balancing
This group MUST be in sys/BALANCE
-l : log level
-n : name of this wire default: FTPWIRE
-O : Name of the output folder if not default default: spool/2brouted
This folder will be under /fip/spool
-P : Port for control default: 9130
-P 21 is normal
-R : round-robin base - see above default: 1
-s : local hostname or ipaddress default: all local addresses
where a server has multiple ip address/hostnames, use '-s' to restrict
connections to a single address
-S : default is standlone and not Fip default: next folder is fip
in this case do NOT add a Fiphdr and preserve exactly the incoming filename
this can be overridden in the Logon file
-T : log timing stats default: no
-V : Debug ssh default: no
-v : Print the version number and exit
-- Log levels for -l input switch are :
default (-1) errors only are logged
0 connections/disconnections
10 logons
20 each file in or out
-- The -D input switch will display all cmds etc as they come in
eg
-- Secure FTP --------------------------------------
Confusingly there are two - completely different - 'Secure' FTPs plus a defunct
company called SecoueFTP etc etc
1. a more secure version of ordinary FTP which uses SSL/TLS in the same way
http and httpS work for secure web sites.
2. a file copy layer which sits ontop of SSH - which has nothing at all to do
with ordinary FTP
The extra confusion is that SSH uses SSL - so dont mix the two up !
((This is from the FileZilla website as they are the kings of ftp :
.. TLS (FTPS) vs SSH (SFTP)
FTPS (FTP encrypted with TLS) should not be confused with SFTP (SSH). The
latter is a completely different protocol.
.. Explicit vs Implicit FTPS
FTPS (FTP over TLS) is served up in two incompatible modes.
If using explicit FTPS, the client connects to the normal FTP port and
explicitly switches into secure (TLS) mode with "AUTH TLS", whereas implicit
FTPS is an older style service that assumes TLS mode right from the start of
the connection (and normally listens on TCP port 990, rather than 21).
In a FileZilla client this means prefixing the host with "FTPES://" to connect
an "explicit" FTPS server, or "FTPS://" for the legacy "implicit" server (for
which you will likely also need to set the port to 990).
Thank you FileZilla))
How do you know which is the one you want ?
- What is the port number on the remote server ?
port 21 - it is BOTH normal FTP and the SSL/TLS version (port 21 is the same
as normal FTP)
test with ordinary 'ftp' client
port 990 - it is ONLY SSL/TLS version
test with 'telnet' to (remhost) 990 and cut the connection once you are
satisfied it connects
port 22 - it is ONLY the sftp on top of SSH
test with 'ssh' or 'sftp'
--- 1. SSL/TLS
YOU MUST USE ftpwiressl for any/all ssl/tls traffic as ftpwire blocks these
commands.
--- 2.a SSH/SFTP
Ok - use sftpwire
.................
-- SFTPWIRE Installation
1. create a new single logon - example is 'fipdata'
As root , add the logon (make sure it has a shell and is NOT nologin
useradd -m fipdata
passwd fipdata
(add a passwd)
- check it works
su - fipdata
.................
2. as fip - make sure sftpwire is in /fip/bin
.................
3. as fip - check the name of the FTPWIRE parameter file in wire
this wil be the '-n' parameter
the following uses FTPWIRE_COMMON as an example
.................
4. as fip - check at least one logon in the logon file pointed to by
wire/FTPWIRE_COMMON
and that the 4th field is a unique apikey (no spaces or | or utf8 chrs)
E|(logon)|(pwd for ftpwire)|(apikey for sftpwire)|...
.................
5. as sudo/root - modify /etc/ssh/sshd_config
Here are the differences
DO THIS VERY CAREFULLY as you might lock existing users out !
Make a copy first !
root@violet:/etc/ssh# cp -p sshd_config sshd_config.asatDATE
5.a if you are restricting users with 'AllowUsers', add fipdata
AllowUsers (previous entries...) fipdata
5.b if you want users to logon as well as shared keys,
# comment OUT 'PasswordAuthentication'
#11may17 PasswordAuthentication no
PermitEmptyPasswords no
5.c add this block at the bottom of the file - all 'Match'es must be the last
part
(NOTE use spaces, not tabs, for the indent :
# ------------------------------------
# these are the last few lines of the sshd_config file
# 11may17
Match User fipdata
ForceCommand /fip/bin/sftpwire -n FTPWIRE_COMMON
AllowTCPForwarding no
X11Forwarding no
# ------------------------------------
or ...
# ------------------------------------
# ONLY allow fipsec : sftp on 9123
Match User fipsec LocalPort 22
DenyUsers fipsec
Match User fipsec LocalPort 9122
DenyUsers fipsec
# -A apikey (not Logon)
Match User fipsec LocalPort 9123
ForceCommand /fip/bin/sftpwire -n FTPWIRE_SFTP -A PDS+JNMND88aj
AllowTCPForwarding no
X11Forwarding no
# ------------------------------------
5.y as sudo/root - restart sshd
systemctl restart sshd
5.z test the new settings have NOT mashed the existing settings
use ssh to connect and check you can STILL logon on to the server
6. File/Folder Permissions - make sure fip can read/write/delete the new files
created by the new logon
6.a groups - as sudo/root
vi /etc/group
find the 'fip' group
add both fip and the new logon to the end of the line :
fip:x:1000:fip,fipdata
6.b chown/chmod
chown -R fip:fip /fip
chown -R fip:fip /fip/(star)
chmod -R 770 /fip/(star)
or, depending pn your preference
chmod -R 775 /fip/(star)
.................
7. check your firewall settings to make sure the remote can login !
eg for IPtables if you want to restrict to IP address range
(check range online at aomewhere like
https://mxtoolbox.com/subnetcalculator.aspx
-A INPUT -p tcp -m tcp -m state --state NEW --dport 22 -s
195.158.151.0/26,58.58.58.43/32 -j ACCEPT
.................
99. test !
...........................................................................
Version Control
; 1a-i5 28apr20 minor ;f added forceApiKey ;g WINNT ;h roundrobin not working
correctly ;i 7oct22 bad encrypt plus thsCmdLoggedOn
; 0a-v 12sep16 chj original version
(copyright) 2024 and previous years FingerPost Ltd.