sftpwire Fip version of Secure Shell SFTP SFTPWIRE is a very simple SFTP daemon running (and started by) ssh. Use 'ipftp' when you need to send data using FTP - or Grab data from a remote FTPd site. Use 'sftpwire' when a remote host needs to send to the Fip. The file is slotted into the spool/2brouted queue for IPROUTE to process and route. Normally it is added as a SubSystem in /etc/ssh/sshd_config Subsystem fipsftp /fip/bin/sftpwire -V -n FTPWIRE_ELECTIONS One useful builtin is merging FipHdrs - if the incoming file has a FipHdr, then any 'extra-fiphdr' info, datetime FipHdrs, system FipHdrs are all merged. -- Optional parameter file is under tables/wire and the '-n' input switch and defaults to SFTPWIRE. A '.FIP' extension may be added. NOTE This can be exactly the same as FTPWIRE - for each program, anything totally specific is just ignored Syntax : ; comment line banner: Replacement banner outque: (folder name in FipSeq) Output folder - this overrides the -O input switch logon-file:(filename) See below - default LOGON.(-n) or LOGON.FTPWIRE.FIP password-is-encoded: (yes/no) Password in normal logon file or w4 lists is encoded (default: no) use-w4-logon-list:(name eg FIPO.EXTERNAL) Use this Fip w4 LogonList file for authentication and ignore all other auth methods such as LOGON.FTPWIRE.FIP w4-logon-restrict-templates:(name(s) eg RADIUS_FTPUSER,RADIUS_FTPADMIN) If using LogonList, only allow logons with this template(s) w4-logon-restrict-pubs:(pub name(s) eg PIF,BIG,DAILY) If using LogonList, only allow logons with this pub code w4-logon-topq: w4-logon-outq: w4-logon-copyq: w4-logon-fiphdr: w4-logon-options: if using LogonList, use these default values (see Logon below for fuller explanation of each) use-etc-passwd: yes/no (Linux/Unix only) - if YES, use the /etc/passwd file for authentication and ignore all other auth methods such as LOGON.FTPWIRE.FIP default: no external-path: (path) for etc/passwd, Only allow logons with a home folder starting with this path eg external-path:/home/ftp expects all the home folders below /home/ftp default: nothing specified for all folders extra-fiphdr: (fipseq) Additional FipHdr info to be added to each incoming file. default none extra-fiphdr-ext: (fipseq) Additional FipHdr info to be added to each incoming file if the file has this extension. default none eg if a file is abc1.jpg extra-fiphdr-ext:jpg SR:FTP_JPEGS replace-space: (fipseq) Replace a space in the filename or folder with this chr. default: SPC (see also allow-spaces:) replace-hash: (fipseq) Replace hash chr in the filename or folder with this chr. default: '#' replace-unsafe: (fipseq) Replace control and meta chrs in the filename or folder with this. default: '-' Ftpwire will make a filename 'safe' for the system by cleaning meta chrs '/' FipHdr ZO contains the safe filename and SN is the 'given' no-archive: do NOT archive the data in the daily archive files in log/data (ZI: fiphdr field) display-log (yes/no/file/logon) (same as -D) Display all commands for each connection default: no Use this to trace problem connections Option 'file' will log the transactions in a dated file in log/ftp Option 'logon' will log the transactions in a dated file for that logon in log/ftp hourly-logs: (time) Remote Trace log files are normally daily (/fip/log/remote_trace/(date) Use this to add a hour extension (only 60 for 60 mins is currently valid) (can be overriden by 'J' or '-J' in the LOGON file for each logon) force-folder-names:lower/upper/nochg Force the case of any request for folder names default: nochange force-file-names:lower/upper/nochg Force the case of any request for file names default: nochange Use these to handle Win2k <->Linux case issues - where a case-INsensitive client is talking to a case-SENS servers One suggestion is to force all folders and files lowercase and set both of these ..:lower balance-group; (Balance Group name) Balance group for balancing doneque items default: none / no balancing This group MUST be in sys/BALANCE balance-folder: (folder under spool) Balance queue for balancing doneque items default: 2balance move-on-read-group: (Balance Group name) Balance group for redundant move-on-read default: none / no balancing This needs a move-on-read-folder to be specified move-on-read-folder: (FipSeq name of a folder) Name of folder for files once read default: none default-move-on-read: (yes/no) Default for Move on Read default: no linger-on-close: Timeout in secs for the data to be sent for Passive connections For VERY slow connections, increase this already-enormous number, default: 20 (secs) to allow up to 20 secs max before cutting the call connection-retries: (number) default 5 send-timeout: (number) default 60 secs recv-timeout: (number) default 60 secs connection-timeout: (number) default 20 secs session-timeout: (number) default 1200 secs (20 mins) of no activity between-files-timeout: (number) default 1200 secs (20 mins) of no activity ?? allow-blank-pwd: (yes/no) If there is a blank password in the logon file, accept ANY password. default-no round-robin: (number) default: none round-robin-fiphdr: (2 letter FipHdr field) default: none round-robin-offset-fiphdr: (2 letter FipHdr field) default: none Round-Robin the output files and add the RR number to the fipHdr. Both parameters are required - the Number is the MAXimum. eg to leave the output in folder1 to folder9 round-robin:9 round-robin-fiphdr:RR round-robin-offset-fiphdr:RO and a suitable output folder might be /fip/spool/xchg\RR (This can be in the LOGON file or the default input switch -O xchg\\R (remember double backers) Note that the round-robin number is NOT added automatically to any output folder - ie you MUST specify a FipHdr as in /fip/spool/2xml\RR The round-robin-offset-fiphdr allows the RR number to be offset by the -R input switch which is the base offset - default 1 So if '-R 8' and 'round-robin:10', the output will be in (folder)8 to (folder)17 So WITHOUT the -R switch both round-robin-fiphdr and round-robin-offset-fiphdr will give the same number. allow: (IPaddress to allow) disallow: (IPaddress to block) use this for blacklist/whitelist certain addresses A '*' or '0' (star or zero) can be used to indicate a range eg 10.3.3.* An extra number with a preceeding space can be used to set a loglevel (see below for values) disconnect-limit: (number) number of logon/password errors before connection is broken default: 30 unsuccessful attempts disable-limit: (number) number of logon/password errors before logon is blacklisted default: 30 unsuccessful attempts allow-site-fiphdr: (yes/no) see SITE FIPHDR below default: no allow-ssh-fiphdr: (yes/no) default: no timing-stats: (yes/no) generate Timing stats (default is now YES) save-data-path: (pathname for data) This puts the data of the incoming data in a file in this folder and creates a FipHdr file that contains 2 FipHdrs containing the full path/filename SX: and FTP_EXTERNAL_FILE: (ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE) eq save-data-path:/fip/data/jpegs/\$e\$y\$i\$d/ Use this for big files that you do not want to copy around the Fip Spool area. ** if specified, ALL non-standalone files will be split like this ** alert-email-address: (one or more addresses separated by a comma) default: none alert-email-queue: under spool default: 2smtp alert-email-extra: (optional fipseq string to add to the FipHdr) default: none alert-email-top: (optional fipseq string to add before any data) default: none alert-email-tail: (optional fipseq string to add after any data) default: none alert-email-data: (yes/no) default: no output-queue: (FipSeq) default: 2brouted or -o switch done-queue: (FipSeq) default: none or -d switch done-name-stub: (FipSeq) filename in done queue default: safe name (ZO) -- Logons and Passwords There are 3 (main) types of Authentication. - default - using a Fip pipe delimited file called LOGON_FTPWIRE.FIP (or the parameter of the '-n' input switch. - for Unix/Linux, using the normal /etc/passwd file - using the Fip w4 LogonList file The following describes the parameter file syntax for the default. Please see the relevant (external) doc for the others. The types of logon/password are - anonymous logon use input switch -A to allow (disallowd by default) use parameters to optionally add more information anon-desc - just a note for logging anon-fiphdr - extra fiphdr to add to each file anon-topq - top folder for LIST and GETs default is LISTs and GETS are not allowed anon-copyq - folder holding a copy of any incoming file anon-curq - under - full logon and password (normal running) - logon and allow any password - just leave the password blank and add -B The Logon file is in tables/wir and is called LOGON.(name) where name is the '-n' switch or FTPWIRE.FIP by default fields in the Password file are pipe delimited and are : field 0 Enabled or Disabled flag E/D 1 LogonName 2 Password 3 last mod time (used by the user interface only) 4 Description/Comment 5 optional home folder which is revealed as '/' to the remote if this is blank, then LS and GETs are returned as 'no such folder/file' 6 Output folder for any incoming files. If it does NOT start with a '/', the folder is under /fip/spool This can be in FipSeq If blank, the default output folder is used. 7 optional Copy folder where an exact copy of the incoming file this can be the same as 5-home folder if the remote needs to see the file 8 any optional ExtraFipHdr info 9 Option single letter Flags (Negate by adding a dash/hyphen prefix - ie to make sure files are NEVER deleted '-Z' S-standalone output file (original filename and no fiphdr) C-standalone copy file (original filename and no fiphdr) H-For Standalone copy, add a FipHdr D-display all commands for this client only (ie -D for this one client) F-allow extra FipHdr data to be added BEFORE the Store as a site command SITE FIPHDR #SU:ZIBBLE#CX:ZIBBLE2EDITO Z-allow delete of any Standalone Copy ('C') files T-allow sessionTimeout of 24 hours for this client (default is 20 mins or the session-timeout parameter) U-allow Last File Timeout of 24 hours for this client (default is 20 mins or the session-file-timeout parameter) M-allow client to MKDIR a sub folder R-allow client to RMDIR a sub folder G-allow client to GET a file (which is the default if 'S'tandalone or 'C'opy is on - so to Disallow, use '-G') Q-on a CD/CWD, check the folder really does exist J-set hourly logs for the remote_trace log file W-if in standalone mode, overwrite files X-Send on Rename - Files are held in the 5-HomeFolder until a rename when they are moved to the 6-Outputfolder 2 further suboptions in () are optional for Pre strings and Post strings (pre=[string in FipSeq]) and (post=[FipSeq string]) where [FipSeq string] is any parsable text ! - if a source sends files preceeded by _^_(filename) then X (pre=_^_) will detect these to be SendOnRename - if a source sends files with a '.tmp' extension and then renames them to '.xml' or someother file type, use X (post=.tmp) NOTE that any files WITHOUT either the pre or post strings will be sent immediately NOTE you cannot rename files if using V-virtual list V-Virtual list - use this to hold a list of files sent by the remote. LIST, SIZE and MDTM commands will show the files Only files from the sender IN THIS SESSION are shown - none from previous sessions or from any concurrent session. NOTE you cannot sendOnRename or rename files if using V-virtual list L-Logging options (sub options in following brackets) C - do NOT log connections/disc N - NewLogon A-Alert - send an alert email when a file arrives this option also requires an email address(es) in alert-email-address:... eg E|Pittlewire|zong|0|Mr Pittles Image Agency||2edsys||#DF:PITTLEWIRE.FIP#EQ:pittle| E|Brittle||0|Mr Brittle HardHat Agency||xchg|#CX:B2FIP#DF:BRITTLE.FIP#EQ:brittle| -- FipHdr fields added to each file UNLESS the Standalone option has been flagged SP IPaddress of the remote host SN Filename given ZO Safe filename SU -n input switch or FTPWIRE SA logon name SC chrset - defaults to ASCII S1 client description for the logon file S2 wire id S3 session id S4 current PWD S5 any extra subfolder in a put command S6 current Client system details (if offered by remote) Input Parameters : Optional : -9 : do not use Speedy on a Speedy system -A : force the ApiKey to this default: always need an apikey as 1st command This is for cases where the UNIX logon maps to a single Fip logon -d : done folder default: none This holds a copy of all incoming data files from every source The structure is (done folder) / (date)_(logon) eg 20110921_fip / (filename as written to the output folder) It can be purged with an entry in maintenance (zapfiplog) eg if '-d raw.ftpwire' and we want the last 30 days data /fip/bin/ipdelque -q/fip/spool/raw.ftpwire -i1 -a30 -E : maximum number of threads default: 1 up to a max of 200 (not Win2k) Note this is also a hardware limit in that small systems may not be able to run more than 50 or so -F : allow SITE FIPHDR commands default: no -I : wire id default: 0 used to track which instance of a multi-ftpwire system a file arrived/logged -j : balance queue for balancing doneque items default: 2balance -J : balance group for balancing doneque items default: -none- no balancing This group MUST be in sys/BALANCE -l : log level -n : name of this wire default: FTPWIRE -O : Name of the output folder if not default default: spool/2brouted This folder will be under /fip/spool -P : Port for control default: 9130 -P 21 is normal -R : round-robin base - see above default: 1 -s : local hostname or ipaddress default: all local addresses where a server has multiple ip address/hostnames, use '-s' to restrict connections to a single address -S : default is standlone and not Fip default: next folder is fip in this case do NOT add a Fiphdr and preserve exactly the incoming filename this can be overridden in the Logon file -T : log timing stats default: no -v : Print the version number and exit -- Log levels for -l input switch are : default (-1) errors only are logged 0 connections/disconnections 10 logons 20 each file in or out -- The -D input switch will display all cmds etc as they come in eg -- Secure FTP -------------------------------------- Confusingly there are two - completely different - 'Secure' FTPs plus a defunct company called SecoueFTP etc etc 1. a more secure version of ordinary FTP which uses SSL/TLS in the same way http and httpS work for secure web sites. 2. a file copy layer which sits ontop of SSH - which has nothing at all to do with ordinary FTP The extra confusion is that SSH uses SSL - so dont mix the two up ! ((This is from the FileZilla website as they are the kings of ftp : .. TLS (FTPS) vs SSH (SFTP) FTPS (FTP encrypted with TLS) should not be confused with SFTP (SSH). The latter is a completely different protocol. .. Explicit vs Implicit FTPS FTPS (FTP over TLS) is served up in two incompatible modes. If using explicit FTPS, the client connects to the normal FTP port and explicitly switches into secure (TLS) mode with "AUTH TLS", whereas implicit FTPS is an older style service that assumes TLS mode right from the start of the connection (and normally listens on TCP port 990, rather than 21). In a FileZilla client this means prefixing the host with "FTPES://" to connect an "explicit" FTPS server, or "FTPS://" for the legacy "implicit" server (for which you will likely also need to set the port to 990). Thank you FileZilla)) How do you know which is the one you want ? - What is the port number on the remote server ? port 21 - it is BOTH normal FTP and the SSL/TLS version (port 21 is the same as normal FTP) test with ordinary 'ftp' client port 990 - it is ONLY SSL/TLS version test with 'telnet' to (remhost) 990 and cut the connection once you are satisfied it connects port 22 - it is ONLY the sftp on top of SSH test with 'ssh' or 'sftp' --- 1. SSL/TLS YOU MUST USE ftpwiressl for any/all ssl/tls traffic as ftpwire blocks these commands. --- 2.a SSH/SFTP Ok - use sftpwire ................. -- SFTPWIRE Installation 1. create a new single logon - example is 'fipdata' As root , add the logon (make sure it has a shell and is NOT nologin useradd -m fipdata passwd fipdata (add a passwd) - check it works su - fipdata ................. 2. as fip - make sure sftpwire is in /fip/bin ................. 3. as fip - check the name of the FTPWIRE parameter file in wire this wil be the '-n' parameter the following uses FTPWIRE_COMMON as an example ................. 4. as fip - check at least one logon in the logon file pointed to by wire/FTPWIRE_COMMON and that the 4th field is a unique apikey (no spaces or | or utf8 chrs) E|(logon)|(pwd for ftpwire)|(apikey for sftpwire)|... ................. 5. as sudo/root - modify /etc/ssh/sshd_config Here are the differences DO THIS VERY CAREFULLY as you might lock existing users out ! Make a copy first ! root@violet:/etc/ssh# cp -p sshd_config sshd_config.asatDATE 5.a if you are restricting users with 'AllowUsers', add fipdata AllowUsers (previous entries...) fipdata 5.b if you want users to logon as well as shared keys, # comment OUT 'PasswordAuthentication' #11may17 PasswordAuthentication no PermitEmptyPasswords no 5.c add this block at the bottom of the file - all 'Match'es must be the last part (NOTE use spaces, not tabs, for the indent : # ------------------------------------ # these are the last few lines of the sshd_config file # 11may17 Match User fipdata ForceCommand /fip/bin/sftpwire -n FTPWIRE_COMMON AllowTCPForwarding no X11Forwarding no # ------------------------------------ or ... # ------------------------------------ # ONLY allow fipsec : sftp on 9123 Match User fipsec LocalPort 22 DenyUsers fipsec Match User fipsec LocalPort 9122 DenyUsers fipsec # -A apikey (not Logon) Match User fipsec LocalPort 9123 ForceCommand /fip/bin/sftpwire -n FTPWIRE_SFTP -A PDS+JNMND88aj AllowTCPForwarding no X11Forwarding no # ------------------------------------ 5.y as sudo/root - restart sshd systemctl restart sshd 5.z test the new settings have NOT mashed the existing settings use ssh to connect and check you can STILL logon on to the server 6. File/Folder Permissions - make sure fip can read/write/delete the new files created by the new logon 6.a groups - as sudo/root vi /etc/group find the 'fip' group add both fip and the new logon to the end of the line : fip:x:1000:fip,fipdata 6.b chown/chmod chown -R fip:fip /fip chown -R fip:fip /fip/(star) chmod -R 770 /fip/(star) or, depending pn your preference chmod -R 775 /fip/(star) ................. 7. check your firewall settings to make sure the remote can login ! eg for IPtables if you want to restrict to IP address range (check range online at aomewhere like https://mxtoolbox.com/subnetcalculator.aspx -A INPUT -p tcp -m tcp -m state --state NEW --dport 22 -s 188.8.131.52/26,184.108.40.206/32 -j ACCEPT ................. 99. test ! ........................................................................... Version Control ;1a-f 28apr20 minor ;f added forceApiKey ;0a-v 12sep16 chj original version (copyright) 2020 and previous years FingerPost Ltd.