ipsmtp This feeds and receives from Internet Mail. Use it to send to an SMTP server (local or remote) and receive either on a local 'sendmail' server (Unix only) or from a POP3 server. We define OUTGOING as FIP -> SMTP and the reverse (SMTP or POP3 -> FIP) INCOMING. (use program 'smtpwire' to receive incoming SMTP traffic or 'imapwire' to grab from an IMAP server) Routing Files Outbound ---------------------- Messages for sending are put in the input queue, usually spool/2smtp, with the FipHdr field DA containing one or more Mail Address (separated by commas if multiple) - such as firstname.lastname@example.org. Messages can be sent directly to the spool/2smtp queue, routed normally by Fip or as a group client for Syndication by IPPOST. In the latter case, the client is specified as type:unixmail and there is a special DU called TOUNIXMAIL in USERS which has the syntax : TOUNIXMAIL= DP:(hostname) DQ:2smtp DC:(whatever) where hostname is the name of the UnixBox doing the transfer and whatever is the character set required. The From field or sender is the header field SU or, if it exists, the SA (Source Author) which is formatted as SA@SU. SU is normally the Source system or type while SA is the normally the logon or sender. If the SA is an Atex style 'LOGON ;dd/mm,hh/nn' then only the logon is used. So if the SA:fippo and SU:liverpool-fc.co.uk, the sender becomes email@example.com If the SA is ALWAYS the full address - like SA:firstname.lastname@example.org, then use the '-F' input switch to NOT add the SU header field. If neither SU nor SA exist, the default sender is 'fip' unless overridden by the 'defsender:' in the parameter file. Note that this name must be in the passwd file as a valid logon. The data of the outgoing messages will be converted if 8 bit or control chrs to the normal '=XX' hex format used. Routing Files Inbound --------------------- Messages from the Internet are sent to the special DU, FIPMSG which is either hardcoded to a specific destination, or more usually, directed to IPPOST to detetmine if the recepient is on the editorial system OR is a mailbox (IPBOX) customer. Testing ------- If things do NOT look like they are working, you can run ipsmtp/ipsmtpssl manually with the -1 and -D to run once and display the handshake. Parameter File -------------- The parameter file, normally tables/mail/SMTP, is read for the logon names to scan on a regular basis plus the destination to send these to. The syntax of the parameter file is : ; comment defsender: (default sender) default: fip Used ONLY if there is NO fipHdr field SU nor SA. This is used for the 'From:' field for outgoing messages. forcesender: (Force sender to this logon) default: no force-sender: (Force sender to this logon) default: no default-sender: (Force sender to this logon) default: no This is the default reply-to address. It must be a valid logon and domain. It is also used for the 'From:' field for ALL outgoing messages if there is NO 'forcefrom' (see below) or FipHdr field 'DJ' in the file to send irrespective of whether there is a FipHdr SA or SU. default-sender is used ONLY if there is no force-sender and nothing in DJ It should be JUST the address : forcesender: email@example.com forcefrom: (Force the From field) force-from: (Force the From field) This is the From field in the outgoing file which does not have to be the same as the sender's address. If using the '-h' host input switch it can contain a 'pretty' version : eg forcefrom: "MP3 king Steinhoff" <firstname.lastname@example.org> defdest: (default Fip Destination (DU FipHdr field) default: FIPMSG helo-host: (hostname.domain) ehlo-host: (hostname.domain) Use this name for the HELO or EHLO (hostname) message to the remote (SMTP outbound) who normally want a full hostname-domanme which is in DNS so they can verify it. default is just the hostname eg ehlo-host:mail.hashtagnotme.com This field can be FipSeq - a temporary FipHdr field S1:(hostname) Of course S1 is dependent on the correct hostname being setup on that server ! add-return-path: (FipSeq for address) Add a 'Return-Path: (address)' to the outgoing SMTP mime header default: no eg combie:QR SA,email@example.com add-return-path:\QR chrset: (Source character set ie SC header field) default: ascii hash-in-address: (FipSeq Chr) Some old-style email addresses have a leading hash/pound/# sign - but a hash is a field sepearator in FipHdr. So another chr is used in the DA FipHdr and mapped at the last moment. default is \243 hash-in-fiphdr: (FipSeq Chr) Sometimes a Subject line might have a hash/pound/# sign - but a hash is a field sepearator in FipHdr. So another chr is used in the DA FipHdr and mapped at the last moment. default is '#' (ie a hash is a hash and that is where the FipHdr field ends !) eg in tables/route file, map a HASH out of the way : HASH-IN-HDR:\035 in the ipsmtp parameter file, to map back : hash-in-fiphdr:\035 fiphdr-remote-address: (2 letter FH) newDA: (2 letter FH) FipHdr field to replace the DA field which default: DA holds the email address of the person to sent to. eg newDA:DZ This can also be an combination field : combie:QA DA|DZ|NA newDA:QA fiphdr-local-address: (2 letter FH) newSA: (2 letter FH) FipHdr field to replace the 'MAIL from' address field. default: SA fiphdr-subject: (2 letter FH) newDI: (2 letter FH) FipHdr field to replace the DI field which is used for the 'Subject' field. default: DI fiphdr-from: (2 letter FH) newDJ: (2 letter FH) FipHdr field to replace the DJ field which is used for the 'From' field. default: DJ fiphdr-reply-to: (2 letter FH) FipHdr field which is used for the 'Reply-to' field. default: none and From: is used balance-seqno: Send the Sequence number to this Balance Group (see 'ipbalan') balance-delete: Delete the Data on remote systems once sent using this Balance Group (see 'ipbalan') forcedomain: (domain name) default: do not Strip any domain in the FipHdr field SA and add this one in its place. forcedomain:commies.com So if the SA:firstname.lastname@example.org, then the from field becomes From: email@example.com strip-domain-on-send: Send only the logon/mail name default: domain too and strip any domain name. defsubject: (default Subject line if nothing specified) default: "Message for you" att-message: Message to replace an attachment with. default:** Attachment has been Stripped, pls contact System Managers for (name of attachment file) log-line: (FipSeq) Additional information for the log file entry in ALL extra-log-fiphdr: Fipseq FipHdr fields to add to any transmission log This is triggered on a file by file basis by the FipHdr field ZSMTP_LOG:(dest) in the file to send - (dest) being a valid DU in tables/sys/USERS. Use this to pass FipHdr fields from the incoming file to the outgoing default: none eg. extra-log-fiphdr:\n\SI:\SI\n send-new-fiphdr: Fipseq FipHdr fields to add to any outgoing file to act as defaults etc send-extra-fiphdr: Fipseq FipHdr fields to add to any outgoing file to act as defaults etc The NEW variant is added BEFORE the real FipHdr, so can be used for Defaults (and the real FipHdr fields can override these). The EXTRA variant will add at the end of the file's FipHdr and so will override any setting, This is triggered on a file by file basis by the FipHdr field default: none eg. send-extra-fiphdr:\nZSMTP_TYPE:text/plain\n stream-connections:yes default: no Do NOT drop the connection/reconnect for every file. Once a connection is up to a mailserver, any subsequent file for the same server is sent in the same transmission. (using SMTP only) keepalive-every: 50 default: 30 secs Streaming only - Min time in secs between keep-alive (CRNL) offline-queue:(FipSeq) default: none Offline queue if item errors (same as and overwrites -x input switch) ipsmtp will NOT scan the offline queue - the assumption is that another ipsmtp will be polling that folder. fiphdr: (Map incoming mail field to a Fip Hdr field) default: none fiphdr:DI Subject fiphdr:AZ Message-Id ignore: (Mail header field to always ignore) default: none ignore: Status ignore: Apparently-To ignore: X-Msmail-Priority decode: unencode: Decode any base64 sections. default: no decode: This will NOT decode attachments decode:text This will NOT decode attachments decode:all This will decode body AND attachments maxattempts: (no of attempts before the file is sent to woops) default: 10 This checks the FipHdr 'DO' which is managed internally. (UNIX only) logeachfile:(dest) Send a Success/failed msg to this destination for each file. There is no default. This log file is just a FipHdr with the following extra fields : DR-File Sent OK DR:ok or DR:error DG-Will Retry later DG:retrying or DG:stopped DT-Some message text DT:No connection DO-Number of last attempt DO:5 HT-Date and Time HT:25 Nov 98 15:35:25 default: no log created. The DR and DG messages can be changed by using : log-dr-ok:(FipSeq) log-dr-error:(FipSeq) log-dg-stop:(FipSeq) log-dg-retry:(FipSeq) eg: log-dr-ok:File \VN sent to Remote Number \DA (\$h:\$n) msgeachfile:(FipSeq) Additional information to add to the FipHdr of the 'logeachfile' msg. This should be in FipHdr format and be in FipSeq. It can be used to pass FipHdr fields in the outgoing file into the log file. eg msgeachfile: DF:logdial\nSS:\SS\n default: nothing added send-via-smtp-relay: (yes/no) For SMTP outgoing files : YES - send everything to a single SMTP Relay host (default) NO - send each file to the mailhost of the recipient (Same as the -X input switch) local-domain: (domain name) (space) (IP address of local mail server) For SMTP outgoing NOT sending via a reply There can be up to 10 local-domains local-domain-queue: (queue in spool) local-domain-name: (filename in FipSeq) For any match to local-domain eg local-domain:220.127.116.11 local-domain-queue:2go ; a single DA is added at the start; add XX:\$u.\$z.\$v for uniqueness local-domain-name:#DU:inhouseEmail#SA:firstname.lastname@example.org#SN:\SN#XX:\$u.\$z.\$v default-smtp-host: (hostname or IPaddress) For sending emails out - Same as the '-h' Input Switch alternate-smtp-host: (hostname or IPaddress) a failover hostname for if the default fails connection-timeout: (secs) Max time in seconds for the connection to the SMTP server The call is aborted if it takes longer than this. If the remote is across the computer room, this time can be reduced to 15 or 20 secs; if over the internet, 120 secs is the norm. default: 120 for 120 seconds, minimum is 10 send-single-smtp: send smtp in one file per session which is then cut and reopened for the next file/session. hostname: (hostname to use) Cosmetic ! port: (port number of the remote) This overwrites the -P input switch timing-stats: yes/no Show timing stats if the incoming file has the HE/HR fiphdr default: no use-sx: or use-external-file: if there is an SX FipHdr field with a path to the data file, use that rather than the data in the input file. wrap-subject-chr: (FipSeq chr) If the Subject field has this chr, wrap the line default: none This might have been added by ipxchg : z:convert-fiphdr-to-qp:DI:DJ:utf-8:\002 wrap-subject-chr:\002 force-subject-wrap: (No of chrs) Normally subject lines at this total default: 76 eg force-subject-wrap:76 locale: use a different 'locale' ( look at the man pages for locale) Most computers are set to US English and never changed. Use this parameter to customise any date/time or Chr translations The parameter MUST be a valid locale on your system! To find out what valid locale exist, type 'locale -a' on unix. Eg ; Set for for Brazil, portugese locale:pt_BR check-primary-server: pseudo-host name that is specified in tables/sys/DEST_REDUN Use this when for POP3 accesses ONLY where ipsmtp on 2 systems is accessing the same POP3 account(s) in a redundant way. It is used to decide whether the current host should be getting the files or not. ie in the parameter file is check-primary-server:timersvr and in the DEST_REDUN is ; psuedohost primary secondary timersvr fip1 fip2 and in the SYSTEM file for both fip1 AND fip2 there is the same 'ipsmtp' line with the same parameter file If fip1 is up, 'ipsmtp' on fip1 will always get while on fip2 it will just check/loop. If fip1 is down 'ipsmtp' on fip2 will start processing the files. use-tls:yes/implicit/explicit/no The commends are for a ftp running over SSL/TLS on the remote server default is NO no - normal, standard SMTP on (normally) port 25 yes or explicit - connect (normally) on port 465 in clear then send STARTTLS and SSL for AUTH implicit - connect (normally) on port 587: use SSL for all conversations tls-auth: (XXX) AUTH type for TLS/SSL default: TLS auth-login: (data) auth-pwd: (pwd) login and password for TLS AUTH LOGIN authentication auth-plain: (string) login and password for TLS AUTH PLAIN authentication To generate the auth-plain string : printf "(logon)\0(logon)\0(password)" | sffb64 -i ssl-method: (1,2,3,23,999) Version number to use for TLS/SSL default: 999 for current default (2 or 3) ssl-password: (password) ssl-passwd: (password) default: none Optional password if the handshake requires a shared secret ssl-pem: (name of a PEM certificate file) default: none ssl-cert: (name of a PEM certificate file) default: none ssl-root-cert: (name of a root PEM certificate file) defaunt: none Optional certificates - held in tables/ssl ssl-verify: yes/no verify certificates default: yes ssl-ciphers: (list) acceptable ciphers (use 'openssl ciphers' to list) default: "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS" (from 16v25) -- For HTTP proxies : proxy-connect: (FipSeq) Proxy Connect (or Wakeup) string for network device default: none This wakes up the pad on startup or Connects via a proxy proxy-host: (hostname) proxy-port: (port) These are ONLY used for the TLS hostname checking as the name may be different to the connection IPaddress proxy-is-squid: (yes/no) - placeholders proxy-handshake: (yes/no) - placeholders -- For Socks 4/5 proxies : use-sock: (4 or 5) socks-host: (hostname of the socks proxy) no default socks-port: (port number of the socks proxy) default: 1080 socks-user: (user name for the socks proxy) no default if nothing specified, assumed that there is none socks-pwd: (password for the socks proxy) no default -- For accessing Oauth protected assets use-oauth:yes/no Use OAUTH to grab/use an access-token or Bearer token eg for Gmail access default is NO oauth-refresh-script: (Script in FipSeq) script to generate the access_token using a refresh_token These 5 FipHdrs are use to generate, check, add/renew permissions to access the remote data - normally Gmail. oauth-client-fiphdr: (FipHdr) default: IC oauth-secret-fiphdr: (FipHdr) default: IS oauth-access-fiphdr: (FipHdr) default: IA oauth-refresh-fiphdr: (FipHdr) default: IR oauth-expiry-fiphdr: (FipHdr) default: IX -- Then for EACH logon you wish to track INCOMING Sendmail mail boxes, specify a line begining : logon: (logon name) default: none or unix: (unixlogonname) default: none These two parameters are identical. There is also a means, for UnixBoxes only, to poll for ALL mail for ALL the members of a group (as in the /etc/group file) : group: (unixgroupname) default: none Whether 'logon', 'unix' or 'group' there are other (optional) parameters: sendto:(newaddress) fiphdr:(FIP hdr sequence) striphdr: keephdr: pop3pwd: (Password for POP3) where sendto: allows you to specify another name for the DA field IPPOST will use this to route. By default, the Fip Hdr field DA will hold the logon name. fiphdr: allows you to add to the FIP hdr. This may be any - perhaps the DU field to change the destination. striphdr: will strip the SendMail header except for the From and Subject lines pop3pwd: specifies the password to use IF you are accessing a POP3 server to receive incoming mail. It is NOT required for UnixBoxes scanning local mail queues. -- To add a new logon : 1. Read the 'passwd' file for the exact Unixlogon name - eg newboy 2. Enter this name in SMTP file as : logon:newboy 3. If this is the same name as in file ATXMSG, enter line newboy edone where edone is the name of your editorial system 4. If 'newboy' is really called 'dirtyoldman' on the 'atex' editorial system, the entries should be: SMTP file - logon:newboy sendto:dirtyoldman ATXMSG file - dirtyoldman atex -- To setup IPSMTP for Sendmail mailboxes : 1. Create logons as above 2. Check you have destinations in the USERS file : FIPMSG= DP:fipmain DQ:post DC:atexeol MSGFORATX_EDSYS= DP:edsys3 DQ:atexq-grp DC:atexeol where 'fipmain' is the name of this UnixBox. Destination FIPMSG is generated by IPSMTP. Destination MSGFORATX_* is generated by IPPOST IF the name is in the FIPMSG file. eg if the Editorial System is called ATEX (environment variable FIP_EDSYS) and teh queue required is email-wir on system3 : MSGFORATX_ATEX= DP:atex3 DQ:email-wir DC:atexeol 3. Input Queue spool/2smtp needs to be added 4. IPPOST needs to be running too (unless you change the FIPMSG DP/DQ to somewhere else) 5. Outbound Format is ATXMSG for IPGTWY/IPOUT/IP2WTR etc. Please check if this exists and has the correct setup The current version can scan up to 1000 Unixmail boxes. -- Where sections of FipHdr fields are required or changes to the output style, use keywords : fixed, partial, combie, optional, repeat, newdate and/or style. (see The SysA dmin manual for more information). They are normally specified : fixed:QZ 1234543 partial:QT ST,3,2,U,<,> combie:QY ep|na,(0000000)a option:QE ep,11,7,s repeat:QK XK,-,3 or repeat:QP PK,,4,#X style:QS XN,%.03d replace:QN NN abc=DEF def=GHI -- For outbound traffic, FipHdr fields used are : Mandatory DA holds one or more remote email addresses multiple addresses are separated by a comma. Can be overridden by 'newDA' Can be changed by 'forcedomain' Optional SA holds the sender's email address can be overridden by 'forcesender' (actual address) can be overridden by 'forcesender' (from field) can be defaulted by 'defsender' DI holds the Subject: field DJ holds the From: field ZSMTP_BINARY if not 'no' will send the file as Binary and make NO attempt to change the length ZSMTP_ENCODE holds the Content-Transfer-Encoding default to Content-Transfer-Encoding:quoted-printable ZSMTP_TYPE holds the Content-Type. no default ZSMTP_LOG will send a log of the SMTP transmission to this destination (for SMTP outbound traffic using the '-h' input switch only) ZSMTP_HOST IPaddress or hostname of the mail server if NOT the default (for SMTP only). ZSMTP_ADDMIME if not 'no', add standard MimeHeaders to the top of the file (default is YES, unless -H is set for NO) ZSMTP_CHECKMIME if not 'no', check the MimeHeaders (overrides check-mime-header:yes/no) Input Switches -------------- Input parameters are : all optional : -1 : Get once only then stop for POP3 or Receive only. default: spool Either -p (path) or -3 (pop3 server) must be used with this option This also incorporates -Q so no queu is scanned and no mail sent. -2 : (filesname) Send a single file by SMTP and stop default: spool -3 : Hostname where the pop3 daemon is running. default: none Note -p and -3 are mutually exclusive. -a : queue for non-text attachments default: none default is that attachments are passed through in text -A : destination for non-text attachments default: none default is that attachments are passed through in text -C : do NOT convert unprintables to "=XX" format default: do -d : do NOT delete each unix msg default: delete -D : display the conversation with the remote server default: no valid ONLY with the -2 or -1 for single shot; used for debugging troublesome connections -e : extra FipHdr/FipSeq which will be added for each outgoing file. default: none -f : do NOT output the original filename as the first line of text. default: output name -F : do NOT output the original sending system (SU) as the From field. default: From: SU -h : Hostname where the smtp daemon is running. default: that system NOTE that sendDirect can be seelcted by either -X or -h+ -H : do NOT add an extra Mime Headers as the file is in the correct format. default: add To/From/Subject -i : Receive spooled queue if you are using 'sffmail' default: using Path -k : alternate host name default: none use this for checking primary/secondary for clustered systems where the actual hostname differs from the cluster name ie if the host name may be fipcluster-a or fipcluster-b but there is a cluster hostname of fipcluster -K : quiet mode - message if NOT the primary system default: do not message -l : do NOT log each msg in or out default: log -m : (ipchkmail folder name) default: process inside ipsmtp Do not convert incoming SMTP and POP3 files immediately but pass them to ipchkmail running on this folder which does a better job! especially for attachments-within-attachments -o : Next fip queue for incoming files default: spool/2go -O : Done queue for spooled files out default: none -p : Path on this system holding mailboxes default: depends on the OS Unix only. The full pathname MUST be stated. Note -p and -3 are mutually exclusive. -P : port of remote host default: 25 Note normal usage is -P 465 if using 'use-tls:implicit', or -P 587 for 'use-tls:explicit' Also note that parameter 'port:' will override this value -q : Send Queue to spool default: spool/2smtp -Q : Do NOT send any mail and therefore do NOT scan any input queue. default: scan -r : eolns for the data part should be CR NL default: just NL -R : do NOT resolve hostnames default: do resolve This will resolve any hostnames in the DEST_REDUN file (if there is one) -S : do NOT split files on the "From ..." line default: do split -t : scan time of queue default: 3 secs -T : log the trace of each transaction default: do not log Normally a one line log of each file sent is stored in log/remote_trace with a name of 'date_Address'. -w : for file->Sendmail, set line length. default: 76 + translate and do NOT translate unprintables to "=XX" format To NOT wrap text and no translations -w 0 This can be ignored for a single file if it has the FipHdr field ZSMTP_BINARY: -W : for files outbound, set JUST the line length default: 76 + translate but still translate unprintables. -x : Offline queue default: none Normally ipsmtp will retry - and possible delay files also queued up - any message that has not been delivered. Use this to shuffle files that error to a different folder where another ipsmtp will take care of the baduns -X : Send directly to the Email server of each destination default: no, use a single relay server This depends on the resolv.conf or Windows DNS to be setup correctly of course. NOTE that sendDirect can be seelcted by either -X or -h+ -z : parameter file default: mail/SMTP -v : display version number and exit. Now a statement of sanity (?) about incoming mail from the Internet : If like most of the civilised world you prefer to keep well away from anything 'sendmail' can throw at you, then let ipsmtp just grab its files from the sendmail internal queue. Or even better use POP3 and forget about it all - no setup, no nothing extra! If you are unlucky enough to have a Sendmail specialist on your staff, before sending for the men in white coats, point out there is a '-p path for unix mailboxes' switch. Your guru should dictate (they always dictate - though they claim they are working with your best interests in mind) where he is willing to drop files for those logons you are interested in and add the -p switch to the tables/sys/SYSTEM file. Easy huh ! Note there was a subtle change at version 15 which changed the default from ATXMSG to FIPMSG. Version Control ;16v50 9nov01 cleanup after failed connections better ;a 16mar02 added remote_trace and -O done queue ;b-c 03jun03 cleanup line-enders in mimehdr and streaming for smtp and added ZSMTP_HOST ;d-i 22sep03 added send-extra-fiphdr: and stream-connections:yes ;j-l 29jun04 added -1 and speedy ;m-p 17sep05 added alternate-smtp-host and hostname: and log-line ;q 10aug06 added remote_trace style timings to send file done msg as H4 ;r-t 21sep06 allow quotes in email address and wrap long subject lines ;u 13apr07 added extra-mime-headers: ;v12 06feb08 added hash-in-address ;2 modette for size of email list ;3-4 added check-primary-host for POP3 and chkmail-folder: ;5-6 bugette with send_smtp and logging long DA ;7 3sep10 added use-sx ;8-9 10nov10 bug in pop3 when Uidl has non-alnum ;10-11 19may11 bugette when TLS does not work and b64 '=' padding ;12 22aug13 added POP3 error message on logon/pwd and added -1 for display ;13-14 30sep13 added hash-in-fiphdr ;15 30apr14 file-trace ;16 7oct14 if -H and not -C, do not encode any mime headers in the data ;17-18 26jun15 make donque variable ;19-22 25jul16 tuning and buffersizes and better error messages ;23-24 3oct17 AUTH PLAIN and AUTH LOGIN now work with new params auth-login, auth-plain, auth-pwd ;25-26 3nov18 redid SSL for new defaults for ciphers and certs plus done-queue as a parameter ;27 18dec18 auth-logon/pwd are now parsable plus added -e (extraFH) ;28-29 tuning maxFipHdr -> limit and maxTO->998 ;30-31 26feb19 senddirect tuned ;32-33 added ssl-display ;34-42 redid direct and added ZSMTP_ADDMIME fiphdr and ZSMTP_CHKMIME fiphdr ;43 added local-domain* ;44-46 11dec19 added S1:(host) and socks-proxy better ;47 WINNT buglette strftime and locale ;48 14dec20 nicer error checking for streaming and added keepalive-every ;49 18aug21 minor for s_PIPE plus fiphdr-* and Reply-to: fields ;50 11mar22 added Oauth2 for Gmail (copyright) 2022 and previous years FingerPost Ltd.