ipsmtp

ipsmtp

This feeds and receives from Internet Mail.

It sends to an SMTP server and can receive either on a local server (Unix only)
or from a POP3 server.

We define OUTGOING as FIP -> SMTP and the reverse (SMTP or POP3 -> FIP)
INCOMING.

Routing Files Outbound
----------------------
Messages for sending are put in the input queue, usually spool/2smtp, with the
FipHdr field DA containing one or more Mail Address (separated by commas if
multiple) - such as wizzo@fingerpost.co.uk.

Messages can be sent directly to the spool/2smtp queue, routed normally by Fip
or as a group client for Syndication by IPPOST.

In the latter case, the client is specified as type:unixmail and there is a
special DU called TOUNIXMAIL in USERS which has the syntax :
    TOUNIXMAIL= DP:(hostname)   DQ:2smtp    DC:(whatever)
    where hostname is the name of the UnixBox doing the transfer
    and whatever is the character set required.

The From field or sender is the header field SU or, if it exists, the SA
(Source Author) which is formatted as SA@SU. SU is normally the Source system
or type while SA is the normally the logon or sender. If the SA is an Atex
style 'LOGON ;dd/mm,hh/nn' then only the logon is used.
So if the SA:fippo and SU:liverpool-fc.co.uk, the sender becomes
        fippo@liverpool-fc.co.uk
If the SA is ALWAYS the full address - like SA:hedgehog@flea-pit.com, then use
the '-F' input switch to NOT add the SU header field.

If neither SU nor SA exist, the default sender is 'fip' unless overridden by
the 'defsender:' in the parameter file. Note that this name must be in the
passwd file as a valid logon.

The data of the outgoing messages will be converted if 8 bit or control chrs to
the normal '=XX' hex format used.

Routing Files Inbound
---------------------

Messages from the Internet are sent to the special DU, FIPMSG which is either
hardcoded to a specific destination, or more usually, directed to IPPOST to
detetmine if the recepient is on the editorial system OR is a mailbox (IPBOX)
customer.

Testing
-------

If things do NOT look like they are working, you can run ipsmtp/ipsmtpssl
manually with the -1 and -D to run once and display t
he handshake.

Parameter File
--------------
The parameter file, normally tables/mail/SMTP, is read for the logon names to
scan on a regular basis plus the destination to send these to. The syntax of
the parameter file is :
    ; comment
    defsender: (default sender)             default: fip
        Used ONLY if there is NO fipHdr field SU nor SA.
        This is used for the 'From:' field for outgoing messages.
    forcesender: (Force sender to this logon)       default: no
    force-sender: (Force sender to this logon)      default: no
    default-sender: (Force sender to this logon)        default: no
        This is the default reply-to address. It must be a valid logon and domain. It
is also used for the 'From:' field for ALL outgoing messages if there is NO
'forcefrom' (see below) or FipHdr field 'DJ' in the file to send irrespective
of whether there is a FipHdr SA or SU.
        default-sender is used ONLY if there is no force-sender and nothing in DJ
        It should be JUST the address :
            forcesender: lucy@sky-with-diamonds.com
    forcefrom: (Force the From field)
    force-from: (Force the From field)
        This is the From field in the outgoing file which does not have
        to be the same as the sender's address.
        If using the '-h' host input switch it can contain a 'pretty'
        version :
        eg  forcefrom: "MP3 king Steinhoff" <dohrayme@singalong.com>
    defdest: (default Fip Destination (DU FipHdr field) default: FIPMSG
    chrset: (Source character set ie SC header field)   default: ascii
    newDA: FipHdr field to replace the DA field which   default: DA
        holds the email address of the person to sent to.
        eg  newDA:DZ
        This can also be an combination field :
            combie:QA   DA|DZ|NA
            newDA:QA
    hash-in-address: (FipSeq Chr)
        Some old-style email addresses have a leading hash/pound/# sign - but a hash
is a field sepearator in FipHdr. So another chr is used in the DA FipHdr and
mapped at the last moment.
        default is \243
    hash-in-fiphdr: (FipSeq Chr)
        Sometimes a Subject line might have a hash/pound/# sign - but a hash is a
field sepearator in FipHdr. So another chr is used in the DA FipHdr and mapped
at the last moment.
        default is '#' (ie a hash is a hash and that is where the FipHdr field ends
!)
        eg in tables/route file, map a HASH out of the way  : HASH-IN-HDR:\035
            in the ipsmtp parameter file, to map back   : hash-in-fiphdr:\035
    newDI: FipHdr field to replace the DI field which   default: DI
        is used for the 'Subject' field.
    newDJ: FipHdr field to replace the DJ field which   default: DJ
        is used for the 'From' field.
    balance-seqno:  Send the Sequence number to this Balance Group
            (see 'ipbalan')
    balance-delete: Delete the Data on remote systems once sent
            using this Balance Group (see 'ipbalan')
    forcedomain: (domain name)              default: do not
        Strip any domain in the FipHdr field SA and add this one
        in its place.
            forcedomain:commies.com
        So if the SA:albert@redsunderthebed.com,
        then the from field becomes
            From: albert@commies.com
    strip-domain-on-send: Send only the logon/mail name default: domain too
        and strip any domain name.
    defsubject: (default Subject line if nothing specified)
            default: "Message for you"
    att-message: Message to replace an attachment with.
        default:** Attachment has been Stripped, pls contact System Managers for
(name of attachment file)
    log-line: (FipSeq) Additional information for the log file entry in ALL
    extra-log-fiphdr: Fipseq FipHdr fields to add to any transmission log
        This is triggered on a file by file basis by the FipHdr field
        ZSMTP_LOG:(dest) in the file to send - (dest) being a valid
        DU in tables/sys/USERS. Use this to pass FipHdr fields from
        the incoming file to the outgoing
        default: none
        eg. extra-log-fiphdr:\n\SI:\SI\n
    send-new-fiphdr: Fipseq FipHdr fields to add to any outgoing file to act as
defaults etc
    send-extra-fiphdr: Fipseq FipHdr fields to add to any outgoing file to act as
defaults etc
        The NEW variant is added BEFORE the real FipHdr, so can be used for Defaults
(and the real
        FipHdr fields can override these).
        The EXTRA variant will add at the end of the file's FipHdr and so will
override
        any setting,
        This is triggered on a file by file basis by the FipHdr field
        default: none
        eg. send-extra-fiphdr:\nZSMTP_TYPE:text/plain\n

    stream-connections:yes              default: no
        Do NOT drop the connection/reconnect for every file.
        Once a connection is up to a mailserver, any subsequent file
        for the same server is sent in the same transmission.
        (using SMTP only)

    fiphdr: (Map incoming mail field to a Fip Hdr field)    default: none
        fiphdr:DI   Subject
        fiphdr:AZ   Message-Id
    ignore: (Mail header field to always ignore)    default: none
        ignore: Status
        ignore: Apparently-To
        ignore: X-Msmail-Priority
    decode:
    unencode: Decode any base64 sections.   default: no
        decode:  This will NOT decode attachments
        decode:text This will NOT decode attachments
        decode:all  This will decode body AND attachments
    maxattempts: (no of attempts before the file is sent to woops)  default: 10
        This checks the FipHdr 'DO' which is managed internally. (UNIX only)
    logeachfile:(dest) Send a Success/failed msg to this destination for each
file.
            There is no default.
            This log file is just a FipHdr with the following extra fields :
                DR-File Sent OK  DR:ok or DR:error
                DG-Will Retry later DG:retrying or DG:stopped
                DT-Some message text    DT:No connection
                DO-Number of last attempt    DO:5
                HT-Date and Time    HT:25 Nov 98 15:35:25
            default: no log created.
            The DR and DG messages can be changed by using :
                log-dr-ok:(FipSeq)
                log-dr-error:(FipSeq)
                log-dg-stop:(FipSeq)
                log-dg-retry:(FipSeq)
            eg: log-dr-ok:File \VN sent to Remote Number \DA (\$h:\$n)
    msgeachfile:(FipSeq) Additional information to add to the FipHdr of the
'logeachfile'
            msg. This should be in FipHdr format and be in FipSeq. It can be
            used to pass FipHdr fields in the outgoing file into the log file.
            eg  msgeachfile:    DF:logdial\nSS:\SS\n
            default: nothing added
    send-smtp-directly: (yes/no)
        Do not use a single or a pair of hosts and send directly to
'mailhost.(domainname of remote)'
        (Same as the -H input switch)
    default-smtp-host: (hostname or IPaddress)
        For sending emails out - Same as the '-h' Input Switch
    alternate-smtp-host: (hostname or IPaddress)
        a failover hostname for if the default fails
    connection-timeout: (secs)
        Max time in seconds for the connection to the SMTP server
        The call is aborted if it takes longer than this.
        If the remote is across the computer room, this time can be
        reduced to 15 or 20 secs; if over the internet, 120 secs is the norm.
        default: 120 for 120 seconds, minimum is 10

    send-single-smtp: send smtp in one file per session which is then cut and
        reopened for the next file/session.
    hostname: (hostname to use)
        Cosmetic !
    timing-stats: yes/no
        Show timing stats if the incoming file has the HE/HR fiphdr
        default: no

    use-sx:
or  use-external-file:
        if there is an SX FipHdr field with a path to the data file, use that rather
than the data in the input file.

    wrap-subject-chr: (FipSeq chr)
        If the Subject field has this chr, wrap the line
        default: none
        This might have been added by ipxchg :
            z:convert-fiphdr-to-qp:DI:DJ:utf-8:\002
        wrap-subject-chr:\002
    force-subject-wrap: (No of chrs)
        Normally  subject lines at this total
        default: 76
        eg force-subject-wrap:76

    check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
        Use this when for POP3 accesses ONLY where ipsmtp on 2 systems is accessing
the same POP3 account(s) in a redundant way.
        It is used to decide whether the current host should be getting the files or
not.
        ie in the parameter file is
            check-primary-server:timersvr
        and in the DEST_REDUN is
            ; psuedohost    primary secondary
            timersvr    fip1    fip2
        and in the SYSTEM file for both fip1 AND fip2 there is  the same 'ipsmtp'
line with the same parameter file
        If fip1 is up,  'ipsmtp' on fip1 will always get while on fip2 it will just
check/loop.
        If fip1 is down 'ipsmtp' on fip2 will start processing the files.

    use-tls:yes/implicit/explicit/no
        The commends are for a ftp running over SSL/TLS on the remote server
        default is NO
        no      - normal, standard FTP on (normally) port 21 for the control
        yes or explicit - connect (normally) on port 110 in clear then use SSL for
USER, PASS and data
        implicit    - connect (normally) on port 995: use SSL for all conversations
    tls-auth: (XXX)
        AUTH type for TLS/SSL                default: TLS

    ssl-method: (1,2,3,23,999)
        Version number to use for TLS/SSL       default: 999 for current default (2 or 3)
    ssl-password: (password)
    ssl-passwd: (password)                  default: none
        Optional password if the handshake requires a shared secret
    ssl-cert: (name of a PEM certificate file)      default: none
    ssl-root-cert: (name of a root PEM certificate file)    default: none
        Optional certificates - held in tables/ssl

Then for EACH logon you wish to track INCOMING mail from the Internet specify a
line begining :
        logon: (logon name)             default: none
    or  unix: (unixlogonname)               default: none

    These two parameters are identical.
    There is also a means, for UnixBoxes only, to poll for ALL mail
    for ALL the members of a group (as in the /etc/group file) :
        group: (unixgroupname)              default: none

    Whether 'logon', 'unix' or 'group' there are other (optional)
    parameters:
        sendto:(newaddress)
        fiphdr:(FIP hdr sequence)
        striphdr:
        keephdr:
        pop3pwd: (Password for POP3)
    where   sendto allows you to specify another name for the DA field
            IPPOST will use this to route. By default
            the Fip Hdr field DA will hold the logon name.
        fiphdr allows you to add to the FIP hdr. This may be any
            - perhaps the DU field to change the destination.
        striphdr will strip the SendMail header except for the
            From and Subject lines
        pop3pwd specifies the password to use IF you are accessing
            a POP3 server to receive incoming mail.  It is NOT
            required for UnixBoxes scanning local mail queues.

To add a new logon :
    1. Read the 'passwd' file for the exact Unixlogon name - eg newboy
    2. Enter this name in SMTP file as :
        logon:newboy
    3. If this is the same name as in file ATXMSG, enter line
        newboy  edone
        where edone is the name of your editorial system
    4. If 'newboy' is really called 'dirtyoldman' on the 'atex'
        editorial system, the entries should be:
        SMTP file - logon:newboy    sendto:dirtyoldman
        ATXMSG file -   dirtyoldman atex

To bringup IPSMTP :

    1. Create logons as above
    2. Check you have destinations in the USERS file :
    FIPMSG=     DP:fipmain  DQ:post  DC:atexeol
    MSGFORATX_EDSYS=    DP:edsys3   DQ:atexq-grp    DC:atexeol
    where 'fipmain' is the name of this UnixBox.
    Destination FIPMSG is generated by IPSMTP.
    Destination MSGFORATX_* is generated by IPPOST IF the name is in the FIPMSG
file.
    eg if the Editorial System is called ATEX (environment variable FIP_EDSYS) and
teh queue required is email-wir on system3 :
    MSGFORATX_ATEX= DP:atex3    DQ:email-wir    DC:atexeol
    3. Input Queue spool/2smtp needs to be added
    4. IPPOST needs to be running too (unless you change the FIPMSG DP/DQ to
somewhere else)
    5. Outbound Format is ATXMSG for IPGTWY/IPOUT/IP2WTR etc.
        Please check if this exists and has the correct setup

The current version can scan up to 1000 Unixmail boxes.

Where sections of FipHdr fields are required or changes to the output style,
use
 keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysA
dmin manual for more information).

    They are normally specified :
        fixed:QZ    1234543
        partial:QT  ST,3,2,U,<,>
        combie:QY   ep|na,(0000000)a
        option:QE   ep,11,7,s
        repeat:QK   XK,-,3
    or  repeat:QP   PK,,4,#X
        style:QS    XN,%.03d
        replace:QN  NN  abc=DEF def=GHI

For outbound traffic, FipHdr fields used are :
Mandatory
    DA  holds one or more remote email addresses
        multiple addresses are separated by a comma.
        Can be overridden by 'newDA'
        Can be changed by 'forcedomain'
Optional
    SA  holds the sender's email address
        can be overridden by 'forcesender' (actual address)
        can be overridden by 'forcesender' (from field)
        can be defaulted by 'defsender'
    DI  holds the Subject: field
    DJ  holds the From: field
    ZSMTP_BINARY    for sending Binary files
    ZSMTP_ENCODE    holds the Content-Transfer-Encoding
        default to  Content-Transfer-Encoding:quotedprintable
    ZSMTP_TYPE  holds the Content-Type. no default
    ZSMTP_LOG   will send a log of the SMTP transmission to this
        destination (for SMTP outbound traffic using the '-h' input
        switch only)
    ZSMTP_HOST IPaddress or hostname of the mail server if NOT the default
            (for SMTP only).

Input Switches
--------------

Input parameters are : all optional :
    -1 : Get once only then stop for POP3 or Receive only.  default: spool
        Either -p (path) or -3 (pop3 server) must be used with this option
        This also incorporates -Q so no queu is scanned and no mail sent.
    -3 : Hostname where the pop3 daemon is running. default: none
        Note -p and -3 are mutually exclusive.
    -a : queue for non-text attachments     default: none
        default is that attachments are passed through in text
    -A : destination for non-text attachments   default: none
        default is that attachments are passed through in text
    -C : do NOT convert unprintables to "=XX" format  default: do
    -d : do NOT delete each unix msg        default: delete
    -D : display the conversation with the remote server    default: no
        valid ONLY with the -1 for single shot; used for debugging troublesome
connections
    -f : do NOT output the original filename as the first
        line of text                default: output name
    -F : do NOT output the original sending system (SU) as the From
        field                   default: From: SU
    -h : Hostname where the smtp daemon is running. default: that system
    -H : do NOT add an extra Mime Headers as the file
        is in the correct format    default: add To/From/Subject
    -i : Receive spooled queue if you are using 'sffmail'   default: using Path
    -k : alternate host name                default: none
        use this for checking primary/secondary for clustered systems where the
actual hostname differs from the cluster name
        ie if the host name may be fipcluster-a or fipcluster-b but there is a
cluster hostname of fipcluster
    -K : quiet mode - message if NOT the primary system default: do not message
    -l : do NOT log each msg in or out      default: log
    -m : (ipchkmail folder name)            default: process inside ipsmtp
        Do not convert incoming SMTP and POP3 files immediately but pass them to
ipchkmail running on this folder which does a better job! especially for
attachments-within-attachments

    -o : Next fip queue for incoming files      default: spool/2go
    -O : Done queue for spooled files out       default: none
    -p : Path on this system holding mailboxes  default: depends on the OS
        Unix only. The full pathname MUST be stated.
        Note -p and -3 are mutually exclusive.
    -q : Send Queue to spool            default: spool/2smtp
    -Q : Do NOT send any mail and therefore do NOT scan any input queue
                            default: scan
    -r : eolns for the data part should be CR NL    default: just NL
    -R : do NOT resolve hostnames      default: do
        This will resolve any hostnames in the DEST_REDUN file (if there is one)
    -S : do NOT split files on the "From ..." line    default: do
    -t : scan time of queue             default: 3 secs
    -T : log the trace of each transaction      default: do not
        Normally a one line log of each file sent is stored in
        log/remote_trace with a name of 'date_Address'.
    -w : for file->Sendmail, set line length.    default: 76 + translate
        and do NOT translate unprintables to "=XX" format
        To NOT wrap text and no translations -w 0
        This can be ignored for a single file if it has
        the FipHdr field ZSMTP_BINARY:
    -W : for files outbound, set JUST the line length   def: 76 + translate
        but still translate unprintables.
    -x : Offline queue              default: none
        Normally ipsmtp will retry - and possible delay files also queued up - any
message that has not been delivered.
        Use this to shuffle files that error to a different folder where another
ipsmtp will take care of the baduns
    -X : Send directly to the Email server of each destination  default: no, use a
single relay server
        This depends on the resolv.conf or Windows DNS to be setup correctly of
course.
    -z : parameter file             default: mail/SMTP
    -v : display version number and exit.

Now a statement of sanity (?) about incoming mail from the Internet :

If like most of the civilised world you prefer to keep well away from anything
'sendmail' can throw at you, then let ipsmtp just grab its files from the
sendmail internal queue. Or even better use POP3 and forget about it all - no
setup, no nothing extra!

If you are unlucky enough to have a Sendmail specialist on your staff, before
sending for the men in white coats, point out there is a '-p path for unix
mailboxes' switch. Your guru should dictate (they always dictate - though they
claim they are working with your best interests in mind) where he is willing to
drop files for those logons you are interested in and add the -p switch to the
tables/sys/SYSTEM file. Easy huh !

Note there was a subtle change at version 15 which changed the default from
ATXMSG to FIPMSG.

Version Control
;16v20  09nov01 cleanup after failed connections better
    ;a 16mar02 added remote_trace and -O done queue
    ;b-c 03jun03 cleanup line-enders in mimehdr and streaming for smtp
        and added ZSMTP_HOST
    ;d-i 22sep03 added send-extra-fiphdr: and stream-connections:yes
    ;j-l 29jun04 added -1 and speedy
    ;m-p 17sep05 added alternate-smtp-host and hostname: and log-line
    ;q 10aug06 added remote_trace style timings to send file done msg as H4
    ;r-t 21sep06 allow quotes in email address and wrap long subject lines
    ;u 13apr07 added extra-mime-headers:
    ;v4 06feb08 added hash-in-address ;2 modette for size of email list
        ;3-4 added check-primary-host for POP3 and chkmail-folder:
        ;5-6 bugette with send_smtp and logging long DA
        ;7 3sep10 added use-sx
        ;8-9 10nov10 bug in pop3 when Uidl has non-alnum
        ;10-11 19may11 bugette when TLS does not work and b64 '=' padding
        ;12 22aug13 added POP3 error message on logon/pwd and added -1 for display
        ;13-14 30sep13 added hash-in-fiphdr ;15 30apr14 file-trace
        ;16  7oct14 if -H and not -C, do not encode any mime headers in the data
        ;17-18 26jun15 make donque variable
        ;19-20 25jul16 tuning and buffersizes and better error messages

(copyright) 2017 and previous years FingerPost Ltd.