ipchkmail

ipchkmail

This program spools spool/xsmtp for files in MIME format left by
Sendmail/sffmail.

If can be used to :
    - check the Sender is valid for sending to that destination by checking
against a list of names in tables/mail/check.
    - zap or reroute any mail without either a 'From' or a 'Reply-to' field.
    - check and/or split off and/or convert attachments

Normal running is to force all attachments into files and route them accoring
to their type - xls, jpeg, xml etc

It uses a parameter file, by default tables/mail/CHKMAIL. The format of this
file is :
    ; comment line
    strip-mime-header: Strip the Mime Header of incoming files. default: no
        strip-mime-header:yes
        strip-mime-header:all   these two will always strip all mimes
        strip-mime-header:no    this will leave all mime lines
        strip-mime-header:some  this will check for 'ignore' see below
    defdest: (default Fip Destination - DU FipHdr field)    default: FIPMSG
    chrset: (Source character set)              default: email
        ie this provides a different FipHdr SC field to SC:email
    extra-fiphdr: (Extra Fip Hdr information to add to each file) def: none
    no-from-message:
    invalid-sender-message:
    check-sender: if there is a file in tables/mail/check, make sure the
        'From' or 'Reply-to' address is included.
    fiphdr-to-field: (2 letter FipHdr field)

    fiphdr: (Map incoming mail field to a Fip Hdr field)    default: none
        fiphdr:DI   Subject
        fiphdr:AZ   Message-Id
    convert-encoded-words: (8bit/utf8/no)       default: utf8
        Mime fields such as Subject are often in a format called 'encoded words'
            =?iso-8859-1?q?this=20is=20some=20text?=
        The DATA of the encoded word will usually be converted to UTF8 or 8bit
    ignore: (Mail header field to always ignore)    default: none
        Used with 'strip-mime-header:some' (see above) to filter out or delete
selected mime header lines.
        ignore: Status
        ignore: Apparently-To
        ignore: X-Msmail-Priority
    fiphdr-TO-field: (FipHdr)
        Use this where you already have the real TO address in a fipHdr field. eg if
you are running sffmail in
        /etc/aliases :   donkeyrides: "|/fip/bin/sffmail -hJA:DKR"
        then use    fiphdr-TO-field:JA
        to capture all DKR traffic no matter what is in the To field.
    max-TO-length: Set the Maximum length of the 'To:' field to this.
        default: 400 chrs.
        Listservers often give tens/hundreds of names in the 'To:' field. This chops
to the first few.
    max-single-fiphdr-size: (number > 400)       default: 8191 chrs (from 2017 - was
4095 and pre 2015, 1023)
        This does NOT override the max-TO-length parameter.
        This overrides the -H input switch
    send-all-with-attachments: (DU)
    queue-all-with-attachments: (folder name)
        Fip Destination and folder name to send any file (complete, unaltered) with
one or more attachments.
        If there is no "queue-..", the default output queue is used.
        This is incompatible with decode, -a/-A and any other attachment processing.
    decode:
    unencode: Decode any base64 sections.   default: no
        decode:     This will NOT decode attachments
        decode:text This will NOT decode attachments
        decode:all  This will decode body AND attachments
        A second parameter may be added 'utf8' which will decode the result to UTF8
        eg  decode:text,utf8
    split-every-attachment: (yes/no)
        Normally any attachment that is has a Content-Type of 'text' is left in the
main message.
        Use this keyword to split ALL attachments and either leave in the folder (-a)
or sent to the destination (-A).
    att-message: Message to replace an attachment with.
        The attachment is called fipHdr DN.
        att-message:<a href="readdisboyo.pl?namo=\\DN>Read this Attachment
\\DN</a><br>
        default:** Attachment has been Stripped, pls contact System Managers for :
\\DN
    alt-html-inline: (yes/no)
        show the Alternate HTML version inline or NOT   default: yes
    add-fiphdr-on-atts:yes/no
        add the whole fiphdr on each attachment (-A only)
    att-data-only:
        do NOT add any mimeheaders on the top of the attachment (-A only)
    fiphdr-att-type: (XX)
    fiphdr-att-chrset: (XX)
    fiphdr-att-name: (XX)
    fiphdr-att-encoding: (XX)
    fiphdr-att-bytes: (XX)
    fiphdr-att-total: (XX)
    fiphdr-att-level: (2 chr FipHdr code to replace PL in start/end-att-marker
    fiphdr-att-inline: (2 chr FipHdr code to replace PI in start/end-att-marker
        2 letter FipHdr fields for the Content-Type, Charset and Filename of any
attachments
        plus level (0 is top, 1++ is embedded attachements)
        plus total :    for the Top, this = Number of attachments
                for each attachment, its sequence number
        plus bytes :    no of bytes/size of file
        no defaults, except for PL and PI
        Note that total bytes for an attachment is only non-zero for END-att-marker
        eg
        ; for each Att, save the type, chrset and name in temp FipHdr fields
        fiphdr-att-type:D1
        fiphdr-att-chrset:D2
        fiphdr-att-name:D3
        ; D4 seqno
        fiphdr-att-total:D4
        ; D5 total bytes of ATT or Top - only available for END-att-marker
        fiphdr-att-bytes:D5
        ; D6 reserved in fip_mail_package.pl
        ; D7 content-encoding
        fiphdr-att-encoding:D7
        ; for each Att, top and tail with a marker - if NOT inline
        ; PI says whether att is inline or not
        option:VN   PI,,,,NO
        start-att-marker:\n
        end-att-marker:\n\VN**FIP-ATT**|\PL|\PI|\D1|\D2|\D3|\D4|\D5|\$o
        att-message:\VN\DN\$o\n

    start-att-marker: (FipSeq)
        Extra Data to insert into the main data file at the point where an
attachement starts
        extra FipHdr fields at this point are  PL : level, PI : inline YES or NO
            Plus the 3 fiphdr-att-* fields if specified
        option:VY   PI,,,,YES
        start-att-marker:\VYInLine attachment starts here ->\n\$o
        no defaults
    end-att-marker: (FipSeq)
        Extra Data to insert into the main data file at the point where an
attachement ends
        PL : level, PI : inline YES or NO or IGNORE (not inline and no file)
        option:VN   PI,,,,NO
        end-att-marker:\VNEndOfStrippedAttachment\n\$o
        no defaults
    before: (FipSeq)
    after: (FipSeq)
        Extra Data to insert into the main data file at the top or tail
        no defaults

    preserve-existing-filename:
    preserve-existing-source:
    ignore-existing-filename:
    ignore-existing-source:
        These two will ignore either the FipHdr field SN or SU for name or source
from the raw file and add new ones.
        The default is to preserve the source but ignore the filename!

    process-if-no-check-file:yes/no
        If you are using Check files and there is no check file for that address, the
default is to SEND. Use the following to NOT send:
            process-if-no-check-file:no
    newDA: FipHdr field to use in place of DA for the 'To:' field
        If you are using smtpwire, DA will already be filled in correctly.
        So map it to another fipHdr newDA:RA
    newDI: FipHdr field to use in place of DI for the 'Subject:' field
        If you are erworking the Subject line - forcing it to QuotePrintable for
example,
        map it to another fipHdr    newDI:RI
    hdr-hash:(FipSeq)
        replacement chr for a hash '#' in a Fiphdr field
        default is '-'
    script: (FipSeq script and Filename)
        Script to run against the MAIN, top file (ie NOT the attachments)
        E1 is the temp file name with FipHdr and Data
        E2 WILL BE the output filename if needed
        no default
        eg  script:/fip/local/list_attachments.php \E1
    round-robin: (number)           default: none
    round-robin-fiphdr: (2 letter FipHdr field) default: none
        Round-Robin the output files and add the RR number to the fipHdr.
        Both parameters are required - the Number is the MAXimum.
        eg to leave the output in folder1 to folder9
            round-robin:9
            round-robin-fiphdr:RR
        and a suitable output folder might be   -o avcheck\RR
        Note that the round-robin number is NOT added automatically to any output
folder - ie you MUST specify a FipHdr as in /fip/spool/2xml\RR
    att-queue: (FipSeq) same as '-a' input switch which it overwrites   default:
none
    balance-queue: (folder under spool)                 default: 2balance
    done-queue: (FipSeq) same as '-d' input switch which it overwrites  default:
none
    output-queue: (FipSeq) same as '-o' input switch which it overwrites    default:
2go
    use-sx:no/yes
or  use-external-file:no/yes
        if there is an SX FipHdr field with a path to the data file, use that rather
than the data in the input file.
        default is check for SX: and use it exists

-- Check files --
There needs to be a check file for EACH address you want checking. The filename
is just the NAME (not the Domain) forced uppercase :
    eg  if the TO address is    billy@never.inhale.com
        the check file is   'tables/mail/check/BILLY'

The entries in the check file are the sender's address. ie the FROM field.

The format of a check file is
    ; is a comment line
    ; there can be one or many entries, each on a separate line
    ; for a single sender
    name@domain
    billyGoat@bridge.to.far.com
    ; for a complete domain - prefix a '*'
    *@completely_idiotic.com

For Example
;
; Control files to NationalDesk
;
; allow all internal users
*@tuebrook-bugle.com
; Star reporter
famousJock@ego.inflate.com
; the editor
S.Leaze@weasel.org

-- Blacklists --

A single BLACKLIST file can be added to ignore files FROM certain addresses
and/or domains.

It is the same syntax as the Check files (see above) but is in the main folder
tables/mail/BLACKLIST.

Note that you can block all domains for a name with a wild card too :

Undisclosed.Recipients@*

-- Conversions --

; What conversions do we want to do (double quotes to keep the spaces)
convert:binhex script:"/usr/forum/bin/alchemy \QN \QN.nohex -Q -o -e --."

-- FipHdr fields --

Note the following FipHdr fields are filled in for each incoming file (in
addition to any 'fiphdr' parameters)
    SA - From address
    DA - To address (can be changed using the 'newDA' parameter)
    DI - Subject field (can be changed using the 'newDI' parameter)
If the contents of these fields are a different charset, it is ignoted and
converted to the Binary chrs
However the 'fiphdr:' fields are preserved without conversion.

Note that FipHdr fields are used for conversions:
    QN - Full Path and Filename of the input file to be converted
    QR - Path or queue for converted files and files to convert
        which defaults to /fip/spool/epsf
    SN - name of the input file to the program !
    QI - name of the scrolled queue

Input parameters are (all optional) :
    -a : Folder to put attachments we have stripped     default: att left in data
        it can be FipSeq : -a /fip/data/raw.data/\$e\$y\$i\$d_attachments
    -A : Destination (FipHdr DU) to send all the
        attachments we have stripped.   default: att left in data
    -d : done queue for incoming        default: none
        this will leave the raw, incoming file in this done folder.
        it can be FipSeq : -d /fip/data/raw.data/\$e\$y\$i\$d_chkmail_done
    -H : max size of any FipHdr field   default :1023
    -i : input queue to scan        default: spool/xsmtp
    -o : output queue for processed files   default: spool/2go
        This can be FipSeq - eg -o avcheck\RR
    -l : do NOT log incoming files      default: log
    -w : file wait interval in seconds  default: no wait
        set this for nfs or slow systems eg -w 10
    -z : name of parameter file if not  default: tables/mail/CHKMAIL
        the default.
    -T : do NOT attempt to clean up the DU  default: do
        only letters, numbers '_' and '-' are allowed.
    -v : print version no and exit

Version Control
;1w43   20jun03 added fiphdr-to-field, process-if-no-check-file
    ;b 26jul03 bugette - dbl ATT on attachment
    ;c-g 30jul03 bugette - decode NOT always working correctly plus newDA added
    ;h 31oct03 timings
    ;i-j 05dec03 added add-fiphdr-on-atts
    ;k-m 30mar04 bugette when hacker sends > 64000 on one line of header
    ;n-o 03sep04 added check_for_fiphdr for attachments too
    ;p-q 20sep04 speedy
    ;r 09feb05 Imail has invalid Content-type
    ;s-t 31oct05 bugette - need to be able to handle blank lines with spaces in
the mime header
    ;u-v1 21mar07 bugette with attachments and long TO: addresses
    ;w1-9 10dec08 attachments inside attachments - major mods ! added -1 for
single ;7 bugette without -a -A ;8-9 newDI added
        ;10-11 added unicode mapping ;12 preserve att fiphdr better
        ;13-18 put inline TEXT back in.. plus allow '+' in DU
        ;19-24 tuning strip-mime-hdr to properly zap all the mimes of multi-level
atts and hdr-hash added
        ;25 2jan2013 de-dup added ;26 1oct13 bugette in dedupbalance ;27 donque move
not link
        ;28 30nov14 trimmed extrafiphdr to 8k
        ;29  8jun15 added strparse to -o outque and round-robin
        ;30-31 29oct15 woops - allow multiline Subject lines ....
        ;32 1sep17 max size of a FipHdr field is now  FIP_limitHdrFieldSize (HDR-STD)
        ;33 2nov17 bugette - att filename had a extra .out suffix
        ;34-36 15feb18 bugette internal may cause dump!
        ;37-38 28may20 done-que and att-queue added as parameters and att-queue is
now parsed and check for SX
        ;39 25nov20 added fiphdr-att-encoding
        ;40-42 21sep22 zap SX on output once used
        ;43 5feb24 moved fiphdr-to-field to AFTER other FipHdr processing
;000z   21jan99 original version
    ;a/b/c/d  3jun99 small mods
    ;e 17feb00 allow fiphdr on incoming file.
    ;f 06mar00  maxTOlength of 400 chrs
    ;g/h 10mar00 added decode and better Dispo-inline handlng
    ;i 10apr00  mod for sending any file with an attachment to another DU
            NOTE need to WINNT DU and DUallAtts **
    ;j/k 14apr00 woops make max len of a FipHdr field. (k) and filename
    ;m/n 09may00 small mods and bugette in do_decode
    ;o/p/q 05jun01 check_sender better; added Blacklist, strip leading NULS/SPCs
    ;r 03aug01 always check for HH fields..
    ;s 29oct01 WINNT filename needed better validation
    ;t 24feb02 added split-every-attachment
    ;u 04jun02 bugette in boundaries etc
    ;v/w/x 13jun02 if there is an SU already - use it
    ;y 03oct02 bugettes - where the data does not contain a single CR or NL
    ;z 23apr03 decode Headers too

(copyright) 2024 and previous years FingerPost Ltd.