imapwire

imapwire

This periodically attachs to, checks for and grabs new articles in a mailbox on
a remote IMAP server.

Nowadays OAUTH2 is the norm and imapwiressl can be used to access Gmail and
Office 365 accounts with Oauth authentication.

It is normally started by 'iptimer' with FipSeq for the mailbox name, password
etc

The whole document is then left, normally, in spool/xsmtp for 'ipchkmail' to
pull apart the Header etc.

The parameter file, normally tables/wire/IMAP, is read for the the names of the
mailbox to scan.
    ; comment

    mailbox: (mailbox name on the remote IMAP server)

        password:(FipSeq/in plain)
            no default
        delete:(yes/no)
            Delete files that have been grabbed - and have the Ok-to-delete flag set (ie
old)
            This is the IMAP Expunge command for the mailbox.
            default is NO
        sendto:(newaddress)
            sendto allows you to specify another name for the DA field
            IPPOST will use this to route. By default
            the Fip Hdr field DA will hold the logon name.
        fiphdr:(FipSeq)
            Add to the FIP hdr - perhaps the DU field to change the destination.
            default: none
        inbox: (inbox name)
            default INBOX

    eg
    mailbox:chris   password:zongle fiphdr:#XX:here delete:yes

Optional keywords / parameters :

    grab-every:(seconds)
        Connect, logon and check for news every X seconds.
        The default is 600 seconds (5 mins) while the minimum
        is 5 seconds.
        The '-t' input switch can also be used.

    defdest: (default Fip Destination (DU FipHdr field) default: "imap"
    chrset: (Source character set ie SC header field)   default: ascii
    imap-host: (hostname or IP address of the host to attach to) nodefault
        (see also -s input switch)
    imap-port: (Port number of the host)            default: 143
        Unless use-tls is set where the default is port 993
        (see also -p input switch)
    connection-timeout: (timeout in seconds wanting to connect to the remote)
default: 120 secs
    connection-retries: (no of connection attempts before erroring      default: 5
    response-timeout: (timeout in seconds wanting for the remote to respond to a
command)    default: 60 secs

    extra-fiphdr: (more FipHdr information to add)      default: none
    archive: yes/no                     default: yes
        Archive the data in log/data
        This parameter will override the -Z switch of that is also specified
    skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute
        the skip file when changed (see doc on 'ipbalan') - for ipftp and webwire.
        This is often used where a second system could be used as a redundant server
        if the main system fails. (see also -B input switch)
    skip-balance-queue: name of queue under /fip/spool  default 2balance
    proxy-server: If using a proxy, these are the name and port to aim at.
    proxy-port:
    proxy-logon: This is the logon and password to get thru the firewall
        if required. The format is (logon) (colon) (password) and is
        converted to base 64.
        proxy-logon:Y2hyaXMuaHVnaGpvbmVzOnBhbnRoZXIK=

        To generate :
            echo -n "logon:password" | sffb64 -i
        eg  echo -n "chris:sleekpanther" | sffb64 -i
        gives   Y2hyaXM6c2xlZWtwYW50aGVy
            proxy-logon:Y2hyaXM6c2xlZWtwYW50aGVy=
    proxy-is-squid:yes/no   Is the proxy a Squid ?  default: no
        For Proxies - Please see note below

    use-oauth:yes/no
        Use OAUTH to grab/use an access-token or Bearer token eg for Gmail access
        default is NO
    use-ssl:yes/implicit/explicit/no
    use-tls:yes/implicit/explicit/no
        The commends are for a ftp running over SSL/TLS on the remote server
        default is NO
        no      - normal, standard FTP on (normally) port 21 for the control
        yes or explicit - connect (normally) on port 110 in clear then use SSL for
USER, PASS and data
        implicit    - connect (normally) on port 993: use SSL for all conversations
    tls-auth: (XXX)
        AUTH type for TLS/SSL               default: TLS

    ssl-method: (1,2,3,23,999)
        Version number to use for TLS/SSL       default: 999 for current default (2 or 3)
    ssl-password: (password)
    ssl-passwd: (password)              default: none
        Optional password if the handshake requires a shared secret
    ssl-key: (name of a certiticate key file)       default: none
    ssl-cert: (name of a certificate file)    default: none
    ssl-root-cert: (name of a root PEM certificate file)    defaunt: none
        Optional certificates are in tables/ssl unless name starts with '/'
    ssl-verify: yes/no  verify server certificates  default: yes
    ssl-ciphers: (list) acceptable ciphers
        (use 'openssl ciphers' to list)
        default:  "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS"
        (from feb2021
ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1:!AESCCM
    ssl-display: yes/no display SSL connection details  default: no

    output-folder: (folder name)
    output-folder1: (folder name)
    ..
    output-folder9: (folder name)
        if the folder does NOT start with a '/', it is assumed to be
        Note these override the default and '-o' input switch..

-- For accessing Oauth protected assets

    ; We need an access token
    use-oauth:yes

    ; which flavour of Oauth2 ? - only the first letter is meaningful
    ; oauth-flavour: Google (Gmail) or Microsoft (Office365)
    oauth-flavour:microsoft for office 365

    ; Current token file will be saved in /fip/fix/goauth2
    oauth-token-file:\OT

    ; Credentials file in /fip/tables/cert
    oauth-credentials-file:\OC

    ; sffoauth and imapwire
    oauth-scope:https://outlook.office365.com/.default

    ; Script to run when token expires - approximately every 12 hours
    oauth-refresh-script: (Script in FipSeq)        script to generate the access_token
using a refresh_token
    oauth-refresh-script:/fip/bin/sffoauth -z wire/IMAP.O365.OAUTH.SEA -c \OC -t
\OT -H '#WN:\WN' -a

    These 5 FipHdrs are use to generate, check, add/renew permissions to access
the remote data - normally Gmail or Office365

    oauth-client-fiphdr: (FipHdr)   default: IC
    oauth-secret-fiphdr: (FipHdr)   default: IS
    oauth-access-fiphdr: (FipHdr)   default: IA
    oauth-refresh-fiphdr: (FipHdr)  default: IR
    oauth-expiry-fiphdr: (FipHdr)   default: IX
-- Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).

    They are normally specified :
        fixed:QZ    1234543
        partial:QT  ST,3,2,U,<,>
        combie:QY   ep|na,(0000000)a
        option:QE   ep,11,7,s
        repeat:QK   XK,-,3
    or  repeat:QP   PK,,4,#X
        style:QS    XN,%.03d
        replace:QN  NN  abc=DEF def=GHI
        newdate:QT  hours+3 "\ZD"

--- Gmail using Oauth and IMAP

--- Gmail using SSL and IMAP

Generally this is being phased out for Oauth - see above

To access a Gmail account :
** beforehand, you must logon to the Gmail account
    - select settings
    - click on Forwarding and POP3/IMAP
    - select enable IMAP
    -( select Auto expunge OFF if you have more than one person/program accessing
!)

- you must use imapwiressl

- add the following to the parameter file

; Use TLS
tls-auth:ssl
use-tls:implicit

; Imap Host
imap-host:imap.gmail.com
imap-port:993

--- Using TIMER to kick off
Easy !

1. wire/IMAP.FIP

; Use FipSeq for the attributes
; If using W4 or Prestige, copy WN to another FipHdr field - RU in this case
mailbox:\WN password:\W7 fiphdr:\W3#RU:\WN#

; hostname of the exchange server
imap-host:(hostname here)

; If you have more than one fip - make sure the other system is up-to-date
; ** Add balskips group in sys/BALANCE - for each host: group:balskips
host:(hostname) ignore-localhost: nofiphdr:
skip-balance-group:balskips

; ---------------
2. setup/TIMER_IMAP.FIP

; If you have more than one fip and are running Primary/Secondary :
; ** Add wiresvr to sys/DEST_REDUN (or use an existing entry)
check-primary-server:wiresvr

group:imap
track-status:no
bandwidth-stats:no

; If you have more than one fip - make sure the other system is up-to-date
; ** Add balskips group in sys/BALANCE - for each host: group:balskips
host:(hostname) ignore-localhost: nofiphdr:
skip-balance-group:balskips

; Then for EACH mail address - add this line - emailaddress does not NORMALLY
need a domain
client:(emailaddress)   type:imap fiphdr:XX:extraStuff days:X every:1
passwd:(password)

; ---------------
3. sys/SYSTEM

; add the line ...
imap    mail    iptimer -n timer_imap.fip

; ---------------
4. check sys/BALANCE and sys/DEST_REDUN as above

--- Testing

If things do NOT look like they are working, you can run imapwire manually with
the -1 and -D to run once and display the handshake.

So if the line in the SYSTEM file is
    imap    wires   imapwire -s mail.bignastycorp.com -n imap.fip
.. You can test from a terminal/CMD with
    imapwire -s mail.bignastycorp.com -n imap.fip -1 -D
or if using ssl
    imapwiressl -s mail.bignastycorp.com -n imap.fip -1 -D

To test AND GRAB NOTHING, add the -V switch too
    imapwiressl -s mail.bignastycorp.com -n imap.fip -1 -D -V

--- Note Imapwire saves the last item, date and time and UID in a file for each
mailbox in /fip/fix/imapwire

The three items are editable on 3 lines, so you can mess around at your peril
if you need !

::::::::::::::
imap_mail.zingle.com_fip$2011%hoho_inbox
::::::::::::::
408
14-Oct-2011
174223

--- Input switches are :
Mandatory :
    -s : Hostname where the IMAP is running.    default: none
Optional :
    -1 : one single pass and then stop      default: continuous
    -B : default balance group for skip files   default: none
        (see skip-balance-group parameter)
    -d : display the conversation with the remote server    default: no
        and pause between files for you to hit return to continue
        valid ONLY with the -1 for single shot; used for debugging troublesome
connections
    -D : display the conversation with the remote server    default: no
    -h : extra FipHdr information           default: none
        This is in FipSeq and should normally be quoted
        Note this is the means that 'iptimer' sends variable information to imapwire
        eg : -h"SN:hello#TC:200401031"
    -k : on Display, send a NOOP instead of a CAPABILITY before LOGON
                            default: send CAPABILITY
    -K : do NOT send anything before LOGON      default: send CAPABILITY
    -l : do NOT log anything except errors      default: log files only
    -L : log every file and every connection    default: log files only
    -n : name of the service            def: name of the parameter file
    -o : Next fip queue for incoming files      default: spool/2go
    -p : port number on the remote host     default: 143
    -t : sleep in seconds between connections/accesses  default: 600 secs
    -V : do NOT grab any files - used with -D -1 to test only   default: run and
grab
    -U : restart on this UID            default: use last saved in the fix file
    -x : Proxy server host or IP address        default: none
    -X : Proxy server port              default: 80
    -y : Proxy logon                default: none
    -Y : Proxy server is Squid          default: no
    -z : parameter file             default: wire/IMAP
    -Z : do NOT archive any incoming files    default: archive
    -v : display version number and exit.

Version Control
;02a-o2 16jun18 fipseq_extras ;b ;c extra SSL details added ;d added defUseSSL
to reset on connect ;e socks4/5 added
        ;f 1mar22 added oauth2 for gmail plus fixed bug when > 64k of ids on SEARCH
plus TLS remhost added
        ;g-h 8jul22 DU and SC are notw FipSeq
        ;i 10nov22 added -Z archive:no
        ;j-m 20dec22 oauth for MS-Office365 and check expiry too ;n connection
timeout for thru the proxy
        ;o 26jun23 ssl-ciphers added and tuned display
;01z    31dec08 cleanups ;f note_balance_action ;g-h 16feb11 added TLS
        ;i-j 31oct11 make sure date is valid and better error msgs
        ;k-m 02apr12 bugette with tls ;m unlink tmp on singleshot
        ;n 21jul12 added output-folderX
        ;o-s 15oct12 bugette - missed first file if folder is reset (or zapped and
remade)
        ;t-u 26may13 added expunge (finally) ;v added 993 as default for use-tls
        ;wx 7mar17 send CAPABILITY before LOGIN so we can check if LOGINDISABLED if
-D !
        ;y 10apr17 made logon,password etc parseable
        ;z 16jun17 more logging

(copyright) 2024 and previous years FingerPost Ltd.