imapwire
imapwire
This periodically attachs to, checks for and grabs new articles in a mailbox on
a remote IMAP server.
Nowadays OAUTH2 is the norm and imapwiressl can be used to access Gmail and
Office 365 accounts with Oauth authentication.
It is normally started by 'iptimer' with FipSeq for the mailbox name, password
etc
The whole document is then left, normally, in spool/xsmtp for 'ipchkmail' to
pull apart the Header etc.
The parameter file, normally tables/wire/IMAP, is read for the the names of the
mailbox to scan.
; comment
mailbox: (mailbox name on the remote IMAP server)
password:(FipSeq/in plain)
no default
delete:(yes/no)
Delete files that have been grabbed - and have the Ok-to-delete flag set (ie
old)
This is the IMAP Expunge command for the mailbox.
default is NO
sendto:(newaddress)
sendto allows you to specify another name for the DA field
IPPOST will use this to route. By default
the Fip Hdr field DA will hold the logon name.
fiphdr:(FipSeq)
Add to the FIP hdr - perhaps the DU field to change the destination.
default: none
inbox: (inbox name)
default INBOX
eg
mailbox:chris password:zongle fiphdr:#XX:here delete:yes
Optional keywords / parameters :
grab-every:(seconds)
Connect, logon and check for news every X seconds.
The default is 600 seconds (5 mins) while the minimum
is 5 seconds.
The '-t' input switch can also be used.
defdest: (default Fip Destination (DU FipHdr field) default: "imap"
chrset: (Source character set ie SC header field) default: ascii
imap-host: (hostname or IP address of the host to attach to) nodefault
(see also -s input switch)
imap-port: (Port number of the host) default: 143
Unless use-tls is set where the default is port 993
(see also -p input switch)
connection-timeout: (timeout in seconds wanting to connect to the remote)
default: 120 secs
connection-retries: (no of connection attempts before erroring default: 5
response-timeout: (timeout in seconds wanting for the remote to respond to a
command) default: 60 secs
extra-fiphdr: (more FipHdr information to add) default: none
archive: yes/no default: yes
Archive the data in log/data
This parameter will override the -Z switch of that is also specified
skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute
the skip file when changed (see doc on 'ipbalan') - for ipftp and webwire.
This is often used where a second system could be used as a redundant server
if the main system fails. (see also -B input switch)
skip-balance-queue: name of queue under /fip/spool default 2balance
proxy-server: If using a proxy, these are the name and port to aim at.
proxy-port:
proxy-logon: This is the logon and password to get thru the firewall
if required. The format is (logon) (colon) (password) and is
converted to base 64.
proxy-logon:Y2hyaXMuaHVnaGpvbmVzOnBhbnRoZXIK=
To generate :
echo -n "logon:password" | sffb64 -i
eg echo -n "chris:sleekpanther" | sffb64 -i
gives Y2hyaXM6c2xlZWtwYW50aGVy
proxy-logon:Y2hyaXM6c2xlZWtwYW50aGVy=
proxy-is-squid:yes/no Is the proxy a Squid ? default: no
For Proxies - Please see note below
use-oauth:yes/no
Use OAUTH to grab/use an access-token or Bearer token eg for Gmail access
default is NO
use-ssl:yes/implicit/explicit/no
use-tls:yes/implicit/explicit/no
The commends are for a ftp running over SSL/TLS on the remote server
default is NO
no - normal, standard FTP on (normally) port 21 for the control
yes or explicit - connect (normally) on port 110 in clear then use SSL for
USER, PASS and data
implicit - connect (normally) on port 993: use SSL for all conversations
tls-auth: (XXX)
AUTH type for TLS/SSL default: TLS
ssl-method: (1,2,3,23,999)
Version number to use for TLS/SSL default: 999 for current default (2 or 3)
ssl-password: (password)
ssl-passwd: (password) default: none
Optional password if the handshake requires a shared secret
ssl-key: (name of a certiticate key file) default: none
ssl-cert: (name of a certificate file) default: none
ssl-root-cert: (name of a root PEM certificate file) defaunt: none
Optional certificates are in tables/ssl unless name starts with '/'
ssl-verify: yes/no verify server certificates default: yes
ssl-ciphers: (list) acceptable ciphers
(use 'openssl ciphers' to list)
default: "HIGH:!aNULL:!kRSA:!SRP:!PSK:!CAMELLIA:!RC4:!MD5:!DSS"
(from feb2021
ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1:!AESCCM
ssl-display: yes/no display SSL connection details default: no
output-folder: (folder name)
output-folder1: (folder name)
..
output-folder9: (folder name)
if the folder does NOT start with a '/', it is assumed to be
Note these override the default and '-o' input switch..
-- For accessing Oauth protected assets
; We need an access token
use-oauth:yes
; which flavour of Oauth2 ? - only the first letter is meaningful
; oauth-flavour: Google (Gmail) or Microsoft (Office365)
oauth-flavour:microsoft for office 365
; Current token file will be saved in /fip/fix/goauth2
oauth-token-file:\OT
; Credentials file in /fip/tables/cert
oauth-credentials-file:\OC
; sffoauth and imapwire
oauth-scope:https://outlook.office365.com/.default
; Script to run when token expires - approximately every 12 hours
oauth-refresh-script: (Script in FipSeq) script to generate the access_token
using a refresh_token
oauth-refresh-script:/fip/bin/sffoauth -z wire/IMAP.O365.OAUTH.SEA -c \OC -t
\OT -H '#WN:\WN' -a
These 5 FipHdrs are use to generate, check, add/renew permissions to access
the remote data - normally Gmail or Office365
oauth-client-fiphdr: (FipHdr) default: IC
oauth-secret-fiphdr: (FipHdr) default: IS
oauth-access-fiphdr: (FipHdr) default: IA
oauth-refresh-fiphdr: (FipHdr) default: IR
oauth-expiry-fiphdr: (FipHdr) default: IX
-- Where sections of FipHdr fields are required or changes to the output style,
use keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The SysAdmin manual for more information).
They are normally specified :
fixed:QZ 1234543
partial:QT ST,3,2,U,<,>
combie:QY ep|na,(0000000)a
option:QE ep,11,7,s
repeat:QK XK,-,3
or repeat:QP PK,,4,#X
style:QS XN,%.03d
replace:QN NN abc=DEF def=GHI
newdate:QT hours+3 "\ZD"
--- Gmail using Oauth and IMAP
--- Gmail using SSL and IMAP
Generally this is being phased out for Oauth - see above
To access a Gmail account :
** beforehand, you must logon to the Gmail account
- select settings
- click on Forwarding and POP3/IMAP
- select enable IMAP
-( select Auto expunge OFF if you have more than one person/program accessing
!)
- you must use imapwiressl
- add the following to the parameter file
; Use TLS
tls-auth:ssl
use-tls:implicit
; Imap Host
imap-host:imap.gmail.com
imap-port:993
--- Using TIMER to kick off
Easy !
1. wire/IMAP.FIP
; Use FipSeq for the attributes
; If using W4 or Prestige, copy WN to another FipHdr field - RU in this case
mailbox:\WN password:\W7 fiphdr:\W3#RU:\WN#
; hostname of the exchange server
imap-host:(hostname here)
; If you have more than one fip - make sure the other system is up-to-date
; ** Add balskips group in sys/BALANCE - for each host: group:balskips
host:(hostname) ignore-localhost: nofiphdr:
skip-balance-group:balskips
; ---------------
2. setup/TIMER_IMAP.FIP
; If you have more than one fip and are running Primary/Secondary :
; ** Add wiresvr to sys/DEST_REDUN (or use an existing entry)
check-primary-server:wiresvr
group:imap
track-status:no
bandwidth-stats:no
; If you have more than one fip - make sure the other system is up-to-date
; ** Add balskips group in sys/BALANCE - for each host: group:balskips
host:(hostname) ignore-localhost: nofiphdr:
skip-balance-group:balskips
; Then for EACH mail address - add this line - emailaddress does not NORMALLY
need a domain
client:(emailaddress) type:imap fiphdr:XX:extraStuff days:X every:1
passwd:(password)
; ---------------
3. sys/SYSTEM
; add the line ...
imap mail iptimer -n timer_imap.fip
; ---------------
4. check sys/BALANCE and sys/DEST_REDUN as above
--- Testing
If things do NOT look like they are working, you can run imapwire manually with
the -1 and -D to run once and display the handshake.
So if the line in the SYSTEM file is
imap wires imapwire -s mail.bignastycorp.com -n imap.fip
.. You can test from a terminal/CMD with
imapwire -s mail.bignastycorp.com -n imap.fip -1 -D
or if using ssl
imapwiressl -s mail.bignastycorp.com -n imap.fip -1 -D
To test AND GRAB NOTHING, add the -V switch too
imapwiressl -s mail.bignastycorp.com -n imap.fip -1 -D -V
--- Note Imapwire saves the last item, date and time and UID in a file for each
mailbox in /fip/fix/imapwire
The three items are editable on 3 lines, so you can mess around at your peril
if you need !
::::::::::::::
imap_mail.zingle.com_fip$2011%hoho_inbox
::::::::::::::
408
14-Oct-2011
174223
--- Input switches are :
Mandatory :
-s : Hostname where the IMAP is running. default: none
Optional :
-1 : one single pass and then stop default: continuous
-B : default balance group for skip files default: none
(see skip-balance-group parameter)
-d : display the conversation with the remote server default: no
and pause between files for you to hit return to continue
valid ONLY with the -1 for single shot; used for debugging troublesome
connections
-D : display the conversation with the remote server default: no
-h : extra FipHdr information default: none
This is in FipSeq and should normally be quoted
Note this is the means that 'iptimer' sends variable information to imapwire
eg : -h"SN:hello#TC:200401031"
-k : on Display, send a NOOP instead of a CAPABILITY before LOGON
default: send CAPABILITY
-K : do NOT send anything before LOGON default: send CAPABILITY
-l : do NOT log anything except errors default: log files only
-L : log every file and every connection default: log files only
-n : name of the service def: name of the parameter file
-o : Next fip queue for incoming files default: spool/2go
-p : port number on the remote host default: 143
-t : sleep in seconds between connections/accesses default: 600 secs
-V : do NOT grab any files - used with -D -1 to test only default: run and
grab
-U : restart on this UID default: use last saved in the fix file
-x : Proxy server host or IP address default: none
-X : Proxy server port default: 80
-y : Proxy logon default: none
-Y : Proxy server is Squid default: no
-z : parameter file default: wire/IMAP
-Z : do NOT archive any incoming files default: archive
-v : display version number and exit.
Version Control
;02a-o2 16jun18 fipseq_extras ;b ;c extra SSL details added ;d added defUseSSL
to reset on connect ;e socks4/5 added
;f 1mar22 added oauth2 for gmail plus fixed bug when > 64k of ids on SEARCH
plus TLS remhost added
;g-h 8jul22 DU and SC are notw FipSeq
;i 10nov22 added -Z archive:no
;j-m 20dec22 oauth for MS-Office365 and check expiry too ;n connection
timeout for thru the proxy
;o 26jun23 ssl-ciphers added and tuned display
;01z 31dec08 cleanups ;f note_balance_action ;g-h 16feb11 added TLS
;i-j 31oct11 make sure date is valid and better error msgs
;k-m 02apr12 bugette with tls ;m unlink tmp on singleshot
;n 21jul12 added output-folderX
;o-s 15oct12 bugette - missed first file if folder is reset (or zapped and
remade)
;t-u 26may13 added expunge (finally) ;v added 993 as default for use-tls
;wx 7mar17 send CAPABILITY before LOGIN so we can check if LOGINDISABLED if
-D !
;y 10apr17 made logon,password etc parseable
;z 16jun17 more logging
(copyright) 2024 and previous years FingerPost Ltd.