Reclaiming Email with Astroid

Migrating Imap, Gmail and Exchange, mail accounts from GUI clients to Astroid

Background

Initially, I had planned this post to start with a brief history of the decline of email clients for Linux. That quickly got out of hand, and was therefore spun out into a post of its own (TBD). To keep things brief. Thanks to the incredible ineptitude of the Thunderbird steering committee, I ended up requiring a new mail client. Having despaired of the GUI based bloat heavy approaches of most clients, I decided to go the old fashioned route and build one up in a modular manner.

The Accounts

In no order of preference, for a variety of reasons, I have 23 distinct email accounts. However, these are actually broken down into a few basic types and associated mail fetching software.

Generic IMAP

Yahoo, Mail.ru and the rest are managed with isync

Exchange

A school account of mine uses Office 365, and will be handled with davmail in conjunction with isync

Gmail

There are a few of these, two personal, and two institutional, all of which are handled with lieer

InSync

For general IMAP accounts (anything which isn’t backed by gmail or outlook) an mbsync approach works best. Exhange accounts need davmail and are described in a separate sub-section. Before we get to the poll.sh script and astroid, it is a good idea to run each of these as we set them up, especially as the first run can take quite a long time (around an hour for 9205 messages).

1mbsync -V blah

General IMAP

For a standard IMAP account (anything which isn’t Exchange or Gmail) the only thing we need to keep track of is a good way to obfuscate passwords. We will use pass for this ¹.

1pass git init $GPG_KEY # Optional, use a secure private git repo
2pass init $GPG_KEY
3pass add mail/rgoswami.iitk

With this, we can configure a general IMAP account (an IITK webmail in this instance, but it could be anything).

 1IMAPAccount rgoswami.iitk
 2Host qasid.iitk.ac.in
 3User rgoswami
 4PassCmd "pass show mail/rgoswami.iitk"
 5AuthMechs LOGIN
 6SSLType IMAPS
 7
 8IMAPStore rgoswami.iitk-remote
 9Account rgoswami.iitk
10
11MaildirStore rgoswami.iitk-local
12SubFolders Verbatim
13Path /run/media/Storage/.mail/rgoswami_iitk/
14INBOX /run/media/Storage/.mail/rgoswami_iitk/Inbox
15Trash trash:///
16
17Channel rgoswami.iitk
18Master :rgoswami.iitk-remote:
19Slave :rgoswami.iitk-local:
20Patterns *
21Create Both
22SyncState *

Exchange Accounts

For Exchange accounts we need to setup davmail. Thankfully the process is actually excessively simple for a single account. For each account, a .properties file needs to be generated. The password section is kept blank in this case, since we will authenticate with the O365Interactive protocol.

1# The default setup
2davmail ~/.davmail.properties

Use the following properties in the .davemail.properties file to prevent timeout errors:

1davmail.folderSizeLimit=50
2davmail.clientSoTimeout=0
3davmail.enableKeepAlive=true

Further information is available at the official docs.

Now we can configure the .mbsyncrc in a manner analogous to the standard IMAP setup, but with the port and host where davmail is running.

 1IMAPAccount rog32
 2Host localhost
 3User rog32@hi.is
 4Pass dummy
 5Port 1143
 6SSLType None
 7AuthMechs LOGIN
 8
 9IMAPStore rog32-remote
10Account rog32
11MaildirStore rog32-local
12SubFolders Verbatim
13Path /run/media/Storage/.mail/rog32/
14Inbox /run/media/Storage/.mail/rog32/Inbox
15
16Channel rog32
17Master :rog32-remote:
18Slave :rog32-local:
19Patterns *
20Create Both
21SyncState *

The SSLType and AuthMechs are important parameters. Note that the Pass is truly not important since we an OAuth token is stored in the properties file.

Gmaileer

The general setup gmi init blah@gmail.com works well for personal accounts. However, there was an additional step for the IEEE account.

IEEE

Some issues with tags led to the following changes in the .gmaileer.json file

1{"replace_slash_with_dot": false, "account": "rgoswami@ieee.org", "timeout": 600, "drop_non_existing_label": false, "ignore_empty_history": false, "ignore_tags": ["TODO","new"], "ignore_remote_labels": ["CATEGORY_SOCIAL", "CATEGORY_FORUMS", "CATEGORY_PROMOTIONS", "CATEGORY_PERSONAL", "CATEGORY_UPDATES"], "remove_local_messages": true, "file_extension": ""}

Essentially just the addition of two new ignore_tags.

Notmuch

At this point, we have all mail synced into local directories, but we have no access to view or interact with them. We will start by setting up notmuch to search and index our mail. This is pretty basic for now.

 1[database]
 2path=/run/media/Storage/.mail/
 3[user]
 4name=Person
 5primary_email=primary@domain.com
 6other_email=one@domain.com;two@domain.com
 7[new]
 8tags=new;unread;inbox;TODO;
 9ignore=/.*[.](json|lock|bak)$/
10[search]
11exclude_tags=deleted;spam;
12[maildir]
13synchronize_flags=true

There are a lot more options which may be configured, but this is enough to get started.

Sending Email

At this stage we need a way to actually send email. A simple configuration can be setup in /.config/msmtp/config is given below (where we use pass again):

 1defaults
 2port 587
 3tls on
 4auth on
 5logfile ~/.config/msmtp/.msmtp.log
 6tls_trust_file /etc/ssl/certs/ca-certificates.crt
 7
 8account r95g10
 9host smtp.gmail.com
10from r95g10@gmail.com
11user r95g10@gmail.com
12tls_starttls on
13tls on
14auth on
15port 587
16passwordeval pass show mail/r95g10.gmail
17
18account rog32
19host localhost
20from rog32@hi.is
21user rog32@hi.is
22tls_starttls off
23tls off
24auth plain
25port 1025
26password dummy

Note that the exchange account again uses a dummy password, and the real password will be prompted for on the first run. The permissions of the file above should be 600. It is prudent to test at-least the exchange section to login.

1echo "Hello world" | msmtp --account=rog32 r95g10@gmail.com

Astroid

The bulk of this setup is by the numbers, with the exception of the poll script. However, minimally configure astroid with the location of our notmuch configuration.

1astroid --new-config
2vim ~/.config/astroid/config

The relevant sections are:

1"notmuch_config": "\/home\/whoever\/.notmuch-config",

The accounts section is fairly self explanatory, but we will need to use the following sendmail line as well:

1"sendmail": "msmtp --read-envelope-from -i -t",

Nix Python Poll Script

Natively only bash appears to be supported. However, with nix, we can use a reproducible python script with the sh library to call system functions instead.

 1from pathlib import Path
 2import sh
 3
 4# For generic IMAP maildirs
 5
 6ISYNC_LABELS = ["rgoswami.iitk", "rog32"]
 7
 8for isync in ISYNC_LABELS:
 9    sh.mbsync("-V",isync,_bg=True)
10
11
12# Gmaileer
13GMAIL_IDENTIFIERS = ["gmail", "univ", "ieee"]
14
15path = Path(r"/run/media/haozeke/Storage/.mail/")
16
17for dirs in path.iterdir():
18    if dirs.is_dir():
19        for gmi in GMAIL_IDENTIFIERS:
20            if gmi in dirs.name:
21                print(f"Syncing {dirs.name}")
22                sh.gmi("sync", _cwd=dirs, _fg=True)

This needs to be coupled with the standard nix shebang in the poll.sh file:

1#!/usr/bin/env nix-shell
2#!nix-shell -i python3 -p "python38.withPackages(ps: [ ps.numpy ps.sh ])"

Navigation

For working with HTML emails, we need to highlight the notice about potentially sketchy HTML using (defaults) j or k and then hit enter or o.

Note that since most email is actually sent with text/html it might make more sense to configure the thread_view in the configuration file.

 1"thread_view": {
 2    "open_html_part_external": "false",
 3    "preferred_type": "html",
 4    "preferred_html_only": "false",
 5    "allow_remote_when_encrypted": "false",
 6    "open_external_link": "xdg-open",
 7    "default_save_directory": "~",
 8    "indent_messages": "false",
 9    "gravatar": {
10        "enable": "true"
11    },
12    "mark_unread_delay": "0.5",
13    "expand_flagged": "true"
14},

Deletion

There are again, two different approaches to deletion.

Gmail

For gmail accounts it is simple, just adding a trash tag will do the trick, so we can select multiple emails with t and then hit + to add trash and everything works out

Other Accounts

We can delete mail forever (from the server as well), by using a safe-tag and then using notmuch

The generic non-lieer method requires:

1notmuch search --output=files --format=text0 tag:killed | xargs -r0 rm

A pre-new hook will work.

Composition

Thankfully, astroid supports GPG encryption as well as markdown support. This makes for simple integration with any popular editor.

Conclusions

This is enough to get started for a while, but it isn’t yet at the stage where I can replace thunderbird unfortunately. However, there are several pain points to be addressed, which will be covered in a future post. Some of these are essentially related to network fluctuations, and the awkward deletion setup.

Update

• astroid appears to be having a bit of a development crisis
  • The sequel (ragnarok) unfortunately seems to have gone towards a browser frontend, which is quite unacceptable to me
    • This makes it far too similar to (in principle) Mailspring (though with less problematic connectivity issues)
    • This means I won’t be moving forward with the next set of posts regarding astroid and will move towards emacs again
• Max Mehl has a good collection of scripts for astroid