Migrating Imap, Gmail and Exchange, mail accounts from GUI clients to Astroid

## Background

Initially, I had planned this post to start with a brief history of the decline of email clients for Linux. That quickly got out of hand, and was therefore spun out into a post of its own (TBD). To keep things brief. Thanks to the incredible ineptitude of the Thunderbird steering committee, I ended up requiring a new mail client. Having despaired of the GUI based bloat heavy approaches of most clients, I decided to go the old fashioned route and build one up in a modular manner.

### Series

This post is part of a series on email.

1. The Decline of Linux Email (TBD)
2. Reclaiming Email with Astroid <– You are here!
3. Emacs and Email (TBD)

## The Accounts

In no order of preference, for a variety of reasons, I have 23 distinct email accounts. However, these are actually broken down into a few basic types and associated mail fetching software.

Generic IMAP
Yahoo, Mail.ru and the rest are managed with isync
Exchange
A school account of mine uses Office 365, and will be handled with davmail in conjunction with isync
Gmail
There are a few of these, two personal, and two institutional, all of which are handled with lieer

## InSync

For general IMAP accounts (anything which isn’t backed by gmail or outlook) an mbsync approach works best. Exhange accounts need davmail and are described in a separate sub-section. Before we get to the poll.sh script and astroid, it is a good idea to run each of these as we set them up, especially as the first run can take quite a long time (around an hour for 9205 messages).

mbsync -V blah


### General IMAP

For a standard IMAP account (anything which isn’t Exchange or Gmail) the only thing we need to keep track of is a good way to obfuscate passwords. We will use pass for this 1.

pass git init $GPG_KEY # Optional, use a secure private git repo pass init$GPG_KEY


With this, we can configure a general IMAP account (an IITK webmail in this instance, but it could be anything).

IMAPAccount rgoswami.iitk
Host qasid.iitk.ac.in
User rgoswami
PassCmd "pass show mail/rgoswami.iitk"
SSLType IMAPS

IMAPStore rgoswami.iitk-remote
Account rgoswami.iitk

MaildirStore rgoswami.iitk-local
SubFolders Verbatim
Path /run/media/Storage/.mail/rgoswami_iitk/
INBOX /run/media/Storage/.mail/rgoswami_iitk/Inbox
Trash trash:///

Channel rgoswami.iitk
Master :rgoswami.iitk-remote:
Slave :rgoswami.iitk-local:
Patterns *
Create Both
SyncState *


### Exchange Accounts

For Exchange accounts we need to setup davmail. Thankfully the process is actually excessively simple for a single account. For each account, a .properties file needs to be generated. The password section is kept blank in this case, since we will authenticate with the O365Interactive protocol.

# The default setup
davmail ~/.davmail.properties


Use the following properties in the .davemail.properties file to prevent timeout errors:

davmail.folderSizeLimit=50
davmail.clientSoTimeout=0
davmail.enableKeepAlive=true


Further information is available at the official docs.

Now we can configure the .mbsyncrc in a manner analogous to the standard IMAP setup, but with the port and host where davmail is running.

IMAPAccount rog32
Host localhost
User rog32@hi.is
Pass dummy
Port 1143
SSLType None

IMAPStore rog32-remote
Account rog32
MaildirStore rog32-local
SubFolders Verbatim
Path /run/media/Storage/.mail/rog32/
Inbox /run/media/Storage/.mail/rog32/Inbox

Channel rog32
Master :rog32-remote:
Slave :rog32-local:
Patterns *
Create Both
SyncState *


The SSLType and AuthMechs are important parameters. Note that the Pass is truly not important since we an OAuth token is stored in the properties file.

## Gmaileer

The general setup gmi init blah@gmail.com works well for personal accounts. However, there was an additional step for the IEEE account.

### IEEE

Some issues with tags led to the following changes in the .gmaileer.json file

{"replace_slash_with_dot": false, "account": "rgoswami@ieee.org", "timeout": 600, "drop_non_existing_label": false, "ignore_empty_history": false, "ignore_tags": ["TODO","new"], "ignore_remote_labels": ["CATEGORY_SOCIAL", "CATEGORY_FORUMS", "CATEGORY_PROMOTIONS", "CATEGORY_PERSONAL", "CATEGORY_UPDATES"], "remove_local_messages": true, "file_extension": ""}


Essentially just the addition of two new ignore_tags.

## Notmuch

At this point, we have all mail synced into local directories, but we have no access to view or interact with them. We will start by setting up notmuch to search and index our mail. This is pretty basic for now.

[database]
path=/run/media/Storage/.mail/
[user]
name=Person
primary_email=primary@domain.com
other_email=one@domain.com;two@domain.com
[new]
ignore=/.*[.](json|lock|bak)\$/
[search]
exclude_tags=deleted;spam;
[maildir]
synchronize_flags=true


There are a lot more options which may be configured, but this is enough to get started.

## Sending Email

At this stage we need a way to actually send email. A simple configuration can be setup in /.config/msmtp/config is given below (where we use pass again):

defaults
port 587
tls on
auth on
logfile ~/.config/msmtp/.msmtp.log
tls_trust_file /etc/ssl/certs/ca-certificates.crt

account r95g10
host smtp.gmail.com
from r95g10@gmail.com
user r95g10@gmail.com
tls_starttls on
tls on
auth on
port 587

account rog32
host localhost
from rog32@hi.is
user rog32@hi.is
tls_starttls off
tls off
auth plain
port 1025


Note that the exchange account again uses a dummy password, and the real password will be prompted for on the first run. The permissions of the file above should be 600. It is prudent to test at-least the exchange section to login.

echo "Hello world" | msmtp --account=rog32 r95g10@gmail.com


## Astroid

The bulk of this setup is by the numbers, with the exception of the poll script. However, minimally configure astroid with the location of our notmuch configuration.

astroid --new-config
vim ~/.config/astroid/config


The relevant sections are:

"notmuch_config": "\/home\/whoever\/.notmuch-config",


The accounts section is fairly self explanatory, but we will need to use the following sendmail line as well:

"sendmail": "msmtp --read-envelope-from -i -t",


### Nix Python Poll Script

Natively only bash appears to be supported. However, with nix, we can use a reproducible python script with the sh library to call system functions instead.

from pathlib import Path
import sh

# For generic IMAP maildirs

ISYNC_LABELS = ["rgoswami.iitk", "rog32"]

for isync in ISYNC_LABELS:
sh.mbsync("-V",isync,_bg=True)

# Gmaileer
GMAIL_IDENTIFIERS = ["gmail", "univ", "ieee"]

path = Path(r"/run/media/haozeke/Storage/.mail/")

for dirs in path.iterdir():
if dirs.is_dir():
for gmi in GMAIL_IDENTIFIERS:
if gmi in dirs.name:
print(f"Syncing {dirs.name}")
sh.gmi("sync", _cwd=dirs, _fg=True)


This needs to be coupled with the standard nix shebang in the poll.sh file:

#!/usr/bin/env nix-shell
#!nix-shell -i python3 -p "python38.withPackages(ps: [ ps.numpy ps.sh ])"


For working with HTML emails, we need to highlight the notice about potentially sketchy HTML using (defaults) j or k and then hit enter or o.

Note that since most email is actually sent with text/html it might make more sense to configure the thread_view in the configuration file.

"thread_view": {
"open_html_part_external": "false",
"preferred_type": "html",
"preferred_html_only": "false",
"allow_remote_when_encrypted": "false",
"default_save_directory": "~",
"indent_messages": "false",
"gravatar": {
"enable": "true"
},
"expand_flagged": "true"
},

#### Deletion

There are again, two different approaches to deletion.

Gmail
For gmail accounts it is simple, just adding a trash tag will do the trick, so we can select multiple emails with t and then hit + to add trash and everything works out
Other Accounts
We can delete mail forever (from the server as well), by using a safe-tag and then using notmuch

The generic non-lieer method requires:

notmuch search --output=files --format=text0 tag:killed | xargs -r0 rm


A pre-new hook will work.

#### Composition

Thankfully, astroid supports GPG encryption as well as markdown support. This makes for simple integration with any popular editor.

## Conclusions

This is enough to get started for a while, but it isn’t yet at the stage where I can replace thunderbird unfortunately. However, there are several pain points to be addressed, which will be covered in a future post. Some of these are essentially related to network fluctuations, and the awkward deletion setup.

### Update

• astroid appears to be having a bit of a development crisis
• The sequel (ragnarok) unfortunately seems to have gone towards a browser frontend, which is quite unacceptable to me
• This makes it far too similar to (in principle) Mailspring (though with less problematic connectivity issues)
• This means I won’t be moving forward with the next set of posts regarding astroid and will move towards emacs again
• Max Mehl has a good collection of scripts for astroid

1. The GPG key itself can be stored with keybase ↩︎