Migrating Imap, Gmail and Exchange, mail accounts from GUI clients to Astroid

## Background

Initially, I had planned this post to start with a brief history of the decline of email clients for Linux. That quickly got out of hand, and was therefore spun out into a post of its own (TBD). To keep things brief. Thanks to the incredible ineptitude of the Thunderbird steering committee, I ended up requiring a new mail client. Having despaired of the GUI based bloat heavy approaches of most clients, I decided to go the old fashioned route and build one up in a modular manner.

### Series

This post is part of a series on email.

1. The Decline of Linux Email (TBD)
2. Reclaiming Email with Astroid <– You are here!
3. Emacs and Email (TBD)

## The Accounts

In no order of preference, for a variety of reasons, I have 23 distinct email accounts. However, these are actually broken down into a few basic types and associated mail fetching software.

Generic IMAP
Yahoo, Mail.ru and the rest are managed with isync
Exchange
A school account of mine uses Office 365, and will be handled with davmail in conjunction with isync
Gmail
There are a few of these, two personal, and two institutional, all of which are handled with lieer

## InSync

For general IMAP accounts (anything which isn’t backed by gmail or outlook) an mbsync approach works best. Exhange accounts need davmail and are described in a separate sub-section. Before we get to the poll.sh script and astroid, it is a good idea to run each of these as we set them up, especially as the first run can take quite a long time (around an hour for 9205 messages).

1mbsync -V blah


### General IMAP

For a standard IMAP account (anything which isn’t Exchange or Gmail) the only thing we need to keep track of is a good way to obfuscate passwords. We will use pass for this 1.

1pass git init $GPG_KEY # Optional, use a secure private git repo 2pass init$GPG_KEY


With this, we can configure a general IMAP account (an IITK webmail in this instance, but it could be anything).

 1IMAPAccount rgoswami.iitk
2Host qasid.iitk.ac.in
3User rgoswami
4PassCmd "pass show mail/rgoswami.iitk"
6SSLType IMAPS
7
8IMAPStore rgoswami.iitk-remote
9Account rgoswami.iitk
10
11MaildirStore rgoswami.iitk-local
12SubFolders Verbatim
13Path /run/media/Storage/.mail/rgoswami_iitk/
14INBOX /run/media/Storage/.mail/rgoswami_iitk/Inbox
15Trash trash:///
16
17Channel rgoswami.iitk
18Master :rgoswami.iitk-remote:
19Slave :rgoswami.iitk-local:
20Patterns *
21Create Both
22SyncState *


### Exchange Accounts

For Exchange accounts we need to setup davmail. Thankfully the process is actually excessively simple for a single account. For each account, a .properties file needs to be generated. The password section is kept blank in this case, since we will authenticate with the O365Interactive protocol.

1# The default setup
2davmail ~/.davmail.properties


Use the following properties in the .davemail.properties file to prevent timeout errors:

1davmail.folderSizeLimit=50
2davmail.clientSoTimeout=0
3davmail.enableKeepAlive=true


Further information is available at the official docs.

Now we can configure the .mbsyncrc in a manner analogous to the standard IMAP setup, but with the port and host where davmail is running.

 1IMAPAccount rog32
2Host localhost
3User rog32@hi.is
4Pass dummy
5Port 1143
6SSLType None
8
9IMAPStore rog32-remote
10Account rog32
11MaildirStore rog32-local
12SubFolders Verbatim
13Path /run/media/Storage/.mail/rog32/
14Inbox /run/media/Storage/.mail/rog32/Inbox
15
16Channel rog32
17Master :rog32-remote:
18Slave :rog32-local:
19Patterns *
20Create Both
21SyncState *


The SSLType and AuthMechs are important parameters. Note that the Pass is truly not important since we an OAuth token is stored in the properties file.

## Gmaileer

The general setup gmi init blah@gmail.com works well for personal accounts. However, there was an additional step for the IEEE account.

### IEEE

Some issues with tags led to the following changes in the .gmaileer.json file

1{"replace_slash_with_dot": false, "account": "rgoswami@ieee.org", "timeout": 600, "drop_non_existing_label": false, "ignore_empty_history": false, "ignore_tags": ["TODO","new"], "ignore_remote_labels": ["CATEGORY_SOCIAL", "CATEGORY_FORUMS", "CATEGORY_PROMOTIONS", "CATEGORY_PERSONAL", "CATEGORY_UPDATES"], "remove_local_messages": true, "file_extension": ""}


Essentially just the addition of two new ignore_tags.

## Notmuch

At this point, we have all mail synced into local directories, but we have no access to view or interact with them. We will start by setting up notmuch to search and index our mail. This is pretty basic for now.

 1[database]
2path=/run/media/Storage/.mail/
3[user]
4name=Person
5primary_email=primary@domain.com
6other_email=one@domain.com;two@domain.com
7[new]
9ignore=/.*[.](json|lock|bak)\$/
10[search]
11exclude_tags=deleted;spam;
12[maildir]
13synchronize_flags=true


There are a lot more options which may be configured, but this is enough to get started.

## Sending Email

At this stage we need a way to actually send email. A simple configuration can be setup in /.config/msmtp/config is given below (where we use pass again):

 1defaults
2port 587
3tls on
4auth on
5logfile ~/.config/msmtp/.msmtp.log
6tls_trust_file /etc/ssl/certs/ca-certificates.crt
7
8account r95g10
9host smtp.gmail.com
10from r95g10@gmail.com
11user r95g10@gmail.com
12tls_starttls on
13tls on
14auth on
15port 587
16passwordeval pass show mail/r95g10.gmail
17
18account rog32
19host localhost
20from rog32@hi.is
21user rog32@hi.is
22tls_starttls off
23tls off
24auth plain
25port 1025


Note that the exchange account again uses a dummy password, and the real password will be prompted for on the first run. The permissions of the file above should be 600. It is prudent to test at-least the exchange section to login.

1echo "Hello world" | msmtp --account=rog32 r95g10@gmail.com


## Astroid

The bulk of this setup is by the numbers, with the exception of the poll script. However, minimally configure astroid with the location of our notmuch configuration.

1astroid --new-config
2vim ~/.config/astroid/config


The relevant sections are:

1"notmuch_config": "\/home\/whoever\/.notmuch-config",


The accounts section is fairly self explanatory, but we will need to use the following sendmail line as well:

1"sendmail": "msmtp --read-envelope-from -i -t",


### Nix Python Poll Script

Natively only bash appears to be supported. However, with nix, we can use a reproducible python script with the sh library to call system functions instead.

 1from pathlib import Path
2import sh
3
4# For generic IMAP maildirs
5
6ISYNC_LABELS = ["rgoswami.iitk", "rog32"]
7
8for isync in ISYNC_LABELS:
9    sh.mbsync("-V",isync,_bg=True)
10
11
12# Gmaileer
13GMAIL_IDENTIFIERS = ["gmail", "univ", "ieee"]
14
15path = Path(r"/run/media/haozeke/Storage/.mail/")
16
17for dirs in path.iterdir():
18    if dirs.is_dir():
19        for gmi in GMAIL_IDENTIFIERS:
20            if gmi in dirs.name:
21                print(f"Syncing {dirs.name}")
22                sh.gmi("sync", _cwd=dirs, _fg=True)


This needs to be coupled with the standard nix shebang in the poll.sh file:

1#!/usr/bin/env nix-shell
2#!nix-shell -i python3 -p "python38.withPackages(ps: [ ps.numpy ps.sh ])"


For working with HTML emails, we need to highlight the notice about potentially sketchy HTML using (defaults) j or k and then hit enter or o.

Note that since most email is actually sent with text/html it might make more sense to configure the thread_view in the configuration file.

 1"thread_view": {
2    "open_html_part_external": "false",
3    "preferred_type": "html",
4    "preferred_html_only": "false",
5    "allow_remote_when_encrypted": "false",
7    "default_save_directory": "~",
8    "indent_messages": "false",
9    "gravatar": {
10        "enable": "true"
11    },
13    "expand_flagged": "true"
14},


#### Deletion

There are again, two different approaches to deletion.

Gmail
For gmail accounts it is simple, just adding a trash tag will do the trick, so we can select multiple emails with t and then hit + to add trash and everything works out
Other Accounts
We can delete mail forever (from the server as well), by using a safe-tag and then using notmuch

The generic non-lieer method requires:

1notmuch search --output=files --format=text0 tag:killed | xargs -r0 rm


A pre-new hook will work.

#### Composition

Thankfully, astroid supports GPG encryption as well as markdown support. This makes for simple integration with any popular editor.

## Conclusions

This is enough to get started for a while, but it isn’t yet at the stage where I can replace thunderbird unfortunately. However, there are several pain points to be addressed, which will be covered in a future post. Some of these are essentially related to network fluctuations, and the awkward deletion setup.

### Update

• astroid appears to be having a bit of a development crisis
• The sequel (ragnarok) unfortunately seems to have gone towards a browser frontend, which is quite unacceptable to me
• This makes it far too similar to (in principle) Mailspring (though with less problematic connectivity issues)
• This means I won’t be moving forward with the next set of posts regarding astroid and will move towards emacs again
• Max Mehl has a good collection of scripts for astroid

1. The GPG key itself can be stored with keybase ↩︎