Managing cloud based calibre libraries

Ruminations on cross-cloud library management

Background

I have a set of different devices across which I need to sync my library (described in an earlier post). Ideally this should be a lazy synchronization mechanism and be as general as possible.

Constraints

calibre should never be modifying the files while they are being synced, not only because it creates and removes a lot of temporary files and folders which will cause churn, but also because it actually locks files during modification and this leads to permission issues.

Approximations to solutions

0th: Cloud native synchronization tools

I use pCloud, which has a nifty GUI client (and CLI) across my operating systems which essentially provides a mounted drive.

• Store directly on the drive
  • Very slow, since many directories and temporaries are created

1st: GUI based synchronization

Next we can consider using sync via the GUI. This essentially binds two folders together and continuously ensures they are up to date.

• This syncs continuously and so ended up with permission errors

2nd: rsync to mounted drive

As a slightly more complicated workflow we can use a local synchronization tool on a cron schedule to sync to the drive mount.

1rsync -avP /path/to/source/directory/ /path/to/destination/directory

Pros

• cron job can be linked to triggers like:
  • Checking if calibre is running
  • Checking for network / power requirements

Cons

• This doesn’t really sync immediately, and the delay might be pretty significant if earlier jobs are skipped

3rd: Use rclone manually

rsync is really meant for SSH connections, while Rclone is fantastic for working with cloud providers. Setting this up is a breeze.

1rclone config
2# Best to encrypt it
3rclone sync -P /path/to/calibre/library remote-name:/path/in/pcloud

Pros

• Simple setup

Cons

• Manual, so this will need to be run every time

Coupling to calibre

calibre is the only program which is supposed to manage the libraries. In that sense, it makes sense to strongly couple syncing with calibre. Most of my machines have gnome-keyring installed, and rclone supports secrets stored in password managers so:

1secret-tool store --label="rclone" service "rclone" user "calisync"

Which can then simply be set to RCLONE_PASSWORD_COMMAND and used directly:

1export RCLONE_PASSWORD_COMMAND='secret-tool lookup service "rclone" user "calisync"'
2rclone sync -P CalibreLibs/Synced pCloud:/Calibre --ask-password=false

Putting this together, we can wrap calibre runs in a script:

 1#!/bin/bash
 2# $HOME/.local/bin/calibre_and_sync.sh
 3
 4# Start calibre and wait
 5calibre
 6wait $! # optional really, but a bit safer
 7
 8# Setup and run sync
 9RCLONE_PASSWORD_COMMAND='secret-tool lookup service "rclone" user "calisync"'
10rclone sync -P ~/CalibreLibs/Synced pCloud:/Calibre --ask-password=false

It can be made more fancy if necessary, but should ideally just be customized for each machine. In any case the script is mostly to be used from a .desktop file, e.g. in .local/share/applications/CaliSync.

1[Desktop Entry]
2Type=Application
3Name=CaliSync
4Exec=/home/username/.local/bin/calibre_and_sync.sh
5Icon=calibre-gui
6GenericName=E-book library management with cloud sync
7Comment=Start Calibre and sync library with pCloud
8Terminal=false
9Categories=Office;Utility;

Do keep in mind that environment variables are not expanded, so generalizing this would require a suitable dotfile management system like chezmoi (as seen here).

Visual Tweaks

The script discussed in the previous section is fine, but it is often nicer to have a visual indication of what’s going on in terms of the upload, especially if there are a lot of files.

 1#!/bin/bash
 2# $HOME/.local/bin/calibre_and_sync.sh
 3
 4# Start Calibre and wait
 5calibre
 6
 7# Setup and run sync
 8zenity --notification --text="Start syncing"
 9RCLONE_PASSWORD_COMMAND='secret-tool lookup service "rclone" user "calisync"'
10(
11  rclone sync -P ~/CalibreLibs/Synced pCloud:/Calibre --ask-password=false \
12      --password-command="$RCLONE_PASSWORD_COMMAND" 2>&1 | \
13  while read -r line; do
14    echo "# $line"
15  done
16) |
17zenity --progress \
18  --title="Syncing Calibre Library with pCloud" \
19  --text="Syncing..." \
20  --percentage=0 \
21  --pulsate \
22  --auto-close \
23  --auto-kill
24zenity --notification --text="Sync completed"

This is nice, but doesn’t actually show the full progress. zenity doesn’t support embedding a terminal, so the only other option is to spawn a shell.

 1#!/bin/bash
 2# $HOME/.local/bin/calibre_and_sync.sh
 3
 4# Start Calibre and wait
 5calibre
 6
 7# Setup and run sync
 8zenity --notification --text="Start syncing"
 9RCLONE_PASSWORD_COMMAND='secret-tool lookup service "rclone" user "calisync"'
10foot -- sh -c "rclone sync -P ~/CalibreLibs/Synced pCloud:/Calibre --ask-password=false --password-command=\"$RCLONE_PASSWORD_COMMAND\""
11zenity --notification --text="Sync completed"

Integrating with zotero

I’ve dicussed working with Zotero before, but when working with cloud synced calibre libraries, to prevent duplication of files, relative linking of base directory attachments works really well¹.

Some reasons to do this:

• The main goal of this is to enable keeping notes per-book, which Calibre doesn’t support
  • Also to get easy access without duplicated files on the backing WebDAV service (pCloud in my case)
• Allows easier, more natural bibtex handling in Zotero, without going through a sync plugin like ZMI

However:

• Do not rename from Zotero, this will break integration with Calibre

Conclusions

With this setup, it is now fairly trivial to sync libraries across devices, and thus, to take advantage of suitably large storage devices and local calibre readers. Android devices and calibre library support has been a bit of a roller-coaster, especially since Calibre Companion was sold off and abandoned, but for now, it works, as does Calibre Sync and KoReader. That being said, the best approach would be pairing this setup with something like FolderSync and Leger Calibre would be the best but Leger Calibre has gone the way of the dodo, leaving one with precious little options, though KoReader is still the best option for local calibre management without cover information.

reprex addenum

This post was executed with binaries at the following versions:

1rclone version
2rclone v1.63.1
3# pcloud 1.14.0
4rsync --version
5rsync  version 3.2.7  protocol version 31
6Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others.
7zenity --version
83.44.2