Written: 2023-08-19 13:33 +0000
Ruminations on cross-cloud library management
I have a set of different devices across which I need to sync my library (described in an earlier post). Ideally this should be a lazy synchronization mechanism and be as general as possible.
calibre should never be modifying the files while they are being synced, not
only because it creates and removes a lot of temporary files and folders which
will cause churn, but also because it actually locks files during modification
and this leads to permission issues.
Approximations to solutions
0th: Cloud native synchronization tools
I use pCloud, which has a nifty GUI client (and CLI) across my operating systems which essentially provides a mounted drive.
- Store directly on the drive
- Very slow, since many directories and temporaries are created
1st: GUI based synchronization
Next we can consider using
sync via the GUI. This essentially binds two
folders together and continuously ensures they are up to date.
- This syncs continuously and so ended up with permission errors
rsync to mounted drive
As a slightly more complicated workflow we can use a local synchronization tool
cron schedule to sync to the drive mount.
1rsync -avP /path/to/source/directory/ /path/to/destination/directory
cronjob can be linked to triggers like:
- Checking if
- Checking for network / power requirements
- Checking if
- This doesn’t really sync immediately, and the delay might be pretty significant if earlier jobs are skipped
rsync is really meant for SSH connections, while Rclone is fantastic for
working with cloud providers. Setting this up is a breeze.
1rclone config 2# Best to encrypt it 3rclone sync -P /path/to/calibre/library remote-name:/path/in/pcloud
- Simple setup
- Manual, so this will need to be run every time
calibre is the only program which is supposed to manage the libraries. In that
sense, it makes sense to strongly couple syncing with
calibre. Most of my
gnome-keyring installed, and
rclone supports secrets stored in
password managers so:
1secret-tool store --label="rclone" service "rclone" user "calisync"
Which can then simply be set to
RCLONE_PASSWORD_COMMAND and used directly:
1export RCLONE_PASSWORD_COMMAND='secret-tool lookup service "rclone" user "calisync"' 2rclone sync -P CalibreLibs/Synced pCloud:/Calibre --ask-password=false
Putting this together, we can wrap
calibre runs in a script:
1#!/bin/bash 2# $HOME/.local/bin/calibre_and_sync.sh 3 4# Start calibre and wait 5calibre 6wait $! # optional really, but a bit safer 7 8# Setup and run sync 9RCLONE_PASSWORD_COMMAND='secret-tool lookup service "rclone" user "calisync"' 10rclone sync -P ~/CalibreLibs/Synced pCloud:/Calibre --ask-password=false
It can be made more fancy if necessary, but should ideally just be customized
for each machine. In any case the script is mostly to be used from a
file, e.g. in
1[Desktop Entry] 2Type=Application 3Name=CaliSync 4Exec=/home/username/.local/bin/calibre_and_sync.sh 5Icon=calibre-gui 6GenericName=E-book library management with cloud sync 7Comment=Start Calibre and sync library with pCloud 8Terminal=false 9Categories=Office;Utility;
Do keep in mind that environment variables are not expanded, so generalizing
this would require a suitable dotfile management system like
chezmoi (as seen
The script discussed in the previous section is fine, but it is often nicer to have a visual indication of what’s going on in terms of the upload, especially if there are a lot of files.
1#!/bin/bash 2# $HOME/.local/bin/calibre_and_sync.sh 3 4# Start Calibre and wait 5calibre 6 7# Setup and run sync 8zenity --notification --text="Start syncing" 9RCLONE_PASSWORD_COMMAND='secret-tool lookup service "rclone" user "calisync"' 10( 11 rclone sync -P ~/CalibreLibs/Synced pCloud:/Calibre --ask-password=false \ 12 --password-command="$RCLONE_PASSWORD_COMMAND" 2>&1 | \ 13 while read -r line; do 14 echo "# $line" 15 done 16) | 17zenity --progress \ 18 --title="Syncing Calibre Library with pCloud" \ 19 --text="Syncing..." \ 20 --percentage=0 \ 21 --pulsate \ 22 --auto-close \ 23 --auto-kill 24zenity --notification --text="Sync completed"
This is nice, but doesn’t actually show the full progress.
support embedding a terminal, so the only other option is to spawn a shell.
1#!/bin/bash 2# $HOME/.local/bin/calibre_and_sync.sh 3 4# Start Calibre and wait 5calibre 6 7# Setup and run sync 8zenity --notification --text="Start syncing" 9RCLONE_PASSWORD_COMMAND='secret-tool lookup service "rclone" user "calisync"' 10foot -- sh -c "rclone sync -P ~/CalibreLibs/Synced pCloud:/Calibre --ask-password=false --password-command=\"$RCLONE_PASSWORD_COMMAND\"" 11zenity --notification --text="Sync completed"
With this setup, it is now fairly trivial to sync libraries across devices, and
thus, to take advantage of suitably large storage devices and local
readers. Android devices and
calibre library support has been a bit of a
roller-coaster, especially since Calibre Companion was sold off and abandoned,
but for now, it works, as does Calibre Sync and KoReader. That being said, the
best approach would be pairing this setup with something like FolderSync and
Leger Calibre would be the best but Leger Calibre has gone the way of the dodo,
leaving one with precious little options, though KoReader is still the best
option for local
calibre management without cover information.
This post was executed with binaries at the following versions:
1rclone version 2rclone v1.63.1 3# pcloud 1.14.0 4rsync --version 5rsync version 3.2.7 protocol version 31 6Copyright (C) 1996-2022 by Andrew Tridgell, Wayne Davison, and others. 7zenity --version 83.44.2