An Orgmode Note Workflow

This post is part of the Orgmode Almanac series.

Background

One of the main reasons to use orgmode is definitely to get a better note taking workflow. Closely related to blogging or writing, the ideal note workflow is one which lets you keep a bunch of throwaway ideas and also somehow have access to them in a coherent manner. This will be a long post, and it is a work-in-progress, so, keep that in mind. Since this is mainly me¹ work-shopping my technique, the philosophy will come in a later post probably. This workflow is documented more sparsely in my config file here, in the noteYoda section². Some parts of this post also include mini video clips for clarity³.

The entire workflow will end up being something like this⁴:

Concept

While working through ideas, it actually was more useful to describe the workflow I want, and then implement it, instead of relying on the canned approaches of each package. So the basics of the ideology are listed below.

Reference Management

Reference management is one of the main reasons to consider a plain-text setup, and mine is no different. The options most commonly seen are:

Mendeley: This is a great option, and the most mobile friendly of the bunch. Sadly, the price tiers aren’t very friendly so I have to give it a hard pass.
Jabref: This is fun, but really more of a per-project management system, but it works well for that. The fact that it is Java based was a major issue for me.
Zotero: This is what I personally use and recommend. More on that in a later post.

Notes

The idea is to be able to create notes for all kinds of content. Specifically, papers or books, along with webpages. This then requires a separate system for each which is described by:

Search Engine: The search engine is key, both in terms of accessibility and scalability. It is assumed that there will be many notes, and that they will have a wide variety of content. The search interface must then simply allow us to narrow down our candidates in a meaningful manner.
Contextual Representation: This aspect of the workflow deals with representations, which should transcend the usage of tags or categories. In particular, it would be nice to be able to visualize the flow of ideas, each represented by a note.
Backlinks: In particular, by backlinks at this point we are referring to the ability to link to a pdf or a website with a unique key such that notes can be added or removed at will.
Storage: Not actually part of the workflow in the same way, since it will be handled at the system level, it is worth nothing, that in this workflow Zotero is used to export a master bib file and keeps it updated, while the notes themselves are version controlled⁵.

The concepts above will be handled by the following packages.

Concept	Package	Note
Search	deft	Has a great interface
Context	org-roam	Allows the export of graphiz mindmaps
Backlinks	org-roam, org-ref, org-noter	Covers websites, bibliographies, and pdfs respectively

A key component in this workflow is actually facilitated by the fabulous org-roam-bibtex or ORB. The basic idea is to ensure meaningful templates which interpolate smoothly with org-roam, org-ref, helm-bibtex, and org-capture.

Basic Variables

Given the packages we will be using, some variable settings are in order, namely:

1(setq
2   org_notes (concat (getenv "HOME") "/Git/Gitlab/Mine/Notes/")
3   zot_bib (concat (getenv "HOME") "/GDrive/zotLib.bib")
4   org-directory org_notes
5   deft-directory org_notes
6   org-roam-directory org_notes
7   )

Search

For the search setup, the doom-emacs deft setup, by adding +deft in my init.el, worked out of the box for me. For those who do not use doom⁶, the following should suffice:

 1(use-package deft
 2  :commands deft
 3  :init
 4  (setq deft-default-extension "org"
 5        ;; de-couples filename and note title:
 6        deft-use-filename-as-title nil
 7        deft-use-filter-string-for-filename t
 8        ;; disable auto-save
 9        deft-auto-save-interval -1.0
10        ;; converts the filter string into a readable file-name using kebab-case:
11        deft-file-naming-rules
12        '((noslash . "-")
13          (nospace . "-")
14          (case-fn . downcase)))
15  :config
16  (add-to-list 'deft-extensions "tex")
17  )

For more about the doom-emacs defaults, check the Github repo. The other aspect of interacting with the notes is via the org-roam interface and will be covered below.

Bibliography

Since I will be using org-ref, it makes no sense to load or work with the +biblio module at the moment. Thus this section is actually doom agnostic. The basic tools of bibliographic management from the emacs end are the venerable helm-bibtex (repo here) and org-ref (repo here). In order to make this guide complete, I will also describe the Zotero settings I have.

Zotero

Without getting too deep into the weeds here, the basic requirements are:

The idea is to then have one top level .bib file in some handy location which you will set up to sync automatically. To make life easier, there is a tiny recording of the next steps.

Helm-Bibtex

This venerable package is really good at interfacing with a variety of externally formatted bibliographic managers.

 1(setq
 2 bibtex-completion-notes-path "/home/haozeke/Git/Gitlab/Mine/Notes/"
 3 bibtex-completion-bibliography "/home/haozeke/GDrive/zotLib.bib"
 4 bibtex-completion-pdf-field "file"
 5 bibtex-completion-notes-template-multiple-files
 6 (concat
 7  "#+TITLE: ${title}\n"
 8  "#+ROAM_KEY: cite:${=key=}\n"
 9  "* TODO Notes\n"
10  ":PROPERTIES:\n"
11  ":Custom_ID: ${=key=}\n"
12  ":NOTER_DOCUMENT: %(orb-process-file-field \"${=key=}\")\n"
13  ":AUTHOR: ${author-abbrev}\n"
14  ":JOURNAL: ${journaltitle}\n"
15  ":DATE: ${date}\n"
16  ":YEAR: ${year}\n"
17  ":DOI: ${doi}\n"
18  ":URL: ${url}\n"
19  ":END:\n\n"
20  )
21 )

doom-emacs users like me might want to wrap the above in a nice after! org-ref expression, but it doesn’t really matter.

Explanation

To break-down aspects of the configuration snippet above:

The template includes the orb-process-file-field function to allow selecting the pdf to be used with org-noter
The file field is specified to work with the .bib file generated by Zotero
helm-bibtex allows for any of the keys in a .bib file to be used in a template, and an overly expressive one is more useful
The ROAM_KEY is defined to ensure that cite backlinks work correctly with org-roam
As I prefer to have one notes file per pdf, I have only configured the bibtex-completion-notes-template-multiple-files variable

Org-Ref

As discussed above, this just makes citations much more meaningful in orgmode.

 1(use-package org-ref
 2    :config
 3    (setq
 4         org-ref-completion-library 'org-ref-ivy-cite
 5         org-ref-get-pdf-filename-function 'org-ref-get-pdf-filename-helm-bibtex
 6         org-ref-default-bibliography (list "/home/haozeke/GDrive/zotLib.bib")
 7         org-ref-bibliography-notes "/home/haozeke/Git/Gitlab/Mine/Notes/bibnotes.org"
 8         org-ref-note-title-format "* TODO %y - %t\n :PROPERTIES:\n  :Custom_ID: %k\n  :NOTER_DOCUMENT: %F\n :ROAM_KEY: cite:%k\n  :AUTHOR: %9a\n  :JOURNAL: %j\n  :YEAR: %y\n  :VOLUME: %v\n  :PAGES: %p\n  :DOI: %D\n  :URL: %U\n :END:\n\n"
 9         org-ref-notes-directory "/home/haozeke/Git/Gitlab/Mine/Notes/"
10         org-ref-notes-function 'orb-edit-notes
11    ))

An essential aspect of this configuration is just that most of heavy lifting in terms of the notes are palmed off to helm-bibtex.

Explanation

To break-down aspects of the configuration snippet above:

The org-ref-get-pdf-filename-function simply uses the helm-bibtex settings to find the pdf
The default bibliography and notes directory are set to the same location as all the org-roam files, to encourage a flat hierarchy
The org-ref-notes-function simply ensures that, like the helm-bibtex settings, I expect one file per pdf, and that I would like to use my org-roam template instead of the org-ref or helm-bibtex one

Note that for some reason, the format specifiers for org-ref are not the keys in .bib but are instead, the following⁷:

 1In the format, the following percent escapes will be expanded.
 2%l   The BibTeX label of the citation.
 3%a   List of author names, see also `reftex-cite-punctuation'.
 4%2a  Like %a, but abbreviate more than 2 authors like Jones et al.
 5%A   First author name only.
 6%e   Works like %a, but on list of editor names.  (%2e and %E work as well)
 7It is also possible to access all other BibTeX database fields:
 8%b booktitle     %c chapter        %d edition    %h howpublished
 9%i institution   %j journal        %k key        %m month
10%n number        %o organization   %p pages      %P first page
11%r address       %s school         %u publisher  %t title
12%v volume        %y year
13%B booktitle, abbreviated          %T title, abbreviated
14%U url
15%D doi
16%S series        %N note
17%f pdf filename
18%F absolute pdf filename
19Usually, only %l is needed.  The other stuff is mainly for the echo area
20display, and for (setq reftex-comment-citations t).
21%< as a special operator kills punctuation and space around it after the
22string has been formatted.
23A pair of square brackets indicates an optional argument, and RefTeX
24will prompt for the values of these arguments.

Indexing Notes

This part of the workflow builds on the concepts best known as the Zettelkasten method. More details about the philosophy behind org-roam is here.

Org-Roam

The first part of this interface is essentially just the doom-emacs configuration, adapted for those who don’t believe in the dark side below.

 1(use-package org-roam
 2  :hook (org-load . org-roam-mode)
 3  :commands (org-roam-buffer-toggle-display
 4             org-roam-find-file
 5             org-roam-graph
 6             org-roam-insert
 7             org-roam-switch-to-buffer
 8             org-roam-dailies-date
 9             org-roam-dailies-today
10             org-roam-dailies-tomorrow
11             org-roam-dailies-yesterday)
12  :preface
13  ;; Set this to nil so we can later detect whether the user has set a custom
14  ;; directory for it, and default to `org-directory' if they haven't.
15  (defvar org-roam-directory nil)
16  :init
17  :config
18  (setq org-roam-directory (expand-file-name (or org-roam-directory "roam")
19                                             org-directory)
20        org-roam-verbose nil  ; https://youtu.be/fn4jIlFwuLU
21        org-roam-buffer-no-delete-other-windows t ; make org-roam buffer sticky
22        org-roam-completion-system 'default
23)
24
25  ;; Normally, the org-roam buffer doesn't open until you explicitly call
26  ;; `org-roam'. If `+org-roam-open-buffer-on-find-file' is non-nil, the
27  ;; org-roam buffer will be opened for you when you use `org-roam-find-file'
28  ;; (but not `find-file', to limit the scope of this behavior).
29  (add-hook 'find-file-hook
30    (defun +org-roam-open-buffer-maybe-h ()
31      (and +org-roam-open-buffer-on-find-file
32           (memq 'org-roam-buffer--update-maybe post-command-hook)
33           (not (window-parameter nil 'window-side)) ; don't proc for popups
34           (not (eq 'visible (org-roam-buffer--visibility)))
35           (with-current-buffer (window-buffer)
36             (org-roam-buffer--get-create)))))
37
38  ;; Hide the mode line in the org-roam buffer, since it serves no purpose. This
39  ;; makes it easier to distinguish among other org buffers.
40  (add-hook 'org-roam-buffer-prepare-hook #'hide-mode-line-mode))
41
42
43;; Since the org module lazy loads org-protocol (waits until an org URL is
44;; detected), we can safely chain `org-roam-protocol' to it.
45(use-package org-roam-protocol
46  :after org-protocol)
47
48
49(use-package company-org-roam
50  :after org-roam
51  :config
52  (set-company-backend! 'org-mode '(company-org-roam company-yasnippet company-dabbrev)))

Once again, for more details, check the Github repo.

Org-Roam-Bibtex

The configuration required is:

 1 (use-package org-roam-bibtex
 2  :after (org-roam)
 3  :hook (org-roam-mode . org-roam-bibtex-mode)
 4  :config
 5  (setq org-roam-bibtex-preformat-keywords
 6   '("=key=" "title" "url" "file" "author-or-editor" "keywords"))
 7  (setq orb-templates
 8        '(("r" "ref" plain (function org-roam-capture--get-point)
 9           ""
10           :file-name "${slug}"
11           :head "#+TITLE: ${=key=}: ${title}\n#+ROAM_KEY: ${ref}
12
13- tags ::
14- keywords :: ${keywords}
15
16\n* ${title}\n  :PROPERTIES:\n  :Custom_ID: ${=key=}\n  :URL: ${url}\n  :AUTHOR: ${author-or-editor}\n  :NOTER_DOCUMENT: %(orb-process-file-field \"${=key=}\")\n  :NOTER_PAGE: \n  :END:\n\n"
17
18           :unnarrowed t))))

Where most of the configuration is essentially the template again. Like helm-bibtex, ORB allows taking arbitrary keys from the .bib file.

Org Noter

The final aspect of a pdf workflow is simply ensuring that every pdf is associated with notes. The philosophy of org-noter is best described here. Only minor tweaks should be required to get this working with interleave as well.

 1(use-package org-noter
 2  :after (:any org pdf-view)
 3  :config
 4  (setq
 5   ;; The WM can handle splits
 6   org-noter-notes-window-location 'other-frame
 7   ;; Please stop opening frames
 8   org-noter-always-create-frame nil
 9   ;; I want to see the whole file
10   org-noter-hide-other nil
11   ;; Everything is relative to the main notes file
12   org-noter-notes-search-path (list org_notes)
13   )
14  )

Evidently, from my configuration, it appears that I decided to use org-noter over the more commonly described interleave because it has better support for working with multiple documents linked to one file.

Org-Protocol

I will only cover the bare minimum relating to the use of org-capture here, because eventually I intend to handle a lot more cases with orca. Note that this part of the workflow has more to do with using org-roam with websites than pdf files.

Templates

This might get complicated but I am only trying to get the bare minimum for org-protocol right now.

 1;; Actually start using templates
 2(after! org-capture
 3  ;; Firefox and Chrome
 4  (add-to-list 'org-capture-templates
 5               '("P" "Protocol" entry ; key, name, type
 6                 (file+headline +org-capture-notes-file "Inbox") ; target
 7                 "* %^{Title}\nSource: %u, %c\n #+BEGIN_QUOTE\n%i\n#+END_QUOTE\n\n\n%?"
 8                 :prepend t ; properties
 9                 :kill-buffer t))
10  (add-to-list 'org-capture-templates
11               '("L" "Protocol Link" entry
12                 (file+headline +org-capture-notes-file "Inbox")
13                 "* %? [[%:link][%(transform-square-brackets-to-round-ones \"%:description\")]]\n"
14                 :prepend t
15                 :kill-buffer t))
16)

Conclusions

At this point, many might argue that since by the end, only one template is called, defining the rest were pointless. They would be right, however, this is just how my configuration evolved. Feel free to cannibalize this for your personal benefit. Eventually I plan to expand this into something with org-journal as well, but not right now.

Rohit Goswami that is, from the landing page; obviously ↩︎
This is a reference to my fantastic pet, named Yoda ↩︎
Recorded with SimpleScreenRecorder, cut with LosslessCut, uploaded to YouTube, and embedded with a Hugo shortcode ↩︎
The video uses org-ref-notes-function-many-files as the org-ref-notes-function so the template looks a little different ↩︎
For some strange reason a lot of online posts suggested Dropbox for syncing notes, which makes no sense to me, it is always better to have version control and ignore rules ↩︎
Therefore clearly proving that the cookies of the dark side have no power in the holy text editor war ↩︎
Where these are from the org-ref documentation ↩︎

Series info

Orgmode Almanac series

Replacing Jupyter with Orgmode
Using Mathematica with Orgmode
Pandoc to Orgmode with Babel
An Orgmode Note Workflow <-- You are here!
Temporary LaTeX Documents with Orgmode
Anki Decks with Orgmode