Request for helm-bibtex-notes-file

[jagrg]

Instead of storing my notes in a separate directory, which I think is what helm-bibtex proposes, I like to keep my notes in one org file, which looks something like this:

** Author (2013) Title
:PROPERTIES:
:Custom_ID: bibtex-key
:END:

So I was wondering if you could add "helm-bibtex-notes-file", which would be the equivalent of "helm-bibtex-notes-path". But instead of searching for the bibtex-key.org file, it would search for the bibtex-key in the file specified by the user, for example:

(setq helm-bibtex-notes-file "~/.emacs.d/annotation.org")

[tmalsburg]

I considered storing all notes in one file when I first wrote helm-bibtex. The reason why I decided against it is that in my experience Emacs can be really slow with bigger org files. So I thought that the one-file solution may not be scaleable.

Are there any true benefits of having all notes in one file or is this just a matter of personal preference?

If a one-file solution were to be added, I think we would simply recycle the variable helm-bibtex-notes-path. If it points to a directory, make one file per publication. If it points to an org-file, store the notes in that file. For that file, I would probably prefer to have a structure like the following:

* DoeEtAl2013
Here are my notes on Jon Doe's recent paper: ...

where DoeEtAl2013 is the BibTeX key. While this is perhaps less human-readable, it has the benefit that it avoids the property drawer, which seem to slow down Emacs a lot. (I believe org-contacts is so unbearably slow because it heavily relies on properties.)

I'm just thinking aloud here. I'm not planning to work on this and I may not even accept a pull-request. At the moment, it seems to me that the one-file approach is neither better nor worse than the approach using multiple files and I don't see why we need two solutions for storing notes.

[jagrg]

The reason for using one file would be to extend the flexibility of helm-bibtex. People have different approaches. Some work better than others. Personally, I've always used one file and never had problems with speed, even when having to retrieve from drawers.

Some of the benefits of using one file is to be able to (1) export all or some of the notes to html, pdf etc.; (2) search with helm-swoop or occur/org-occur; (3) sort and move notes to different subtree categories. Another advantage is that the note heading and metadata are added automatically using the information provided in the bib file, which avoids having to name files in a separate directory.

[tmalsburg]

Ok, convinced, and I may actually switch to the one-file approach when the code is ready. I will probably introduce a customization variable that allows users to define their own template for notes. Something like the following could be used to implement your proposal.

(defcustom
  helm-bibtex-notes-template
  "\n* ${author} (${year}): ${title}\n  :PROPERTIES:\n  :BIBTEX-KEY: ${=key=}\n  :END:\n"
  "Template for new notes."
  :group 'helm-bibtex
  :type 'string)

[tmalsburg]

Could you please test the version in the new branch note-files? To configure it, all you have to do is to set helm-bibtex-notes-path to a file. The default template (defined in helm-bibtex-notes-template) assumes that this file is an org file but you could also use markdown or whatever you want. If there already is a note for the publication, the cursor jumps to that note (and unfolds the org node if org-mode is on). If there is no note for the file, the template is used to create one at the end of the file.

One issue: when existing notes are searched, the code simply looks for the first occurrence of the BibTeX key in the notes file. If that key is used in the notes of another publication ("This paper reports a failed replication of XYZ", where XYZ is the BibTeX key), you might land there. Not sure how to deal with this.

[jagrg]

I tested and it's working fine, expect the subtrees are not showing. This is what I would expect to see:

** Author (2013) Title

bla bla

*** chapter 1...
*** chapter 2...

This would be my suggestion:

(if (re-search-forward (concat "\\b" key "\\b") nil t)
    (when (eq major-mode 'org-mode)
      (org-show-entry)
      (show-children)
      (outline-previous-visible-heading 1)
      (recenter-top-bottom 0))

I'm not sure I understand your issue. Why would you reuse the key?

[tmalsburg]

Here's an example:

* Author1 (year1): Title1
  :PROPERTIES:
  :BIBTEX-KEY: Key1
  :END:

As was shown in [[/path/to/pdf/Key2.pdf][Key2]], ...

* Author2 (year2): Title2
  :PROPERTIES:
  :BIBTEX-KEY: Key2
  :END:

The re-search-forward would position the cursor not at the entry for Key2 but in the notes about Key1 where Key2 is referenced.

[tmalsburg]

In a89bde5, I added something similar to what you suggest: the complete subtree for the publication's notes is made visible and all other trees are hidden.

[jagrg]

I see the problem. Could you exclude the brackets from the search?

[BrianZbr]

Before I saw this issue, I made a hack: http://pastebin.com/r8NuSNKv I will have to give the branch a try.

As for the search problem, why not use ":BIBTEX-KEY: Key2" instead of just "Key2"?

[BrianZbr]

Just made a pull request (my first one ever!) but there is another issue with this branch that I couldn't solve. The ${author} part of the default template does not work for me. I get a message like 'Wrong type argument: sequencep, 3'. The number corresponds to the length of the author's last name. Any ideas how to fix this?

[tmalsburg]

Thanks for the PR. I'll look at it on the weekend. The reason why I hesitated to make a decision is that, once people start using a certain format, making changes is going to be hard.

[BrianZbr]

I see... Well for now anyway, aside from the ${author} thing, this branch is working very nicely for me and I'm very glad to have this feature!

[tmalsburg]

Still thinking about the best way to use and detect BibTeX keys in the notes file. What about this solution:

* [[/path/to/pdf][Author (year): Title]]                      :TheBibTeXKey:

I think this is an appropriate use for Org tags and if we search for :TheBibTeXKex: the chance of false positives is reasonably low. (And perhaps Org already has facilities for jumping to a headline with a specific key.) I like this solution better than the one using property drawers which I find incredibly clunky. Opinions?

[jagrg]

That's a big change. As far as I know, most people are using properties, which doesn't mean that it's any better. I'd be happy to give the tag method a try and see how it goes. Another solution would be to implement both methods, so that if no tags are found, helm-bibtex would look in the properties drawer.

[tmalsburg]

The code for the property drawers is not yet merged into the official version. So I doubt that most people are using drawers at the moment. However, I understand that it would be inconvenient for you to transition to a solution using tags. If I decide to implement it with tags, I will try to make it easy for you to stick with property drawers. Shouldn't be too difficult.

[jagrg]

Helm-bibtex provides a very nice (and fast) interface for managing bibliographies, but people who use it might use it in conjunction with other tools, or hacking their own solutions for retrieving notes. One could use org-ref to retrieve notes and helm-bibtex to open pdfs, for example, so I think the community would benefit from both options. Which ever way you decide, let us know. I'd be happy to test it.

[tmalsburg]

I don't use org-ref. Are you saying that org-ref already has support for notes and that it is using property drawers?

[jagrg]

It does support notes. As far as using properties drawer, I believe so.

[llcc]

Because I saw you guys talking about the org-ref notes properties. I just give a screenshot of one note entry generated by org-ref. Hope it will help you know it.

[tmalsburg]

Thank you, @llcc. I don't understand why they duplicate the information from the BibTeX entries. If something changes (e.g., a paper goes from in press to published online, or from published online to published in print with page numbers etc.) you have to change the BibTeX entry and the notes file. Also, I don't see a BibTeX key (the name Custom_ID suggests that this may not be the BibTeX key but I'm not sure). I don't think this is going to be the default format in helm-bibtex but I will probably implement something that makes it easy to customize the format for people who prefer the org-ref style.

[jagrg]

I'm also not sure why it's duplicated, unless it's able to sync with the bib file, which I doubt. The only information really needed is the bibtex-key under Custom_ID, used to retrieve the notes, and maybe the URL. In any case, the process of viewing notes could be simplified if the most cited, and most recently cited, authors were on top, but that's another subject.

[tmalsburg]

I updated the note-files branch. There are several changes:

When one big notes file is used:

• The new default template is simpler than what org-ref uses but it should be compatible because it uses the Custom_ID property to store the key.
• The list of publications now shows a mark if notes are available for a publication.
• Finding the correct entry is now robust. The code looks for Custom_ID: key not just for the key.

Multiple notes file (one for each publication):

• The template for new notes is used to populate new note files. Adresses Feature request: Option to auto-populate orgmode notes file #46.

Please test and let me know if that works for you. If everything works as expected, I will merge these changes into master. Thank you.

[jagrg]

I get an error

Debugger entered--Lisp error: (void-function outline-hide-sublevels)
  (outline-hide-sublevels 1)
  (progn (outline-hide-sublevels 1))
  (if (eq major-mode (quote org-mode)) (progn (outline-hide-sublevels 1)))

[tmalsburg]

Thanks for testing! Which version of Emacs are you using? I'm on the development version.

[jagrg]

24.5.1

[tmalsburg]

@jagrg It should work now. I was using the new names of some functions in outline.el. See 92fda4c.

[jagrg]

I'm having a few problems. Pressing f9 inserts a new note template at the top of the page, above #+TITLE:. You probably want as the first heading instead, below #+TITLE:. It's also not finding notes and I'm not seeing the notes icon displayed.

[tmalsburg]

Thanks for the feedback. Another user also reported these issues. So far, I haven't been able to reproduce them on my system (apart from inserting at the top, of course). Weird. I'll try with an older version of Emacs.

[jagrg]

In jagrg@6c82d40, I was able to make the notes-file branch work. With helm-bibtex-notes-path set to a single file, I can now open the notes correctly, and if the key is not found, a new heading is created at the end of the document.

[tmalsburg]

Hm, I can't see what your changes were. Can you commit your changes into the current note-files branch (without changes from master) instead of making a new branch?

[jagrg]

The notes-file branch was too old to merge.

[tmalsburg]

There were a couple of problems which I hope are resolved in this commit: 282f507

[jagrg]

Looks good, Titus, except you brought back the problem of templates being inserted on top of the file. I would suggest replacing org-show-subtree with org-show-entry to avoid drawers from being expanded. In any case, I got better results with jagrg@f0b0df8, although the template is being inserted before the last heading. I'm not sure why. Maybe you have a better idea.

[tmalsburg]

To be honest, I chose insertion at the top because it didn't work at the bottom. I think I'm hitting a bug in Emacs but don't have the nerve or time to track it down.

Ok, regarding org-show-entry.

[jagrg]

I also think it's a bug. In any case, my notes branch works well with a single file, so maybe we could try splitting the function with a variable to choose between one or multiple files. This means that when the variable is set to one-file, only the one-file function is called. Just a thought. No pressure!

(defun helm-bibtex-edit-notes ()
  (if (eq helm-bibtex-use-multiple-files t)
      (helm-bibtex-edit-notes-from-multiple-files)
    (helm-bibtex-edit-notes-from-single-file)))

[tmalsburg]

Appending at the end now seems to work reliable. I have a vague sense of what the problem was. Probably an Emacs bug.

The code now uses org-show-subtree but follows up with org-cycle-hide-drawers.

My impression that this feature is now finally complete and working correctly. Any objections against merging into master?

[jagrg]

You solved the puzzle. Thanks!

[tmalsburg]

I added this feature in 5d028b9. I did not merge the branch because the commit history was fairly messy. Thanks everyone for your input.