Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Org 9.7 org-element API incompatible changes #364

Closed
muntean-alex opened this issue Aug 2, 2023 · 34 comments
Closed

Org 9.7 org-element API incompatible changes #364

muntean-alex opened this issue Aug 2, 2023 · 34 comments
Assignees
Milestone

Comments

@muntean-alex
Copy link

Good evening all,
I've had a small agenda-like org-ql setup that I used with org-mode for a while.
After not using it for a month or two, I've gotten back to using it and I've discovered that it throws an error.

"replace-regexp-in-string: Wrong type argument: stringp, [org-element-deferred org-element--headline-parse-title (nil) nil]"

(defun kr/agenda-next-month ()
  (interactive)
  (require 'org-ql)
(org-ql-search (org-agenda-files)
  '(and (or (deadline :to 30)
            (scheduled :to 30)
            (todo :to 30))
        (not (done))
        (not (tags "tag"))
        )
  :title "My next month"
  :super-groups '(
                  (:name "Today"
                         :scheduled today
                         :deadline today
                         :date today)
                  (:name "Due soon"
                         :scheduled future
                         :deadline auto)
                  (:name "Overdue"
                         :scheduled past
                         :deadline past)
                  (:name "Bills soon"
                         :tag "bills")
                  (:name "Due later"
                         :scheduled future
                         :deadline future)
                  (:name "Projects"
                         :tag "projects")
                  (:name "Misc"
                         :todo "TODO")
                  (:todo "READ"
                         :order 7)
                  (:todo "WAIT"
                         :order 6)
                  (:priority "A" :order 1)
                  (:priority "B" :order 2)
                  (:priority "C" :order 3)))
)

Anybody facing something similar?
Any hints on tracking down this issue?

Thank you kindly!

@yantar92
Copy link
Contributor

yantar92 commented Aug 2, 2023

radian-software/straight.el#1107

TL;DR: If you upgraded Org mode recently, you may need to clear your straight/build directory and re-compile.

@muntean-alex
Copy link
Author

Hello!
I've nuked the straight directory and rebuilt with doom upgrade
Sadly, no cigar.
Gives me the same error.

@yantar92
Copy link
Contributor

yantar92 commented Aug 2, 2023

Hmm. Then, may you do M-x debug-on-error and post the full backtrace?

@muntean-alex
Copy link
Author

Debugger entered--Lisp error: (wrong-type-argument stringp [org-element-deferred org-element--headline-parse-title (nil) nil])
  string-match("\\[\\[\\(\\(?:[^][\\]\\|\\\\\\(?:\\\\\\\\\\)*[][]\\|\\\\+[^][]\\)+\\)..." [org-element-deferred org-element--headline-parse-title (nil) nil] 0)
  replace-regexp-in-string("\\[\\[\\(\\(?:[^][\\]\\|\\\\\\(?:\\\\\\\\\\)*[][]\\|\\\\+[^][]\\)+\\)..." #f(compiled-function (m) #<bytecode -0x8ce38b99175cad3>) [org-element-deferred org-element--headline-parse-title (nil) nil] nil t)
  org-link-display-format([org-element-deferred org-element--headline-parse-title (nil) nil])
  org-ql-view--format-element((headline (:standard-properties [667 707 679 707 0 667 (:title) nil nil 681 705 nil nil 1 nil [org-element-deferred org-element--headline-deferred nil t] nil #<buffer inbox.org>] :pre-blank 0 :raw-value [org-element-deferred org-element--headline-parse-title (nil) nil] :title [org-element-deferred org-element--headline-parse-title (nil) nil] :level [org-element-deferred org-element--headline-parse-title (nil) nil] :priority [org-element-deferred org-element--headline-parse-title (nil) nil] :tags [org-element-deferred org-element--headline-parse-title (nil) nil] :todo-keyword [org-element-deferred org-element--headline-parse-title (nil) nil] :todo-type [org-element-deferred org-element--headline-parse-title (nil) nil] :footnote-section-p [org-element-deferred org-element--headline-parse-title (nil) nil] :archivedp [org-element-deferred org-element--headline-parse-title (nil) nil] :commentedp [org-element-deferred org-element--headline-parse-title (nil) nil] :org-marker #<marker in no buffer> :org-hd-marker #<marker in no buffer>)))
  mapcar(org-ql-view--format-element ((headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) ...))
  -map(org-ql-view--format-element ((headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) (headline ...) ...))
  org-ql-search(("[[myfiles.org]]") (and (or (deadline :to 30) (scheduled :to 30) (todo :to 30)) (not (done)) (not (tags "tag"))) :title "My Agenda View" :super-groups ((:name "Today" :scheduled today :deadline today :date today) (:name "Due soon" :scheduled future :deadline auto) (:name "Overdue" :scheduled past :deadline past) (:name "Bills soon" :tag "bills") (:name "Due later" :scheduled future :deadline future) (:name "Projects" :tag "projects") (:name "Misc" :todo "TODO") (:todo "READ" :order 7) (:todo "WAIT" :order 6) (:priority "A" :order 1) (:priority "B" :order 2) (:priority "C" :order 3)))
  kr/agenda-next-month()
  funcall-interactively(kr/agenda-next-month)
  call-interactively(kr/agenda-next-month nil nil)
  command-execute(kr/agenda-next-month)

@yantar92
Copy link
Contributor

yantar92 commented Aug 2, 2023

I see now. This is because org-ql-view--format-element relies on internal org-element representation, which has been changed in the cutting edge Org.

@alphapapa Please check out https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/etc/ORG-NEWS#n40

@yantar92
Copy link
Contributor

yantar92 commented Aug 2, 2023

@muntean-alex Meanwhile, you may need to downgrade Org mode. org-roam/org-roam#2361 has a lot of instructions how to do it.

@alphapapa
Copy link
Owner

alphapapa commented Aug 2, 2023

I see now. This is because org-ql-view--format-element relies on internal org-element representation, which has been changed in the cutting edge Org.

@alphapapa Please check out https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/etc/ORG-NEWS#n40

ISTM that this change, while it may be an improvement in several ways, is going to break a LOT of third-party packages, like this one. The org-element sexp representation has been unchanged for many years and versions. I feel like there should be a transitional period with the new representation under a different API to preserve compatibility with other code in the wild.

I mean, thinking of all the places in org-ql and org-super-agenda that will likely have to be changed for this, and have multiple code paths by Org version, for YEARS to come, because of this change... the various org-ql predicates and org-super-agenda group selectors... It makes me want to import the old org-element API and just use that instead.

On top of that, there is no guidance for developers to adapt their code to the new API, no helper functions or macros...? This is looking like many, many hours of work and hundreds or thousands of lines of extra code to work with multiple Org versions and APIs, and that cluttering up the packages' source code for years until the old Org versions are truly not used anymore.

Version 9.7 (not released yet)

Ok, so, please, please, PLEASE, do not release Org 9.7 with this change as-is. There has got to be some kind of compatibility layer for downstream libraries. This burden can't just be dumped on downstream developers and users.

Or if I'm wrong and this is all much simpler than it appears, then please educate me, and please improve the documentation accordingly.

@alphapapa alphapapa changed the title Error when I try to run custom agenda-like org-ql command Org 9.7 org-element API incompatible changes Aug 2, 2023
@yantar92
Copy link
Contributor

yantar92 commented Aug 3, 2023 via email

@muntean-alex
Copy link
Author

@muntean-alex Meanwhile, you may need to downgrade Org mode. org-roam/org-roam#2361 has a lot of instructions how to do it.

For reference, this worked for me and fixed the issue.
Thank you very much!

@maikol-solis
Copy link

I had the same issue, and I thought that it was something to do with org-ql-block. I reported the problem here #38 (comment).

However, downgrading org to 9.6, solved completely the problem. In doom, I added this line into my packages.el:

(package! org :pin "806abc5a2bbcb5f884467a0145547221ba09eb59")

Best.

alphapapa added a commit that referenced this issue Sep 4, 2023
@alphapapa
Copy link
Owner

Thanks to all for reporting. Please try the commit I just pushed on the wip/364 branch and let me know if that works on Org 9.7.

@gdindi
Copy link

gdindi commented Sep 11, 2023

Hi,
I have tried to replace org-ql-viel.el with the version in the wip/364 branch and I keep getting this error:

(void-function org-ql-view--resolve-element-properties)

I have byte-compiled and native-compiled (I am using native-comp). I am doing something wrong but I don't see it.

What is the recommended way of testing this? My org-ql is installed with package-install and I am using "20230908.722".

Thanks.

@alphapapa
Copy link
Owner

@gdindi You shouldn't try to mix and match files from different commits. You should follow the instructions in the readme to install with Quelpa, adding this to the recipe form: :branch "wip/364".

@alphapapa alphapapa added the bug label Sep 11, 2023
@gdindi
Copy link

gdindi commented Sep 11, 2023

Makes sense. Thanks. I unistalled org-ql with package-delete and installed with this in my init.el:

(quelpa
 '(org-ql 
           :fetcher github
           :repo "alphapapa/org-ql"
           :branch "wip/364"
           :files (:defaults (:exclude "helm-org-ql.el"))))
(require 'org-ql)

The package is installed in ~/.emacs.d/elpa/org-ql-20230904.214910. I restarted Emacs, but I still get this backtrace:

(void-function org-ql-view--resolve-element-properties)
org-ql-view--resolve-element-properties((headline (:standard-properties [699314 699314 699346 699691 699691 0 (:title) nil nil nil nil 699348 699691 2 #<buffer biblio.org> [org-element-deferred org-element--headline-deferred nil t] nil nil] :pre-blank 0 :raw-value [org-element-deferred org-element--headline-parse-title (nil) nil] :title [org-element-deferred org-element--headline-parse-title (nil) nil] :level [org-element-deferred org-element--headline-parse-title (nil) nil] :priority [org-element-deferred org-element--headline-parse-title (nil) nil] :tags [org-element-deferred org-element--headline-parse-title (nil) nil] :todo-keyword [org-element-deferred org-element--headline-parse-title (nil) nil] :todo-type [org-element-deferred org-element--headline-parse-title (nil) nil] :footnote-section-p [org-element-deferred org-element--headline-parse-title (nil) nil] :archivedp [org-element-deferred org-element--headline-parse-title (nil) nil] :commentedp [org-element-deferred org-element--headline-parse-title (nil) nil] :org-marker #<marker at 699314 in biblio.org> :org-hd-marker #<marker at 699314 in biblio.org>)))
org-ql-view--format-element((headline (:standard-properties [699314 699314 699346 699691 699691 0 (:title) nil nil nil nil 699348 699691 2 #<buffer biblio.org> [org-element-deferred org-element--headline-deferred nil t] nil nil] :pre-blank 0 :raw-value [org-element-deferred org-element--headline-parse-title (nil) nil] :title [org-element-deferred org-element--headline-parse-title (nil) nil] :level [org-element-deferred org-element--headline-parse-title (nil) nil] :priority [org-element-deferred org-element--headline-parse-title (nil) nil] :tags [org-element-deferred org-element--headline-parse-title (nil) nil] :todo-keyword [org-element-deferred org-element--headline-parse-title (nil) nil] :todo-type [org-element-deferred org-element--headline-parse-title (nil) nil] :footnote-section-p [org-element-deferred org-element--headline-parse-title (nil) nil] :archivedp [org-element-deferred org-element--headline-parse-title (nil) nil] :commentedp [org-element-deferred org-element--headline-parse-title (nil) nil] :org-marker #<marker at 699314 in biblio.org> :org-hd-marker #<marker at 699314 in biblio.org>)))

I don’t understand the void function message, since the code contains the defalias, but I don’t understand how defalias works wrt defun.

I may be introducing more noise than help in this issue. I just wanted to test seing that the OP had found another solution and that nobody seemed to have tested your fix.

If I can be of any help, do not hesitate to tell me what to do, but I don’t want to pollute the issue if the problem comes from my config.

Thanks.

alphapapa added a commit that referenced this issue Sep 11, 2023
@alphapapa
Copy link
Owner

@gdindi Thanks for testing that. I think I made a mistake, wrapping the defalias call in eval-and-compile (a "thinko," I think). I just pushed to that branch again. Please try again and let me know if it works. (You may find it useful to use this command to easily upgrade the Quelpa-installed package.)

@gdindi
Copy link

gdindi commented Sep 12, 2023

Hi. I just tried and keep getting the same error.

@alphapapa
Copy link
Owner

Hi. I just tried and keep getting the same error.

Did you delete other versions of org-ql and restart Emacs before testing?

@gdindi
Copy link

gdindi commented Sep 12, 2023 via email

@alphapapa
Copy link
Owner

alphapapa commented Sep 12, 2023

Looking more closely at the backtrace, the element seems to have no :begin property at all. It looks like the value might be in the :standard-properties vector as an integer position rather than as a marker.

But I also see what appear to be some still-deferred properties, which I don't understand, because we should be forcing them to resolve here:

org-ql/org-ql-view.el

Lines 821 to 833 in 1bd7bb4

(defalias 'org-ql-view--resolve-element-properties
;; It would be preferable to define this as an inline function, but
;; that would mean that users would have to recompile org-ql when
;; upgrading to Org 9.7 or else get weird errors.
;; TODO(someday): Define `org-ql-view--resolve-element-properties' as inline.
(if (version<= "9.7" org-version)
(lambda (node)
"Resolve NODE's properties using `org-element-properties-resolve'."
;; Silence warnings about `org-element-properties-resolve'
;; being unresolved on earlier Org versions.
(with-no-warnings
(org-element-properties-resolve node 'force-undefer)))
#'identity))

Maybe the problem is that the :begin property was always an integer, so org-is-habit-p was always being called in the wrong buffer, and the error is just being revealed by org-element-at-point being changed to signal an error when it's not called in an org-mode buffer now.

@gdindi Please try the commit I just pushed which changes it to use the heading marker instead. (And thank you for your help testing this!)

@gdindi
Copy link

gdindi commented Sep 13, 2023

I will pretend I understand your explanation ;)

Good news: it works at least with my custom searches that I use most frequently. I will keep on testing in case I find other issues.

Thanks!

@yantar92
Copy link
Contributor

yantar92 commented Sep 13, 2023 via email

@yantar92
Copy link
Contributor

yantar92 commented Sep 13, 2023 via email

@alphapapa
Copy link
Owner

@yantar92 Thanks. (And apologies for the ping; I realized afterward that I didn't need to bother you about this, but it was too late.)

@alphapapa alphapapa modified the milestones: 0.7.2, 0.8 Sep 14, 2023
@alphapapa
Copy link
Owner

@gdindi Thanks. I'll target this change for v0.8 of org-ql, since it's not a fix per se, and since Org 9.7 is yet to be released. Please do let me know if you find any problems.

@alphapapa
Copy link
Owner

I've pushed these fixes to master under v0.8-pre. I hope they resolve this issue, but I'll leave it open for a while in case more problems are found.

To all: Please test and let me know if the problem is resolved for you. Thanks.

@kunalomizer
Copy link

I just upgraded and it appears to be working now. The org-ql query does execute and display the results as expected. Thank you!
Although I do still see the error in Messages : Error running timer ‘org-reveal’: (error "‘org-element-at-point’ cannot be used in non-Org buffer #<buffer Org QL View: MWTasks> (org-agenda-mode)") [19 times]

@alphapapa
Copy link
Owner

@kunalomizer Thanks. I don't know what's going on with that error from a timer. org-reveal is called from this hook:

(defcustom org-ql-find-goto-hook '(org-show-entry org-reveal)

But that's not called from a timer:

(run-hook-with-args 'org-ql-find-goto-hook))))

And I don't know what would be calling org-element-at-point in the org-ql-view buffer.

@treitmaier
Copy link

The error was also raised as an org-roam issue here: org-roam/org-roam#2391

@alphapapa
Copy link
Owner

AFAICT this is fixed. Please file a new issue if it happens again after I tag v0.8.

@TristanCacqueray
Copy link

@alphapapa not sure if that is expected, but the entries returned by org-ql-select (Version: 20240905.2358) may still contain deferred values. In my case I was reformatting them like this:

(defun one/format (item)
  (let* ((properties (cadr item))
         (title (plist-get properties :raw-value))
         (scheduled (plist-get properties :scheduled))
         (ts (format-time-string "%FT%T%z" (org-timestamp-to-time scheduled))))
    (format "%s %s" ts (or/remove-links title))))

This was broken by s-trim: Wrong type argument: stringp, [org-element-deferred org-element--headline-parse-title (nil) t], and I needed this tweak to fix the bug:

-(defun one/format (item)
-  (let* ((properties (cadr item))
+(defun one/format (node)
+  (let* ((item (org-element-properties-resolve node 'force-undefer))
+         (properties (cadr item))

@alphapapa
Copy link
Owner

@TristanCacqueray The commit that resolves this issue, 8250349, resolves the deferred node properties in the function org-ql-view--format-element, which is used in org-ql-view views. If you are directly calling org-ql-select and handling the org-element nodes yourself, you will need to handle property resolution yourself. It would not be appropriate for org-ql-select to do so automatically, because in some cases it could be unnecessary, which would cause an unnecessary performance hit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants