Todo.txt advanced search system #1901

harshad1 · 2022-10-17T03:22:42Z

In this PR I have created a fully functional advanced search / filter system for todo.txt

For example @home & (pri:A | pri:B) will return tasks with priority A or B and which have the context @home. The existing filter system has been updated to work on top of this query language (i.e. by generating the appropriate query).

This is now fully functional and works correctly.

Remaining tasks:

Document the query language
Add more unit tests
More testing

Other fixes in this PR include:

Done file is now remembered per file (each todo.txt file can have it's own done file)
If the keyboard was open when a dialog was opened, we will open the keyboard again when the dialog is closed (this is only enabled in some dialogs where it seemed to be useful / easy to implement)

app/src/main/java/net/gsantner/markor/frontend/MarkorDialogFactory.java

app/src/main/java/net/gsantner/opoc/frontend/GsSearchOrCustomTextDialog.java

harshad1 · 2022-10-18T05:49:32Z

Description of the advanced filter syntax

I will add this to the source and a few other places soon.

The todo.txt advanced filter syntax allows one to filter todo items using a simple query language.

For example (pri:A | pri:B) & +home will match tasks with priority A or priority B and which have the project +home

Filter types

Priority: pri:[(A through Z)].
- pri:A will match tasks with priority A
- pri will match any task with a priority
- !pri with therefore match tasks with no priority
Context: Words starting with '@' will match contexts. @ alone will match tasks with any context
- Use !@ to match tasks with no contexts
Project: Words starting with '+' wll match projects. + alone will match tasks with any project
- Use !+ to match tasks with no projects
Due date:
- 'due' will match tasks with any due date
  - Use !due to match tasks with no due date
- 'due=' will match tasks with due date today
- 'due<' or will match tasks with due date which has already passed (overdue)
- 'due>' will match tasks which are due in the future
Done: the word done will match done tasks

Operators

Operation	Symbol	Examples	Translation
NOT	`!`	!done	not done
AND	`&`	pri:A & due=	priority A and due today
OR	`\|`	pri:C \| pri:D \| !pri	priority C or priority D or no priority

( and ) can be used to group expressions
Each element in the expression must be surrounded by spaces. Exceptions are:
1. No need for a space after open parenthesis
2. No need for a space before closing parenthesis
3. No need for space following !
4. No need for spaces at the beginning and end of the query

Evaluation

The filter expression is evaluated left to right (i.e. 'and' and 'or' have the same precedence). 'Not' is applied to the result of the expression to it's immediate right.

Done tasks are not excluded by default. They must be excluded explicitly.

Examples:

!+ & !@ & !pri & !done - No projects, no contexts, no priority and not done
!+ | !@ & !done - will match tasks which have either no projects or no contexts and which are not done
- i.e. this is equivalent to (!+ | !@) & !done

A note on implementation:

A series of preprocessing stages are used to split the query into a sequence of symbols - operators and queries
Each query of the expression is evaluated to construct a boolean expression. For example the expression (pri:C | pri:D | ~) & !done applied to the todo D 2022-01-01 foo will evaluate to the string "( F | T | F ) & !F" (spaces added here for clarity)
The boolean expression is then evaluated to 'T' or 'F' using a simple stack based algorithm
Any errors (syntax errors, mismatched parenthesis etc) will cause a task to evaluate to 'F' (i.e. false)

harshad1 · 2022-10-22T15:33:55Z

This is basically ready to go. TBD:

Clean up the description above and put it somewhere accessible.
Link to the description from the advanced search menu
Make saved filters copyable
Add a few example filters

gsantner

My suggestion: Reduce the amount of possible operators to minimum. This makes it later more easy to migrate to something else if required, and to support by other means (uh oh, suddenly somebody comes up with the idea to use this operator stuff somewhere else?!).

What happens when "or" "and" "AND" "OR" are reserved keyword is also that they are excluded from being usable (you can't search for todos containing "and"). With it also comes, how to deal with translations, why can't I as a german user type "blau oder gruen und gelb"??!^^

So I suggest to keep it simple: Boolean operators, no logical keywords generally. For todo.txt specific case with "today", "nopriority" - this is fine, and we also can't magically invent a operator for them 😃.

harshad1 · 2022-10-24T23:31:59Z

Yeah. I see your point. I'll make the change when I am able - probably not this week :(

I thought nopriority etc were acceptable as we already use 'due' etc as keywords. But I think I know of a way around even that.

gsantner · 2022-10-25T02:55:15Z

No stress, and make it like you want it. It's my suggestion, not the must.

For due etc I think thats fine

harshad1 · 2022-11-03T06:34:01Z

I've updated this to reduce the reliance on special words. Still not happy with ~ for no priority and x for done. Will need to think more about this.

gsantner · 2022-11-03T13:03:39Z

It's OK if it's not perfect, it's never perfect^^. So if it's some todo specific keywords, thats fine

harshad1 · 2022-11-04T04:00:24Z

I know. I just want to get it to a stage where I am satisfied (after all, I am building this for me :p)

harshad1 · 2022-11-04T05:57:35Z

Updated. I think I am happy with this form.

Of interest: I found this a couple days ago: https://github.com/ransome1/sleek/wiki/Filter-Expressions-for-Advanced-Search

The syntax is not exactly the same, but it is close enough for comparison / inspiration

gsantner · 2022-11-05T02:35:31Z

app/src/main/java/net/gsantner/markor/model/AppSettings.java

@@ -355,6 +351,22 @@ public void toggleFavouriteFile(File file) {
    private static final String PREF_PREFIX_AUTO_FORMAT = "PREF_PREFIX_AUTO_FORMAT";
    private static final String PREF_PREFIX_VIEW_SCROLL_X = "PREF_PREFIX_VIEW_SCROLL_X";
    private static final String PREF_PREFIX_VIEW_SCROLL_Y = "PREF_PREFIX_VIEW_SCROLL_Y";
+    private static final String PREF_PREFIX_TODO_DONE_NAME = "PREF_PREFIX_TODO_DONE_NAME";
+
+    public void setLastTodoDoneName(final String path, final String name) {


setPerFileTodoTxtDoneFilepath ?

We force the done file to be in the same folder as the todo.txt file, so it is a name and not a path. And the other per file functions are all setLast*. We can change this, but I think we should change them all if we change this one.

Oh I thought you did some change to select the file from anywhere, didn't check that

app/src/main/java/net/gsantner/markor/frontend/textview/TextViewUtils.java

app/src/main/java/net/gsantner/markor/format/todotxt/TodoTxtFilter.java

harshad1 · 2022-11-11T07:14:05Z

@gsantner Not sure if you are waiting for me to address any issues here. Please let me know if you are.

Thanks
H

harshad1 added 4 commits October 16, 2022 10:50

Query system

8de9d66

Tweaks to display etc

bb557ce

Some documentation and exception handling

7d7a659

Cleanups

b0918d3

harshad1 commented Oct 17, 2022

View reviewed changes

app/src/main/java/net/gsantner/markor/frontend/MarkorDialogFactory.java Show resolved Hide resolved

harshad1 commented Oct 17, 2022

View reviewed changes

app/src/main/java/net/gsantner/markor/frontend/MarkorDialogFactory.java Show resolved Hide resolved

harshad1 commented Oct 17, 2022

View reviewed changes

app/src/main/java/net/gsantner/opoc/frontend/GsSearchOrCustomTextDialog.java Show resolved Hide resolved

harshad1 commented Oct 17, 2022

View reviewed changes

app/src/main/java/net/gsantner/opoc/frontend/GsSearchOrCustomTextDialog.java Show resolved Hide resolved

harshad1 changed the title ~~Todo.txt query language and sytem~~ WIP: Todo.txt query language and sytem Oct 17, 2022

More cleanups

514d41b

Order of saved filters, simpler syntax

0418173

harshad1 changed the title ~~WIP: Todo.txt query language and sytem~~ Todo.txt advanced search system Oct 21, 2022

Fixing tests

bcb9c46

gsantner reviewed Oct 24, 2022

View reviewed changes

gsantner assigned harshad1 Oct 25, 2022

gsantner added the hacktoberfest label Oct 25, 2022

Made query keywords language independant

9a7ef14

Using correct done

859b756

using differnt keywords

d8141ad

gsantner reviewed Nov 5, 2022

View reviewed changes

gsantner and others added 2 commits November 5, 2022 04:19

reformat code

ce7658d

Merge branch 'master' into query_system

f4b8901

gsantner added this to the Markor v3.0 milestone Nov 11, 2022

gsantner added enhancement feature labels Nov 11, 2022

gsantner approved these changes Nov 11, 2022

View reviewed changes

gsantner merged commit c7616f7 into gsantner:master Nov 11, 2022

harshad1 deleted the query_system branch October 10, 2023 05:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Todo.txt advanced search system #1901

Todo.txt advanced search system #1901

harshad1 commented Oct 17, 2022 •

edited

Loading

harshad1 commented Oct 18, 2022 •

edited

Loading

harshad1 commented Oct 22, 2022

gsantner left a comment •

edited

Loading

harshad1 commented Oct 24, 2022 •

edited

Loading

gsantner commented Oct 25, 2022

harshad1 commented Nov 3, 2022

gsantner commented Nov 3, 2022 •

edited

Loading

harshad1 commented Nov 4, 2022

harshad1 commented Nov 4, 2022

gsantner Nov 5, 2022

harshad1 Nov 5, 2022

gsantner Nov 5, 2022

harshad1 commented Nov 11, 2022

Todo.txt advanced search system #1901

Todo.txt advanced search system #1901

Conversation

harshad1 commented Oct 17, 2022 • edited Loading

harshad1 commented Oct 18, 2022 • edited Loading

Description of the advanced filter syntax

Filter types

Operators

Evaluation

Examples:

A note on implementation:

harshad1 commented Oct 22, 2022

gsantner left a comment • edited Loading

Choose a reason for hiding this comment

harshad1 commented Oct 24, 2022 • edited Loading

gsantner commented Oct 25, 2022

harshad1 commented Nov 3, 2022

gsantner commented Nov 3, 2022 • edited Loading

harshad1 commented Nov 4, 2022

harshad1 commented Nov 4, 2022

gsantner Nov 5, 2022

Choose a reason for hiding this comment

harshad1 Nov 5, 2022

Choose a reason for hiding this comment

gsantner Nov 5, 2022

Choose a reason for hiding this comment

harshad1 commented Nov 11, 2022

harshad1 commented Oct 17, 2022 •

edited

Loading

harshad1 commented Oct 18, 2022 •

edited

Loading

gsantner left a comment •

edited

Loading

harshad1 commented Oct 24, 2022 •

edited

Loading

gsantner commented Nov 3, 2022 •

edited

Loading