Extremely Slow to Sync Tasks

[iarxm]

Great plugin, almost exactly what I've been looking for.

It's extremely slow to call all of the syncing related commands on my setup. I don't have much compute on an x240, however, this was even the case for updating a single task, which might take some 10+ seconds.

This seems to be mostly fixed by removing 'render_virtual_due_dates'.

That function seems to freeze my system, even when just switching buffers due to it being invoked in the autocmds.

It's still extremely slow to sync the whole buffer with that function removed. If I have approximately 40 tasks in a buffer, it can take more than 30/40 seconds. The bulk of which seems to be 'add_or_sync_task'.

This is a tiny operation so my assumption would be that a reasonable expected duration for such a command would be closer to 1 second.

[huantrinh1802]

Hi @zbyyt, thanks for the feedback. I am going to add an option for enabling rendering virtual text.

Beside that, could you confirm that you have

vim.api.nvim_create_autocmd({ "BufEnter", "BufWritePost" }, {
        group = vim.api.nvim_create_augroup("TWTask", { clear = true }),
        pattern = "*.md,*.markdown", -- Pattern to match Markdown files
        callback = function()
          vim.cmd('TWSyncTasks')
        end,
      })

in the plugin setup? If yes, I would recommend to remove it from your config and manually sync tasks when needed

[iarxm]

Nice. However I would still expect virtual text rendering to run smoothly.

On the autocmd, yes I have that disabled. I tried quick manual tests disabling various functions in utils and the time seemed to be mostly coming from 'add_or_sync_task'. It seems to take about 1 second per task in the buffer, so the time to sync the whole buffer seems to scale linearly with the task count.

[huantrinh1802]

It seems to take about 1 second per task in the buffer, so the time to sync the whole buffer seems to scale linearly with the task count.

This is true, because for every task, the plugin will perform similar number of operations. A possible cause is that the plugin currently utilise the CLI interface to interact with Taskwarrior.

One potential solution is to use async so it doesn't block the main thread

[iarxm]

Yeh that would be it. And just looking and virtual dates also calls via the cli.

Taskwiki uses tasklib and imports relevant task data to cache in one go.

I think a lot of people would be interested in a super lightweight and filetype independent way to have their tw tasks sit within any syntax they desire.

[huantrinh1802]

There is a lot of room to improve regarding the performance here.

There are two main reasons I don't use tasklib:

1. It is no longer maintained and doesn't support Taskwarrior 3.x. There is an effort of creating the Python APIs via Pyodie (to integrate with Taskwarror/Taskchampion Rust API). Taskchampion Python bindings GothenburgBitFactory/taskchampion#385
2. My assumption is by using the CLI, the API in Lua will be more stable with the tradeoff of having poorer performance

With this one, I am hoping to utilise tree-sitter for detect embedded Markdown. Though, I can see this plugin can be tweaked to work without relying on markdown.

[iarxm]

Makes sense. Will you make use of the new Python API once it becomes stable?

I think though independent of current API usage, majority of the apparent performance can be improved by some minor tweaks. The CLI can be called far fewer times. Currently, it can sometimes be called many times per individual task. For a list of 30 tasks, that could mean about 100 shell spawns.

For multi task operations, rather than looping the cli query's as the buffer lines are looped, it would be more efficient to scan all task lines within the buffer into a dictionary, then query all their data via one cli command. This might reduce the shell spawns by at least 60%.

Also, where possible, the cli calls necessary to push changes to the task database, can all be qued up and forked to the background so they don't have to be waited on.

For me, this would probably reduce a 30s wait to about 5s.

[huantrinh1802]

Will you make use of the new Python API once it becomes stable?

It is very unlikely unless the performance is massively outweighed the stability of the APIs. For example, in the very worst case that Taskwarrior APIs will change from Rust to Zig, C++, it will require a massive refactor, which I don't wish to do.

For the later points, I fully agree, this is why I haven't released v1.x yet. I am very new to Lua so I am still exploring the APIs. One option I am currently considering is https://github.com/nvim-lua/plenary.nvim?tab=readme-ov-file#plenaryasync.

[iarxm]

I'm not a dev and won't have much insight on design but I would consider contributing and helping out with this in whichever ways I can. However, I'm not sure if what I'm personally looking for would be within your project scope.

I'm interested in the vision of having taskwarrior tasks sitting in any filetype as embedded markdown syntax and ultimately with some configurable syntax flavour.
I'm also interested in a configurable representation of the task data to be integrated into the task line, equivalent to customisable report column ordering within 'task report'.
Also to have bidirectional syncing of this customisable line which then pushes modifications of more than just the description of the task within the task db. Essentially 'task edit' but represented on one single line, with as an example, tasks recognised by '+' or projects by :project: etc.

Lastly, I would be hesitant based on the design principle of using the CLI as the interface. I could be wrong on this but using CLI commands doesn't seem like an optimal long term solution for a plugin like this and seems a bit hacky. In the future I could see this as being one of the most popular TW tools and would think you might want to use a proper API for it rather than mediating via cli calls.

Maybe this a comment for a discussion thread but would be good to hear your thoughts

[huantrinh1802]

Hi @iarxm, before addressing each topic, I would suggest looking at this plugin: https://github.com/duckdm/neowarrior.nvim?tab=readme-ov-file. A few interesting features there.

1. Support embedded markdown: definitely in the roadmap, though I expected to work out-of-box currently.
2. Report/display task within a buffer
   2.1 I am not planning to add more metadata in the buffer/markdown file because I think it will pollute the file.
   2.2 One option here to use virtual text, similar to due date/scheduled date feature
3. Bidirectional modification
   3.1 Currently, it can be achieved by:
   • TWRunWithCurrent: you can run this command, it will prompt to enter taskwarriror command that will run as task {uuid} {input via user}
   • TWEditTask: effectively a wrapper for task {uuid} edit
   • Using query view (see below), if you add/remove tags, due date,etc. and then run TWSync, the changes will get apply

# Today task $query{project:Personal +shopping}
 - Buy milk $id{some-uuid}

4. Use APIs or CLI,

There are two main reasons I don't use tasklib:
It is no longer maintained and doesn't support Taskwarrior 3.x. There is an effort of creating >the Python APIs via Pyodie (to integrate with Taskwarror/Taskchampion Rust API). >GothenburgBitFactory/taskchampion#385
My assumption is by using the CLI, the API in Lua will be more stable with the tradeoff of >having poorer performance

By using the CLI, there are two advantages:

• It will survive any underline APIs changes (such as Taskwarrior v2 to v3)
• Setting up the plugin is more straight forward (https://github.com/tools-life/taskwiki?tab=readme-ov-file#installation is an example of using Python library)