org-roam-server slows down unrelated interactive commands

[AloisJanicek]

Hi, I am relatively new to org-roam world, and I've been using it just few days. But I started to notice weird performance issues occurring when executing unrelated interactive commands and I think this issue is related to org-roam-server.

Steps to reproduce

1. start org-roam-server-mode on org-roam-directory with ~200 files
2. open one or two files by both org-roam-find-file in emacs and org-roam-protocol click in org-roam-server browser window
3. switch to completely unrelated buffer like your emacs config file
4. turn on emacs profiler
5. jk scroll for couple of seconds (basically DoSing emacs)
6. turn off emacs profiller

Now you can see that aside from expected redisplay_internal, there is httpd--filter taking most of the CPU time.

...
+ httpd--filter                        474  55%
+ redisplay_internal (C function)      233  27%
...

full report

When I do something more natural in emacs, like searching for something in M-x and switching between few buffers, it is a server-process-filter which takes most of the CPU time.

full report

Expectations

Emacs should not feel sluggish when doing things unrelated to org-roam.

My setup

emacs 27.0.91, Arch Linux, vanilla doom emacs with org-roam and org-roam-server from github. Everything latest and up-to-date.

[goktug97]

First I thought it is related to

org-roam-server/org-roam-server.el

Line 316 in a245296

(add-hook 'post-command-hook #'org-roam-server-find-file-hook-function)

org-roam-server/org-roam-server.el

Line 341 in a245296

(defun org-roam-server-find-file-hook-function ()

but it is only executed if the file is an org-roam file and it has nothing to with httpd--filter. Does the performance issue persist after some time? Maybe it is taking some time to export and load all of the files. I have 68 org files and haven't had any issues like this.

(defun httpd--filter (proc chunk)
  "Runs each time client makes a request."

https://github.com/skeeto/emacs-web-server/blob/22ce66ea43e0eadb9ec1d691a35d9695fc29cee6/simple-httpd.el#L417-L418

[AloisJanicek]

After initial high CPU usage caused by fancy animation of aligning items in browser window (I like it) and maybe some exporting (even though my files currently don't have any content, only majority of them have backlinks), CPU goes idle.

This performance issues occurs anytime after that. Currently I have emacs running where I started org-roam-server about 30 minutes ago and I am cooking in scratch pad and researching this issue in helpful buffers and various elisp files. CPU is idle when I am not in emacs. But when I switch to emacs and start doing something, well actually anything, emacs feels sluggish and commands lags a lot.

This of course doesn't happen when org-roam-server-mode isn't on.
I usually can debug and track down issues like this, write some advise and get over it, but this time I am totally clueless.

[goktug97]

Does the same thing happen if the org-roam-server tab is closed in your browser?

[AloisJanicek]

No it doesn't.
When the visualization pages isn't displayed in browser, emacs behaves normally. When I open http://127.0.0.1:8080/ again, emacs immediately starts to lag. Oh, that's interesting discovery.
And it doesn't matter which browser I use. I tried EAF, Chromium and Firefox. Always the same result.

[goktug97]

Can you check Network in Firefox Developer tools, maybe we can find something.

[AloisJanicek]

In semi-regular interval, there are request like the ones in your screenshot. Whatever I do in emacs doesn't seem to have any effect on what and how often is printed into Network console in browser.

No difference between Firefox and Chromium in this matter.

[goktug97]

It shouldn't be a problem but to see if the size of the db is causing this, can you

• copy random 60 org-roam files to a new directory
• set org-roam-directory as the new directory (dir-locals.el doesn't work Set a different org-roam-directory in the server #8)
• Close emacs to stop the httpd-server httpd-stop doesn't stop the process skeeto/emacs-web-server#23
• Reopen emacs and the server
• Check if the performance issue is still there.

[AloisJanicek]

With only 62 files, my "hold some movement key for couple of seconds" test produces better results: emacs freezes only for ~200ms every ~2 or 3 seconds and performance of other parts of emacs is acceptable. With 200+ files, emacs freezes for almost for ~1 second every 2 or 3 seconds when there is movement key pressed and all other usual heavy operation like filtering M-x candidates or using swiper on big file are noticeably slower too and results in occasional freeze of whole emacs.

So yes, there is a correlation between number of files and performance. It corresponds to what I have been experiencing since I begun my brain-dump about 4 days ago. I first noticed slight drop of performance when there was about 50 files and when I was above 150, I begun to worried.

If I understand the issue correctly, server is running inside of emacs and it receives request from client (browser) in semi-regular intervals. If the server (emacs) is busy with something else, addressing request from client at top of that may result in choking and lagging.

[goktug97]

I can't find anything that might be causing this issue. If I had a fast computer, I would think that I can't see this issue because it is handled so quickly but as you can see from the neofetch output it is not that high-end. I did 10 seconds jk test and couldn't see any freezing. The client constantly checks the event-stream but the server only sends the data if there is a change.

The only thing that came to my mind is full RAM. It is probably not but can you check your RAM while using the program.

                   -`                    goktug@Goktug
                  .o+`                   -------------
                 `ooo/                   OS: Arch Linux x86_64
                `+oooo:                  Host: 23257R2 ThinkPad X230
               `+oooooo:                 Kernel: 5.6.11-arch1-1
               -+oooooo+:                Uptime: 5 days, 22 hours, 22 mins
             `/:-:++oooo+:               Packages: 2084 (pacman)
            `/++++/+++++++:              Shell: zsh 5.8
           `/++++++++++++++:             Resolution: 1366x768
          `/+++ooooooooooooo/`           WM: i3
         ./ooosssso++osssssso+`          Theme: Arc-Dark [GTK2/3]
        .oossssso-````/ossssss+`         Icons: Xenlism-Wildfire-SaturDay [GTK2/3]
       -osssssso.      :ssssssso.        Terminal: st
      :osssssss/        osssso+++.       Terminal Font: Fira Code
     /ossssssss/        +ssssooo/-       CPU: Intel i5-3320M (4) @ 3.300GHz
   `/ossssso+/:-        -:/+osssso+-     GPU: Intel 3rd Gen Core processor Graphics Controller
  `+sso+:-`                 `.-/+oso:    Memory: 5027MiB / 7662MiB
 `++:.                           `-/+/
 .`                                 `/

With 68 files this is my current graph

[goktug97]

I opened a http server in python and then tried to open org-roam-server to see if they will interfere with each other but of course as the socket is occupied it didn't start.

[AloisJanicek]

Now I even tried this with vanilla emacs (just ivy/counsel, evil and org stuff). At the time of the test I had slightly over 2GB of RAM full. Afterwards consumption went to 4GB 5GB 6GB - and the cause was emacs. So there is memory leak.

Some observation on this:

• it does happen too with fewer (62) files, but on slower rate
• it is not caused by profiler (I re-run emacs again and I didn't enabled it in the session)
• it doesn't happen when org-roam-server-mode is off.
• It doesn't happen when just normal org-roam mode with side bar buffer is on

At the time when emacs occupies more then 4GB of RAM, the memory consumption rate speeds up to additional 100MB every 2 or 3 seconds according to htop.

But I am able to experience the performance choking issue right from the beginning when there is a plenty of RAM available.

I have practically same hardware specs as you and my graph is proportionally larger, maybe slightly complicated. screenshot

I will try this with emacs 26.3 and emacs 28 (now I am using 27.0.91 build few days ago).
Another thing I could try is to disable apparmor and firejail - I don't jail emacs so it should not matter, but at this point who knows.

[goktug97]

At the time when emacs occupies more then 4GB of RAM, the memory consumption rate speeds up to additional 100MB every 2 or 3 seconds according to htop.

when org-roam-server-mode is enabled, my emacs occupies 100 MB RAM. Also without org-roam-server-mode it occupies 100 MB.

Another thing I could try is to disable apparmor and firejail - I don't jail emacs so it should not matter, but at this point who knows.

I am okay with anything random that can be tried because I am out of ideas.

Your graph looks cool :D

[AloisJanicek]

I have some updates:

memory leak occurs with both emacs 27 and 28, it is serious (within 40 minutes emacs goes to 6+GB) and it doesn't occur with emacs 26

booting without apparmor and running browser without firejail didn't had any affect as expected

For me the issue is performance of org-roam-server-visjs-json. Both org-roam-db-query and org-roam--path-to-slug are quite expensive to have them running every second for just having org-roam-server browser tab ready for when I will need it.

I am planing to have thousands of files across two or three slipboxes (roams). I really want to put everything out of my head, bytes are cheap I am telling myself. I expect the number of edges and nodes hit more then 1 thousand per slipbox.

I first thought that maybe using hash tables and some kind of "intelligent" caching for those repeating calls would solve this, but now I am more inclined to implementing server functionality outside of emacs even if that means some compromises on functionality. But things like keeping track of current buffer would be still possible. I am not familiar with the whole functionality of org-roam-server, so maybe there will be some issue.

I will start researching how to do this, I would like to come with some proof of concept as soon as possible.

[wohanley]

My graph is quite a bit larger than either posted in this thread, and performance is definitely an issue for me. I've started looking into this myself - I think you're right that ultimately we just can't be recomputing the whole graph every few seconds. I'm trying out web-server.el right now in a branch in my fork, which has a proper WebSocket implementation. Hopefully I can get it to a point where we watch org-roam-directory for changes and push them when needed.

I have no real point to make here, just wanted to post about it for visibility's sake. If anyone has any thoughts I'm happy to hear them.

[goktug97]

My graph is quite a bit larger than either posted in this thread, and performance is definitely an issue for me. I've started looking into this myself - I think you're right that ultimately we just can't be recomputing the whole graph every few seconds. I'm trying out web-server.el right now in a branch in my fork, which has a proper WebSocket implementation. Hopefully I can get it to a point where we watch org-roam-directory for changes and push them when needed.

I have no real point to make here, just wanted to post about it for visibility's sake. If anyone has any thoughts I'm happy to hear them.

Only the json is calculated and if there is a change the network is calculated but of course adding a node doesn’t require a lot of computing. Does calculating the json slow down with large db?

[wohanley]

json-encode on my 419-node graph takes about half a second.

[goktug97]

And with that much node, I assume you also have lots of edges. I don't know if it helps or works for this issue but maybe this can be added

Checking for changes in a folder is a good idea but when only an edge is added it won’t be registered

json-encode on my 419-node graph takes about half a second.

@wohanley
I am thinking about adding a customizable variable org-roam-server-network-autoreload. If it is set to nil, it won't calculate the json until you press the reload button. But I think @AloisJanicek might have a different problem because it has considirable amount of RAM usage.

[goktug97]

@wohanley As I said, I added org-roam-server-network-autoreload to https://github.com/org-roam/org-roam-server/tree/dev branch. Can you test it to see if it helps to your problem.

[wohanley]

Yes, this is a huge help to me! Everything is smooth with org-roam-server-network-autoreload set to nil. Thanks! At risk of bikeshedding I might reconsider the name - something like org-roam-server-network-poll is more informative and wouldn't get in the way of alternative auto-reload implementations down the road.

[goktug97]

Yes, this is a huge help to me! Everything is smooth with org-roam-server-network-autoreload set to nil. Thanks! At risk of bikeshedding I might reconsider the name - something like org-roam-server-network-poll is more informative and wouldn't get in the way of alternative auto-reload implementations down the road.

I am glad, it is solved. It is merged to master. The naming also changed from autoreload to poll.

[AloisJanicek]

I am back with basic external org-roam-server hacked together in python.
I grabbed org-roam-server assets and some simple python server skeleton and hacked something together.

Performance is amazing. I had to do it because even with the org-roam-server-network-poll some slowdown was still noticeable.

Now I am enjoying org-roam without any limit.

[goktug97]

Now I am enjoying org-roam without any limit.

Awesome!