AL languages internal cache is not always correctly refreshed

[Thomas-Torggler-EOS]

1. Describe the bug
The AL language editor host executable has an internal cache where it holds things like referenced symbols and project symbols and so on. I assume for performance reasons. This cache however is not always refreshed correctly and requires the user to run a "Reload Window" or reload the entire project workspace to refresh it.

I have a few scenarios where this happens and is a problem for me:

1. When I externally download symbols they are not detected by AL language.
2. Modifying the app.json triggers a cache refresh, but only when done so from inside VS Code. If app.json is modified externally, AL language does not detect a change to it and thus does not refresh anything.
3. Switching branches in git causes the cache of the AL language to become stale. The symbols in it's cache still refer to the old version of my objects. This until I either reload the project (as described above) or I open the object in question. This however refreshes the cache for this object only.

2. To Reproduce
Steps to reproduce the behavior:

Scenario 1:

1. Download a symbol package manually from somewhere and place it in the .alpackages folder.
2. The new symbol package is not detected.

Scenario 2:

1. Use a project with a dependency to an app that you already have in your .alpackages folder.
2. Update the minimum version to something higher that is not available.
3. You correctly get an error stating that symbols are missing.
4. Externally modify app.json and reset the version to what it was before.
5. The symbol is still missing according to AL language.

Scenario 3:

1. Work on a feature branch where you removed a method from an object.
2. Close the editor for that object.
3. Switch back to a different branch where the method from 1 still exists.
4. Go to a different object and reference this method.
5. AL language argues that this method does not exist, but it does.

3. Expected behavior
Option 1: detect file changes even when made from outside the project. There are numerous tools around now that interface with AL projects, tying it's caching mechanism to only what happens under AL languages watch is very limiting. You could, for example, bind this to the app.json file only. Anyone who wants to cause a cache refresh from outside VS Code then just has to touch that file and you would not have to watch all project files for changes.
Option 2: give us a command that let's us explicitly cause a cache refresh without having to reload the entire editor. This command would then also be required to be callable from other extensions, pretty please.

4. Actual behavior
The cache often becomes stale and requires me to reload my editor, causing lots of reload times.

5. Versions:

• AL Language: 15.0.1177813
• Visual Studio Code: 1.95.1
• Business Central: n/a
• Operating System:
  • Windows
  • Linux
  • MacOS
    Internal work item: AB#556873

[BazookaMusic]

I'll accept this since I think it's worth investing into this. However, the issue is not caching. The problem is that the extension and VSCode are not processing external changes properly.

We'll definitely work on improving the GitHub branch change scenario. For the symbols, it is possible that this is a larger investment which might take more time.

[Thomas-Torggler-EOS]

As far as symbols go, I do think it is (somewhat) a caching problem:

• At the end of the symbol download command the symbols are invalidated in the MemoryCachedSymbolReferenceLoader using InvalidateDependencies.
• After doing so, UpdateSymbolReferences is called on VsCodeWorkspace and thus reloading the ones that were just downloaded.

If we had a command in VSCode that would replicate this, at least the symbols could be manually refreshed.

As far as the source files go, I did not investigate and that is surely a different kind of behaviour / issue.

[kalberes]

Agree with @BazookaMusic . We will try to make github branch changes not to blow up the language server and the diagnostics service.
Exposing the command is a bit more complicated. There are more then just what you wrote. There are coupling to other subsystems like the project system, diagnostics service, tracking compilation. It makes things a bit more complicated to be treated as a simple bug.

[HiddenDude]

I'd say this (or similar effects) doesn't just happen around external changes, but also often happens with changes in VS code.
It might be related to / occur more often in big projects with a lot of files, but in my opinion thats no excuse for it to happen.

#1
Sometimes when adding a line to a block of single line comments, the editor starts syntax-highlighting parts of the following code like a comment too.. (and it doesn't recover unless you reload)

#2
Often, when partially writing a variable name in a function block and choosing the intellisense completion, the editor still complains about the partial variable name not being known. And it can take quite a long time to recognize the variable name in the code block has been completed. Even saving the file doesn't trigger an instant full reparse of the current file. (which could probably solve the problem as long as that action doesn't get stuck in a queue thats still busy processing incomplete / outdated information)

If you write more of the variable name yourself you can sometimes watch the editor slowly catch up to what you have typed in multiple steps, but it is really unnecessarily slow..

#3
When anything like that happens, the intellisense also often breaks.
E.g. hovering stuff might then just produce a "Loading..." tooltip, and peek/goto no longer works.. And it is just faster to simply reload the whole program/project than to wait for the AL language extension to continue working as expected.. I'm probably reloading my project at least once every 5 minutes to fix random issues with the AL language extension and it's understanding of the symbols in my dependencies/application, because it is just faster and less buggy..

...

But even small workspaces with e.g. <20 files can take upwards of 2 minutes to be fully loaded with just 6MB of base application symbol packages.. (e.g. with 8 cores, enough RAM and an SSD - thats crazy slow)

I expect a lot of that probably comes from most of the extension being coded in JS..?
(aside from the actual compiler and some extra command line tools)
If JS is the only interface that can directly talk to VS code from an extension, why not just use JS to wrap a native or managed DLL that contains the actual functionality, instead of having the world coded in JS (4MB+?!) and local performance slow like in the cloud or a browser..
What does the extension even do self-initializing for 400ms even before starting to load any workspace data..? (maybe telemetry-ception or sending a webrequest to openAI to generate some more bloat to include in the JS files?)

[BazookaMusic]

@HiddenDude what you're talking about is something entirely different and what you're describing sounds like a bug.

I just tried the MS base app and could open it and load it in around 2 minutes, so 2 minutes for 20 files sounds like the language server gets stuck somewhere.

This is even more apparent by the symptoms you are describing. It's not about tracking changes, if goto definition and intellisense gets slow that means that the compilation is not being generated fast enough, not that changes are being missed.

I'd suggest trying this out (especially the troubleshooting part and setting code analysis to file) to see if there's some analyzer rule eating up all the cpu:
https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/devenv-code-analysis-performance-configuration

If that doesn't fix it, the next step would be to try turning off other performance impacting features especially if everything is so slow that you are not using it.

I'd suggest turning off other AL related extensions, code lens and finally code actions and seeing if that makes a difference.

If it's the last two that helps or nothing helps, I'd appreciate if you share some example project where the language server gets borked so we can look at it.

However, please share the details in a new issue so as not to cause noise on this one.