Start-up time again

[kybos]

We had this discussion before (and I noticed once again that it seems impossible to find anything in this team space, so lots of precious information is lost - or am I just to stupid to find it?).
Anyway: I noticed that the delays at startup are caused by the lv2 scanning. Without it (with the -2 switch), MusE is up almost immediately.
From the lv2 specs I gather that all plugins must offer a small manifest.ttl that can be parsed quickly to get the basic info. The full parsing should only be done if the plugin is actively used, as it of course takes considerable time.
But MusE seems to do the exact opposite and do the full parsing of all lv2 plugins at startup. This increases the startup time in factors of 10-1000, depending of how many plugins are installed.
Have I misunderstood something or do we have a problem here?

[terminator356]

No we do not 'fully parse' the LV2 plugins. We don't even open them until instantiation time.
We simply read the all the available basic info in the text files presented to us by the LV2 'world'.
This takes less time than other plugin types because we don't have to open the library files
to discover their properties. That's one of the huge very smart benefits of LV2 - basic info is available
in text files without opening the libraries.

I've not noticed any unusual LV2 delays, and I have a lot of them.
How many LV2 plugins do you reckon you have installed?

[kybos]

With "parsing" I mean parsing the ttl files, not opening the binaries.
There are various ttl files for each plugin - the small manifest.ttl with basic info and one or several others which are typically much bigger.
From the lv2 book:
"Hosts typically read the manifest of every installed bundle to discover plugins on start-up, so it should be as small as possible for performance reasons. Details that are only useful if the host chooses to load the plugin are stored in other files and linked to from manifest.ttl."
lilv_world_load_all() seems to parse everything at start (=all the ttl files), but I don't know how it works internally, maybe it starts with the manifest and loads the other ttls only when needed...
Anyway, I will have to look into it in more detail on my system to get out where the time gets lost.

There could be several hundreds of lv2 plugins on my system. I'm trying to reduce the number now (I had to install additional plugin packs because of the resizing and other issues I was working at), but for some synths even the individual sounds/patches are delivered as individual "plugins" so I suppose there will always be quite a high number of them.

[spamatica]

Yeah, it does take a significant amount of time on my system too. I have kxstudio installed and it pulls in a pretty large amount of plugins so there is probably several hundred lv2 plugins here too. Tried to time startup in different configurations. With no plugins MusE starts in ~2s With all but LV2 ~3s With all plugins ~11s Could the discovery be done in a background thread? Den tors 25 mars 2021 kl 09:28 skrev kybos ***@***.***>:

…

With "parsing" I mean parsing the ttl files, not opening the binaries. There are various ttl files for each plugin - the small manifest.ttl with basic info and one or several others which are typically much bigger. From the lv2 book: "Hosts typically read the manifest of every installed bundle to discover plugins on start-up, so it should be as small as possible for performance reasons. Details that are only useful if the host chooses to load the plugin are stored in other files and linked to from manifest.ttl." lilv_world_load_all() seems to parse everything at start (=all the ttl files), but I don't know how it works internally, maybe it starts with the manifest and loads the other ttls only when needed... Anyway, I will have to look into it in more detail on my system to get out where the time gets lost. There could be several hundreds of lv2 plugins on my system. I'm trying to reduce the number now (I had to install additional plugin packs because of the resizing and other issues I was working at), but for some synths even the individual sounds/patches are delivered as individual "plugins" so I suppose there will always be quite a high number of them. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/orgs/muse-sequencer/teams/musedevteam/discussions/83/comments/2>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCFAN5YVXZZ2SOX37OEEDTTFLXKTANCNFSM4ZYAOZ7Q> .

[kybos]

Unfortunately I was right in my assumptions. All lv2 plugins are instantiated at startup and the LV2Synth constructor reads all data from the ttl files. There can be megabytes of them per synth (e.g. amsynth).

Profiling shows that the main performance killer is this function (also called in the LV2Synth constructor):
LV2Synth::lv2state_UnloadLoadPresets(this, true);

Here this lilv call alone can take up to about 60% of the overall startup time:
lilv_world_load_resource(lilvWorld, preset);

All that looks like a considerable wasting of system resources (and of the startup time) and clearly contradicts the smart approach of lv2 (separation of basic info from the details for efficient scanning).
I'm afraid this design can't be changed easily (and I suppose we don't want to delay the final 4.0 release any further). But at least reading the presets should be avoided IMO (i.e. before the plugin is really used), that would already help a lot as it seems.
I should have some time tomorrow so will try to check for possible improvements (unless one of you knows the lv2 area well and would like to take over).

[spamatica]

No hurry for my sake, there is room for improvement but it's not a breaking issue. If there is some improvement before 4.0 goes live, fine, but if not that's fine by me. Den fre 26 mars 2021 kl 17:03 skrev kybos ***@***.***>:

…

Unfortunately I was right in my assumptions. All lv2 plugins are instantiated at startup and the LV2Synth constructor reads all data from the ttl files. There can be megabytes of them per synth (e.g. amsynth). Profiling shows that the main performance killer is this function (also called in the LV2Synth constructor): LV2Synth::lv2state_UnloadLoadPresets(this, true); Here this lilv call alone can take up to about 60% of the overall startup time: lilv_world_load_resource(lilvWorld, preset); All that looks like a considerable wasting of system resources (and of the startup time) and clearly contradicts the smart approach of lv2 (separation of basic info from the details for efficient scanning). I'm afraid this design can't be changed easily (and I suppose we don't want to delay the final 4.0 release any further). But at least reading the presets should be avoided IMO (i.e. before the plugin is really used), that would already help a lot as it seems. I should have some time tomorrow so will try to check for possible improvements (unless one of you knows the lv2 area well and would like to take over). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/orgs/muse-sequencer/teams/musedevteam/discussions/83/comments/4>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCFANYKCMMOFIKXOMTDMEDTFSVUNANCNFSM4ZYAOZ7Q> .

[terminator356]

Thanks for the profile. I see the code and call sequence you mention.
Right. It loads the presets into the Synth object before anyone actually instantiates it.
I can fix it, I've done a lot of the LV2 code (Andrew wrote the first code).
I suspect it might be straight forward.
Keep the presets list empty until there is at least one instantiation of SynthI.
Maybe even unload the presets to save resources when the instance count zeros.
There is the question of when to do it - obviously not directly in the realtime thread.
Fortunately I think much of the impetus for creating/destroying synth instances is initiated by
user clicks or song loading in the GUI thread. Where we can afford to wait a bit. But not too long, eh?

But I don't know if I can do this quickly. I was hoping to get my fixes in before release.
There are a couple of important fixes and neat enhancements, related to the main fixes.
I have made progress and I'm satisfied with the outcome of my techniques. It took a long time.
I have to finish it and do big cleanups. But time is at a premium for me. But a holiday on April 2...

[terminator356]

I also thought about populating the presets the first time someone opens the presets popup menu.
("Ooookay, you asked for a presets list, here we go...")

Well, lookie here what ol' Andrew commented:

void LV2Synth::lv2state_populatePresetsMenu(LV2PluginWrapper_State *state, MusEGui::PopupMenu *menu)
{
    menu->clear();
    menu->setIcon(QIcon(*MusEGui::presetsNewIcon));
    LV2Synth *synth = state->synth;
    //this is good by slow down menu population.
    //So it's called only on changes (preset save/manual update)
    //LV2Synth::lv2state_UnloadLoadPresets(synth, true);

Yeah, lv2state_UnloadLoadPresets is only called from lv2state_applyPreset and from
the constructor and destructor.

Intriguing.

[terminator356]

I wrote:

Fortunately I think much of the impetus for creating/destroying synth instances is initiated by
user clicks or song loading in the GUI thread. Where we can afford to wait a bit. But not too long, eh?

Eh, like eight seconds too long:
Robert wrote:

With all plugins ~11s
Could the discovery be done in a background thread?

Right. (Not so much plugin discovery, but loading presets.)
Let's not slow down the GUI thread as I wrote.
Let's start a preset loader worker thread say, as soon as the Plugin/Synth object is created.
(When else could we start it, after the Arranger has shown? Something like that?)
By the time anyone instantiates a PluginI/SynthI of the thing and clicks the presets menu,
it could be ready to go. A signal or flag tells when it's ready.