SMP Timeout Issue with Large Number of Service Groups

[woutray]

Hi Philip,

We are currently facing a challenge with our SMP setup. Specifically, we’re encountering a timeout when attempting to log in. By default, logging in opens the service group overview, and with approximately 12,000 service groups, it seems this operation times out.

Our SMP is running inside an ECS service with 4 GB RAM and 1 vCPU. From what I can observe, it looks like everything is being loaded into memory during this process, which may be causing the timeout. However, I could be misunderstanding the root cause.

Am I possibly missing some configuration that could alleviate this issue? Or do you think the allocated resources (especially the vCPU) are insufficient to realistically support an SMP of this size?

Any guidance or suggestions would be greatly appreciated.

We are running version 7.1.7 of the phoss SMP.

Thank you again for your work and for making tools like phoss SMP available!

Best regards,
Wout

[phax]

Hello @woutray,
Thanks for starting the discussion.
Indeed, the user interface is not really optimized for so large amount of participants.
May I ask what backend you are using? When running it with an SQL or MongoDB backend, the network overhead communicating the data may also have an impact.

The performance problem has already been mentioned by other users as well, and the only workaround currently is to work primarily on the API level. The root cause of this problem is the design of the Service Group and Endpoint UI pages. They were never designed for so large amount of participants. I would need to switch from an application server side filtering to a data source based filtering (SQL ORDER BY or LIMIT) to reduce the amount of data fetched from the DB.

These changes will take some effort, so I can't promise anything in the near future as other (paid) activities take precendence. If you are thinking about providing a PR or similar, please let me know, so that I can eventually align with the other party that is facing the same problem.

Thanks for your understanding.

[woutray]

Hello @phax

Thank you for your detailed response and for clarifying the root cause of the performance issue.

I am currently using the SQL backend, so I can see how the network overhead and the current design of the Service Group and Endpoint UI pages could be contributing to the problem. I completely understand that implementing a solution would require significant effort, and I appreciate the transparency regarding prioritisation of other activities.

At this point, I don’t have immediate plans to provide a PR, but I’ll keep it in mind as a possibility for the future. If I do decide to contribute, I’ll make sure to align with you.

Thanks again for your hard work.

Best regards,
Wout

[mirvnillith]

Beside UI problems, how many service groups would you say phoss-smp-sql would be comfortable with?

[phax]

Sorry, I can't say anything else than "it depends" - too many parameters here. Usually a couple of thousand should not be an issue, but DB performance, network performance and general server performance have an impact....

[mirvnillith]

Is there caching? I.e. could you run multiple instances towards a single database (assuming that’s not the bottleneck)? Our case would go into the hundred thousands … tors 19 dec. 2024 kl. 12:55 skrev Philip Helger ***@***.***>:

…

Sorry, I can't say anything else than "it depends" - too many parameters here. Usually a couple of thousand should not be an issue, but DB performance, network performance and general server performance have an impact.... — Reply to this email directly, view it on GitHub <#300 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOQLONDSWWZCOSIZNGYAWL2GKXZZAVCNFSM6AAAAABTKZUIEGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRRGY2TOMQ> . You are receiving this because you commented.Message ID: ***@***.***>

[phax]

I will prepare an SMP instance locally and provide you some details about the responsiveness. Give me some time pls

[mirvnillith]

Now I've run a loop of createServiceGroup-updateServiceMetadata-lookupServiceMetadata for 100 000 participants and all round-trip times for all three calls (made 100 000 times) stay <= 20ms. And this is running on my laptop with both phoss-smp and its Postgres in Docker containers so it looks like it scales very well. And assuming no caching any application server bottleneck can be scaled away.

[phax]

The problem is primarily the UI. If you rely on the API only, everything should be fine

[mirvnillith]

I will do the same, even if it’ll only be on my laptop. tors 19 dec. 2024 kl. 17:53 skrev Philip Helger ***@***.***>:

…

I will prepare an SMP instance locally and provide you some details about the responsiveness. Give me some time pls — Reply to this email directly, view it on GitHub <#300 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOQLOLPLVSAMMOHEZPRNKT2GL2Z5AVCNFSM6AAAAABTKZUIEGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRRHE4DMNA> . You are receiving this because you commented.Message ID: ***@***.***>

[mirvnillith]

Yes, we’re using our own backend for the data and the SMP only for publication. fre 20 dec. 2024 kl. 12:39 skrev Philip Helger ***@***.***>:

…

The problem is primarily the UI. If you rely on the API only, everything should be fine — Reply to this email directly, view it on GitHub <#300 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOQLOJQX6B5PYVAK2B3P6L2GP6YRAVCNFSM6AAAAABTKZUIEGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRSG44DSOI> . You are receiving this because you commented.Message ID: ***@***.***>

[phax]

Release 7.2.0 increases the general performance on the Service Group and Endpoint listing. Please give it a try

[woutray]

@phax Thank you for your hard work! Coincidentally, we noticed the new release this morning and have already bumped to version 7.2.0. We're happy to report that we can once again visit the Service Group page without any issues. Your efforts to enhance the performance are greatly appreciated.