Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Screen Output Stopping Issue Due to Lack of Load-Balancing in socket.io Sessions #1492

Closed
Haruki112 opened this issue Oct 31, 2024 · 4 comments
Closed

Screen Output Stopping Issue Due to Lack of Load-Balancing in socket.io Sessions #1492

Haruki112 opened this issue Oct 31, 2024 · 4 comments
Labels
bug Something isn't working needs-triage

Comments

@Haruki112
Copy link

Describe the bug：
The implementation of stream output has been completed. However, the socket.io session lacks a load-balancing mechanism, causing significant processing speed degradation when sending a large volume of events. Additionally, screen output frequently stops.

To Reproduce：
Steps to reproduce the behavior:

  1. Implement stream output.
  2. Send a high volume of events continuously.
  3. Monitor the screen output.

Expected behavior：
Expected continuous stream output without significant degradation in processing speed, even with high event volume.

Screenshots：
Screenshot 2024-10-31 at 13 57 28

Desktop：
OS: Mac, Windows
Browser: Chrome, Edge, Safari
Version: Chainlit 1.1.404, 1.1.304

Additional context：
After reviewing the code, it appears that the socket.io session lacks a load-balancing mechanism.
Could you please provide guidance on how to address this?

Additionally, I would like assistance with adding a load-balancing mechanism to the socket.io session.
@dosubot dosubot bot added the bug Something isn't working label Oct 31, 2024
@dokterbob
Copy link
Collaborator

Thanks for the feedback.

Could you please share a minimal example and/or whatever tooling you used for your stress testing?

Also, please share any measurements of what was going on your server (particularly CPU load, RAM usage, disk I/O). It might be that you just need to scale-up your deployment (e.g. have more instances running or beefier metal etc.).

@Haruki112
Copy link
Author

Haruki112 commented Nov 5, 2024 •
edited
Loading

@dokterbob
I ran Chainlit usingchainlit run app.pyand sent multiple text-based questions.

@dokterbob
Copy link
Collaborator

Thanks @Haruki112, please share a main.py and any specific steps etc. required to replicate.

@dokterbob dokterbob mentioned this issue Nov 11, 2024
Open
dokterbob pushed a commit that referenced this issue Dec 12, 2024
@willydouhard
feat: enhance socket.io configuration and CORS handling (#1575)
d531633 
This improves Socket.IO configuration and CORS handling for cross-origin use cases,
particularly focusing on scenarios where Chainlit is embedded in other websites (copilot, closes #1279) or deployed behind load balancers:

- Make Socket.IO client transports configurable via `project.transports` setting
- Move connection parameters from headers to Socket.IO auth object for better
  websocket compatibility
- Update CORS handling to properly support authenticated cross-origin requests. Closes #1359.
- Remove unnecessary task start/end events from window_message handler

BREAKING CHANGE: For cross-origin deployments, wildcard "*" is no longer supported
in allow_origins when using authenticated connections. Specific origins must be
listed in the project config.

Possibly resolves #719, #1339, #1369, #1407, #1492, #1507.
@dokterbob
Copy link
Collaborator

Should be resolved with #1575.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs-triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dokterbob @Haruki112