Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce the frequency for database updating #562

Open
zhuoyuan-liu opened this issue Nov 7, 2024 · 2 comments
Open

Reduce the frequency for database updating #562

zhuoyuan-liu opened this issue Nov 7, 2024 · 2 comments
Labels
✨ enhancement New feature or request ⚡️ performance Performance related issues

Comments

@zhuoyuan-liu
Copy link
Contributor

zhuoyuan-liu commented Nov 7, 2024

All read requests such as config and query read would update the database about the node info. However, these updates would become a bottleneck for database performance.

I understand that many functions reply on the last seen timestamps, but it would be not scaleable with such heavily written requests to the database. I have two ideas, one is to store these data in Redis/memory and update them periodically. Another is that we process these changes async.

In our test, we only have 3 req/s but the latency between database and osctrl is around 50ms.

Looking forward to hearing from your idea.

@javuto javuto added ✨ enhancement New feature or request ⚡️ performance Performance related issues labels Nov 7, 2024
@javuto
Copy link
Collaborator

javuto commented Nov 8, 2024

I have thought about this several times and I believe combining those two ideas may be the way to go:

  • We load nodes in Redis/memory and each field that is now refreshed and updated in the backend, it is up-to-date only in Redis/memory.
  • A task is kicked off in the background to keep the backend in sync with what it is in Redis/memory on an interval basis. This can be a goroutine that runs in osctrl-admin or osctrl-tls, like we have others now, and runs every hour or so, with less impact to the backend. Those operations can also be batched.

Thoughts?

@zhuoyuan-liu
Copy link
Contributor Author

It would definitely increase the scalability, but not sure how far this method would go. I would like to focus on this issue #558 and get some of your feedback. We will also test the osctrl on a larger scale this/next month and provide more feedback/improvement about the performance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
✨ enhancement New feature or request ⚡️ performance Performance related issues
Projects
None yet
Development

No branches or pull requests

2 participants