Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scheduled Task That Fetches Costs Data of Each Wallet and Saves Them Into the Plugin Database #130

Open
gluckzhang opened this issue Jan 14, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@gluckzhang
Copy link
Member

When there are many integrations, InfraWallet likely hits some rate limits (such as the AzureClient). If we have a scheduled task that fetches historical cost data and saves them into the plugin database, the rate limit issue can be significantly mitigated and also the performance is further improved.

Some thoughts before an actual implementation:

  • Different cost clients may generate cost reports with different fields, such as in MongoAtlasClient there are extra fields like project and cluster. How to design the database table properly?
  • We may only need to fetch costs at the daily granularity and for monthly costs, we use SQL aggregation queries.
  • When there is no record in the table for a specific wallet, the task should fetch as many historical records as possible (such as the last 12 months' costs), and then, the task updates the previous 30 days' cost.
  • This should be implemented in the base class (InfraWalletClient.ts) for all the providers whose type is Integration
@gluckzhang gluckzhang added the enhancement New feature or request label Jan 14, 2025
gluckzhang added a commit that referenced this issue Jan 21, 2025
gluckzhang added a commit that referenced this issue Jan 21, 2025
gluckzhang added a commit that referenced this issue Jan 21, 2025
gluckzhang added a commit that referenced this issue Jan 22, 2025
because if the task is currently running, retriggering it will throw an exception and stop the backend

Signed-off-by: Long Zhang <[email protected]>
gluckzhang added a commit that referenced this issue Jan 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant