sim4life.io - WP3: Tracking of resource usage

[drniiken]

Description

To ensure transparency and accuracy in billing, it is essential to collect and keep track of all resource usage from users, including compute time for simulation hours and S3 storage per user as well as egress costs for download of files. This information must be synchronized with what the customer has paid for, and the data must be stored permanently to accommodate the pay-per-use model.

To make this data accessible to all relevant parties, it will be integrated into multiple systems, including the webpage, product, and API, as well as potentially the finance department for billing.

Tasks

Preview Give feedback

1. Potentail Enhancement: Monitor storage costs (e.g. via prometheus S3 exporter ?)
2. Overload testing (DB,..)
3. Review Pricing Data Backups @mrnicegyu11 @YuryHrytsuk
4. AWS managed rabbitMQ replicas @mrniceguy11 @YuryHrytsuk #1018
   +0
   
5. Create infrastructure to make the resource usage data accessible to all relevant parties #956
   +0
   
6. Add fields to RabbitResourceTrackingStartedMessage osparc-simcore#4674
   t:enhancement +1
   
7. "Incremental SUM" of credits #1308
   +0
   
8. Reduce prometheus retention time osparc-ops-environments#174
   observability t:enhancement wontfix +3
   
9. Setup infrastructure for the second database
10. Show credits burnt by the running project @odeimaiz @jsaq007
11. Credits indicator: Bar, Text or Text with Bar to Preferences @odeimaiz

Schoggilebe

Preview Give feedback

1. Create api for creation of pricing plan / pricing detail for "ADMIN" user #1133
   +0
   
2. React when 0 credits are reached #1170
   3 of 4
   +0
   

   
3. Tagging for cost separation @matusdrobuliak66 @mrnicegyu11

ThisIsSparta

Preview Give feedback

1. Add filtering/sorting in resource usage API #1214
   +0
   
2. Add export CSV functionality #1242
   +0
   

Kobayashi Maru

Preview Give feedback

1. Fetching data when I click on the tabs in the billing center
2. Add integration test to check whether credits burned == resources used #1213
   +0
   
3. Improve reporting/investigation on resource usage/costs #1239
   +0
   

Microhistory

Preview Give feedback

1. Add service key/version to the dynamic sidecar labels osparc-simcore#4475
   t:enhancement +1
   
2. Create Admin/Accountant endpoint for downloading data & modifying prices
3. Have a script that collects all container seconds per user group for jupyter-smash in AWS prod @mrnicegyu11
4. Connect External Service for User Credits
5. Credits indicator: yellow when it goes below 100$, red bellow 20$ @odeimaiz
6. Modify backend of resource tracker based on consolidated container labels #1004
   +0
   
7. Inject AWS EC2 instance type to the container labels @sanderegg @matusdrobuliak66
8. Adding user-selected resoures (cpu/memory) labels to Computational services on start @sanderegg
9. ✨Resource usage tracker service (⚠️ devops changes) osparc-simcore#4244
   a:resource-usage-tracker +1
   
10. Adding user-selected resoures (cpu/memory) labels to Dynamic services on start @sanderegg
11. Inject project name/node label/user email to the container labels @sanderegg @matusdrobuliak66
12. Mechanism for handling lost heartbeats @matusdrobuliak66
13. Fuse pricing units and service resources osparc-simcore#4958
    1 of 1
    a:dask-service a:director-v2 a:dynamic-sidecar a:webserver +4
    

    
14. API: access to wallets/payment for job submission osparc-simcore#4768
    +0
    
15. Deal with pricing plans for computational pipelines - extend DB (pricing info, EC2 instances) #1131
    +0
    
16. get invoices (from the 3rd party payment system) @pcrespov
17. Few free welcome credits could be loaded in this private wallet (Show it in dollars, 100$ by default) @pcrespov
18. Tier selection, hide it if not Advanced @odeimaiz

Quilmes

Preview Give feedback

1. Backend: API for showing wallet credits & User usage costs #1071
   +0
   
2. Wallets should be on the product level #1132
   +0
   
3. User's default wallet (on account creation) @pcrepov
4. Frontend: Show user credits @odeimaiz
5. Create api for pricing plans/pricing units to be used by webserver & api-server #1142
   +0
   

Baklava

Preview Give feedback

1. Collect and store messages (about running services) in resource usage tracker #1051
   +0
   
2. Dynamic Sidecar (dynamic services) send message #1007
   +0
   

   
3. Backend: Modify API to User Service Run Table @matusdrobuliak66 #1067
   +0
4. M8 Attribute (& Charge) Resource Usage #697
   1 of 1
   PO issue y6 +2
   

   
5. Backend: Credit computation & Update of user credits #1070
   +0
   
6. Add tracking of iSolve @matusdrobuliak66
7. Start to trace additional services @matusdrobuliak66

Sundae

Preview Give feedback

1. Backend: Introduce Wallets @matusdrobuliak66 #1039
   +0
   
2. Frontend: Show wallet info @odeimaiz
3. Define Rabbit MQ message #1019
   +0
   
4. Comp backend: collect metrics and send in RabbitMQ osparc-simcore#4523
   a:director-v2 +1
   
5. Create User Credit History table @matusdrobuliak66
6. Create Credit table (mapping of instances, CPU hours to price) @matusdrobuliak66

Watermelon

Preview Give feedback

1. Find suitable mechanims to gather resource usage data reliably #955
   +0
   
2. ✨ (⚠️ devops) 🗃️ Is922 resource tracking/1. version of regular scraping osparc-simcore#4380 @matusdrobuliak66
3. Make resource tacking service able to populate missing data after outage #994
   +0
   
4. Backend: API for Resources/Usage widgets #995
   +0
   
5. Dask backend: pass labels to be set on run containers osparc-simcore#4350
   a:dask-service a:director-v2 +2
   
6. ✨ Computational backend: set container limits as labels ⚠️ (devops checks on grafana dashboards!) 🚨 osparc-simcore#4453
   a:director-v2 a:models-library a:webserver +3
   
7. Maintenance: adjust the labels set on all services to follow inversed url notation osparc-simcore#3638
   6 of 6
   t:enhancement t:maintenance +2
   

   
8. ✨ Frontend: Select resources & Credits workflow osparc-simcore#4322
   a:frontend changelog:✨new-feature +2
   
9. Add Second Prometheus for resource usage service osparc-ops-environments#175
   3 of 3
   observability t:enhancement +2
   

[sanderegg]

Goal for sprint Jelly Beans

• Use jsmash e2e to collect data (cpu/gpu hours, s3 storage, egress network)
  • see what is missing.

[mrnicegyu11]

Update for sprint Jelly Beans

We have a very rough PoC python script that:

• gets service running time per user per service
• gets service actual cpu usage per user per service
• get service CPU reservation/limitation per user per service

Prometheus is used as a datasource. Potentially, this can serve as a basis for a new osparc-service that collects and exposes resource usage.

To make it robust, we need:

• Wrap PoC in robust tested osparc code
• Add a secondary Prometheus which is robustly set up (federated or separate, best managed prometheus from aws)
• Add regular dumps of usage reports to file / S3 / backup for billing and storing these numbers at Z43. This needs to be reliable and tested
• Potentially interface with postgres to get list of valid userids or resolve userID to email, etc.

The PoC script is found here ITISFoundation/osparc-simcore#4168

[mrnicegyu11]

Question from my side for the sprint planning PastelDeNata (I will be absent on PM2a):

• Who is billed in case of unscheduled or scheduled maintenances? Imagine a solver job that is long running and killed due to an incident or scheduled downtime. How do we detect it inside our (to be devised) cost-system?

• Is it truly necessary that we track additional metrics apart from simulation container seconds? It is at least conceivable to me to have a business model we charge the users a "flatrate" prices based only on simulation hours, that includes some additional charge for egress and S3 costs on our side as well. It would make the whole pipeline much smoother. I dont see value in devising a complicated system to track for example egress costs per user at this point (Personal opinion) . If a user needs to download a file for their scientific project, they are gonna do it anyway. A straight forward billing model might be easier to comprehend for end users. I doubt scientist want to even think about whether they cause egress or not. ---> I propose a flatrate charge based only on simulation seconds.

[mguidon]

• Failed jobs will not be billed (we will need to make sure that we disallow starting jobs prior to a downtime). For iSolve it might be possible to estimate the runtime
• We will have two scenarios:
  1. Cloud users will be billed by simulation hours
  2. desktop users that submit simluations will be billed by 100 + X % of our costs (ec2 + egress, egress could be tracked by API)
     In general, even if we do not bill it, I would like to know how much S3 storage a user consumes. Also I would like to know how much egress cost the video streaming costs (no on a per user basis). The latter is to get an idea of the cost and whether we should go for a AWS solution for webrtc.

[pcrespov]

Goal for sprint Pastel de Nata

• Have a script that collects all container seconds per user group for jupyter-smash in AWS prod @mrnicegyu11
• Do the same with iSolve @mrnicegyu11

[sanderegg]

Update for sprint Pastel de Nata

• new service resource-usage-tracker ready to be used:
  • ✨Resource usage tracker service (⚠️ devops changes) osparc-simcore#4244
  • ♻️ Uniformize Dockerfiles osparc-simcore#4275

[matusdrobuliak66]

Update Watermelon

Working comments

• adding needed labels to the computational services
• based on @GitHK input: Always check the last updated row -> and start the data fetching/populating of the database from there.
  • ✨ Is994/resource tracking service auto populate missing data osparc-simcore#4420
• adding endpoint for frontend (so we can get quick feedback)
  • ✨ 🗃️ (⚠️ DEVOPS) Is995/resource tracking list containers osparc-simcore#4445
• adding useful CLI commands (for example during an outage to be able to run/rerun the concrete task at a specific time)
  • ✨ (⚠️ devops) 🗃️ Is922 resource tracking/1. version of regular scraping osparc-simcore#4380 (comment)
• adding an additional test that will test concurrency (inspired by test_multiple_creation_deletion_of_nodes)
• IMPORTANT: adding a second data source from the application to get info when computational/dynamic services started/stopped.

Notes & Discussions:

• as promql doesn't support pagination when querying, we need to think of a list of regex expressions that would cover containers we would like to monitor (which will fetch a reasonable amount of data) and we will run more background tasks for each item in the list
  • OPEN DISCUSSION (with @mrnicegyu11): should we run all background tasks in one container, or should each background task run as a separate container in our docker swarm.
• NOTE: potentially introduce a daily aggregated table which will aggregate the data needed for billing service (therefore we would be able to keep the resource_tracker_container table data amount under control.
• NOTE: point of failure
  • resource tracking service is unavailable -> The scheduled task in this service can run even backward, the only important thing is that the Prometheus and its database is available
  • Prometheus outage -> There is a time range where we do not have data because Prometheus was not scraping them -> We will run at least 2 replicas of Prometheus in different nodes
  • There should be another source of data! -> We will store when services were started and stopped from the application.

[matusdrobuliak66]

Update Baklava

Done:

• User center dashboard
• Collecting and storing events/resources now from all running services (Computational & Dynamic)
• Created core backend for credit computation/mapping credits to pricing plans
• Expose top-up credit endpoint to Payment Service

In progress:

• Expose available credit information to the frontend
• Provide wallet_id (where to bill) and pricing_plan_id (what to bill) when starting a service via UI or API.

To do:

• Add a mechanism to close the running status of the service, if the resource tracker will stop recieving heartbeat events
• Stop the running service if there are not enough credits
• Connect simcore to AWS-managed RabbitMQ service
• Connect resource usage tracker to new AWS-managed Postgres database
  ...

[matusdrobuliak66]

Update Quilmes / TheNameless

Done:

• Showing real credits in the wallet endpoint
• Sending updates on current available credits in wallets to frontend via websocket
• Wallet constrained to the specific product (example: osparc, s4l, ...)
• User default wallet
• Defining and listing pricing plans

In progress/To do:

• API for storing the selected pricing plan for user service
• Extend computation backend to use selected pricing plans
• Introduce the lost heartbeat mechanism in RUT (to close the transaction if we lose the heartbeat)
• e2e
• Send to the RabbitMQ exchange that there are no credit left.

[matusdrobuliak66]

Update Microhistory

Done and working

• Selection of default wallet and pricing plan in the backend if not provided
• Emit message when 0 credits are reached
• Do not allow to start services when 0 credits are reached
• Default wallet endpoint in the webserver
• one-time-payment

should work in 3 weeks

• Mechanism for handling lost heartbeat (almost done)
• Billing data in the second database (if there is time left)
• Admin endpoint for creation of pricing plans/units (if there is time left)

should not be available in 3 weeks

• Optimization to have "Incremental SUM of credits table" so we do not need to always compute the whole history, which long term may get slow

[matusdrobuliak66]

Update ThisIsSparta

Done

• All known issues with computational backend sending events were fixed and now we are collecting all running jobs correctly.
• Add filtering/sorting in resource usage API #1214
• Add export CSV functionality #1242
• 🎨 Emit proper product name on socketio connection osparc-simcore#5285 (Burning credits now also works for different products, ex. academia, desktop,...)

Improvements to be done (not blocking release)

• Create api for creation of pricing plan / pricing detail for "ADMIN" user #1133
• "Incremental SUM" of credits (Currently when computing credits on the fly, we are counting the whole history of transactions, in the long term this is unsustainable and we need to compute credits incrementally)
• Continually bill while the transaction is running (Currently we consider the transaction billed only when it is successfully finished)