Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do cloud billing upto end of Q3 2023 #3350

Closed
6 tasks
yuvipanda opened this issue Oct 31, 2023 · 7 comments
Closed
6 tasks

Do cloud billing upto end of Q3 2023 #3350

yuvipanda opened this issue Oct 31, 2023 · 7 comments
Assignees
@yuvipanda

Comments

@yuvipanda
Copy link
Member

yuvipanda commented Oct 31, 2023 •
edited
Loading

I'll work with @colliand in doing cloud billing, upto Q3 of 2023 (until end of September 2023).

This entails:

  • Cloud costs for AWS clusters we need to pass through
  • Cloud costs for GCP clusters we need to pass through
  • Cloud costs for hubs on 'shared' clusters
    • For 2i2c GCP cluster
    • For 2i2c-aws-us AWS cluster
    • For 2i2c-gcp-uk GCP cluster

I'll start working on this sometime early November.

Process improvements
@yuvipanda yuvipanda self-assigned this Oct 31, 2023
@choldgraf
Copy link
Member

If it's a useful datapoint, it looks like we have about $70,000 of un-billed cloud costs right now, based on our AirTable graph. I don't know if that's an exact number, but maybe is a ballpark.

CleanShot 2023-10-31 at 16 38 46@2x

@yuvipanda
Copy link
Member Author

ooof that is a lot of money.

@damianavila
Copy link
Contributor

Copy-pasting some thoughts I shared from Slack (that I think it would be useful to record here as well):

Can we make sure we (at least briefly) document the process in that very issue so other engineers (besides @yuvipanda) can help with this recurring process in the future? Thanks!

@yuvipanda yuvipanda mentioned this issue Nov 7, 2023
Closed
yuvipanda added a commit to yuvipanda/pilot-hubs that referenced this issue Nov 10, 2023
@yuvipanda
gcp: Tag nodes with their purpose for costing purposes
4699f1c 
Helps more easily separate the cost of various parts of
the infrastructure.

Ref 2i2c-org#3350
yuvipanda added a commit to yuvipanda/pilot-hubs that referenced this issue Nov 10, 2023
@yuvipanda
gcp: Tag nodes with their purpose for costing purposes
c7aa65c 
Helps more easily separate the cost of various parts of
the infrastructure.

This doesn't force a recreation of the nodepools, which is
pretty awesome. I tested this on the 2i2c-uk cluster, and
it rolled out without needing any node restarts.

Ref 2i2c-org#3350
@yuvipanda yuvipanda mentioned this issue Nov 10, 2023
Merged
13 tasks
yuvipanda added a commit to yuvipanda/pilot-hubs that referenced this issue Nov 10, 2023
@yuvipanda
gcp: Tag nodes with their purpose for costing purposes
8e6f669 
Helps more easily separate the cost of various parts of
the infrastructure.

Unfortunately this *does* recreate the nodes in the node-pool,
so will have to be rolled out slowly with appropriate timing.

I've rolled this out for the 2i2c-uk cluster.

Ref 2i2c-org#3350
@yuvipanda
Copy link
Member Author

Another goal is to build out some more infrastructure to make this easier next time.

#3397 tags nodes appropriately so we can easily differentiate between core and user and dask worker nodes.

@damianavila damianavila moved this from Needs Shaping / Refinement to In progress in DEPRECATED Engineering and Product Backlog Nov 15, 2023
@yuvipanda yuvipanda mentioned this issue Nov 16, 2023
Merged
@yuvipanda
Copy link
Member Author

Unfortunately due to #3429, the billing export for the billing account we used was changed in september, so our automation has broken from about mid-september :( I'll do this round manually as well, and document the process.

@yuvipanda
Copy link
Member Author

Moved to https://github.com/2i2c-org/meta/issues/742

@consideRatio
Copy link
Contributor

Closing this as the issue is otherwise duplicated with https://github.com/2i2c-org/meta/issues/742, and I think the intent was to close this.

@yuvipanda yuvipanda mentioned this issue Jan 2, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuvipanda yuvipanda

Labels
None yet
Projects
No open projects
DEPRECATED Engineering and Product Backlog
Status: Complete
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yuvipanda @damianavila @choldgraf @consideRatio