Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Cloud Run out of memory #90

Open
jonaraphael opened this issue Aug 8, 2023 · 1 comment
Open

Error: Cloud Run out of memory #90

jonaraphael opened this issue Aug 8, 2023 · 1 comment

Comments

@jonaraphael
Copy link
Collaborator

jonaraphael commented Aug 8, 2023

Memory limit of 5120 MiB exceeded with 5261 MiB used.
Happens about 25 times / day with the orchestrator (@5gb).
Happens about 10 times / day with the offset tiles (@5gb).

Can be mitigated by increasing memory value in the following locations. Note that if you increase it substantially, you will also have to increase the number of vCPUs according to https://cloud.google.com/run/docs/configuring/memory-limits

cerulean-cloud/stack/cloud_run_orchestrator.py
cerulean-cloud/stack/cloud_run_offset_tile.py

@jonaraphael jonaraphael changed the title Error Cloud Run out of memory Error: Cloud Run out of memory Aug 8, 2023
@jonaraphael
Copy link
Collaborator Author

Reduced:
Happens about 25 times / day with the orchestrator (@6gb).
Happens about 0 times / day with the offset tiles (@6gb).

Examples of Scenes triggering the failure in the orchestrator:
S1A_IW_GRDH_1SDV_20230809T080754_20230809T080823_049796_05FD0B_1519
S1A_IW_GRDH_1SDV_20230809T064557_20230809T064622_049795_05FD02_7AAF
S1A_IW_GRDH_1SDV_20230808T234551_20230808T234620_049791_05FCD9_CEA1
S1A_IW_GRDH_1SDV_20230809T050408_20230809T050433_049794_05FCF7_3878
S1A_IW_GRDH_1SDV_20230809T050314_20230809T050343_049794_05FCF7_1027

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant