Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto Scale Failure #1543

Closed
btylerburton opened this issue Feb 1, 2025 · 2 comments
Closed

Auto Scale Failure #1543

btylerburton opened this issue Feb 1, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@btylerburton
Copy link
Contributor

btylerburton commented Feb 1, 2025

Workflow with Issue: 7 - Scale catalog-web
Job Failed: scale
Last Commit: 9b9a234
Number of times run: 1
Last run by: FuhuXia
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/13092889269

@btylerburton btylerburton added the bug Something isn't working label Feb 1, 2025
@btylerburton
Copy link
Contributor Author

Image

Auto-scale failed to curl catalog with error code 22 which should be a 4xx series error.

@FuhuXia is curl expected to fail in this scenario? shouldn't cg router serve traffic only to healthy instances?

@btylerburton btylerburton moved this to 🏗 In Progress [8] in data.gov team board Feb 6, 2025
@FuhuXia
Copy link
Member

FuhuXia commented Feb 6, 2025

This is abnormal. Our scaling script expects to see catalog-web instance in running states. It does not know how to handle "crashed" state. We intentionally generate an error in this scenario because we want to be notified when catalog-web is in crashed state, or other abnormal state.

If it recovers on its own, then we don't need to worry about it. We need to take a closer look if it keeps coming back.

@FuhuXia FuhuXia closed this as completed Feb 6, 2025
@github-project-automation github-project-automation bot moved this from 🏗 In Progress [8] to ✔ Done in data.gov team board Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: ✔ Done
Development

No branches or pull requests

2 participants