Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unclear failure mode with "undefined" version #59

Closed
PeterJCLaw opened this issue Jan 30, 2024 · 16 comments · Fixed by #60
Closed

Unclear failure mode with "undefined" version #59

PeterJCLaw opened this issue Jan 30, 2024 · 16 comments · Fixed by #60
Assignees
Labels
awaiting-response Waiting for user feedback triage

Comments

@PeterJCLaw
Copy link
Contributor

PeterJCLaw commented Jan 30, 2024

I'm not completely sure what failed, however one a couple a few of our CI runs just failed on this action with:

Installing DVC from: https://dvc.org/download/linux-deb/dvc-undefined
DVC Download Failed, trying from GitHub Releases
Installing DVC from: https://github.com/iterative/dvc/releases/download/undefined/dvc_undefined_amd64.deb

A retry of the same run shortly afterwards succeeded.

My guess is that the logic to determine the version number failed, however did so in a manner which wasn't immediately an error. Perhaps if the version number lookup fails it should error immediately?

@shcheklein shcheklein added triage awaiting-response Waiting for user feedback labels Jan 30, 2024
@shcheklein shcheklein self-assigned this Jan 30, 2024
@shcheklein
Copy link
Member

hey, thanks for the repo. Yep, it seems that it was not able to determine the version. Does it happen often?

@PeterJCLaw
Copy link
Contributor Author

I hadn't seen it before this afternoon, though I've now seen it twice.

@shcheklein
Copy link
Member

Hmm. Hard to tell what's going on atm. It is relying on the https://updater.dvc.org/ service. And it's working fine (I think it's AWS lambda underneath it). Let's observe it for a bit, and if see this again - please let us know.

@PeterJCLaw
Copy link
Contributor Author

PeterJCLaw commented Jan 30, 2024

Are there any debug flags I could turn on in our CI which might yield more information?
For example, perhaps it would be useful to know if the error is in failing to talk to that service or whether that service is returning unexpected (but valid looking) results?

Edit: looks like not, based on reading the action.yml.

@shcheklein
Copy link
Member

Yes, I also don't see it. Here is the method I would assume that is getting the latest version id:

https://github.com/iterative/setup-dvc/blob/v1/src/utils.js#L35-L41

@PeterJCLaw
Copy link
Contributor Author

Hrm, there's no checking there of the HTTP status code. I'm not an expert on fetch though -- is that checked automatically and converted to an error? (Otherwise if the response was either empty json (perhaps {}) or an HTTP 4xx or 5xx then presumably the undefined error would be the outcome)

@PeterJCLaw
Copy link
Contributor Author

Looks like HTTP status codes are not considered exceptions and need explicit handling (via https://www.npmjs.com/package/node-fetch#handling-exceptions)

@shcheklein
Copy link
Member

Yep, we would be happy to accept a quick PR and release if you have time for this.

PeterJCLaw added a commit to PeterJCLaw/setup-dvc that referenced this issue Feb 1, 2024
This attempts to detect such errors, assuming that they appear with
suitable HTTP status codes, and raises useful errors for them.

Fixes iterative#59
PeterJCLaw added a commit to PeterJCLaw/setup-dvc that referenced this issue Feb 1, 2024
This attempts to detect such errors, assuming that they appear with
suitable HTTP status codes, and raises useful errors for them.

Fixes iterative#59
@PeterJCLaw
Copy link
Contributor Author

FWIW, I don't think I've seen this since the spate of occurrences around the time I raised this issue.

@PeterJCLaw
Copy link
Contributor Author

We've just seen this again. Any news on #60? I think that would at least help diagnose things.

0x2b3bfa0 added a commit that referenced this issue Apr 4, 2024
* Detect and handle errors fetching the latest version

This attempts to detect such errors, assuming that they appear with
suitable HTTP status codes, and raises useful errors for them.

Fixes #59

* Make a trivial change to trigger workflows

* Undo trivial change

---------

Co-authored-by: Helio Machado <[email protected]>
@shcheklein
Copy link
Member

@PeterJCLaw hey, sorry for the delay - we merged it.

@BotScutters
Copy link

I've seen this issue occurring a few times in the past couple of weeks.

Run iterative/setup-dvc@v1
  with:
    version: latest
  env:
    SEMVER_SUFFIX: pr
    SSH_AUTH_SOCK: /tmp/ssh-i8XfxLMrNycQ/agent.132198
    SSH_AGENT_PID: 132199
    AWS_DEFAULT_REGION: us-east-2
    AWS_REGION: us-east-2
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
    pythonLocation: /opt/hostedtoolcache/Python/3.10.15/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.15/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.15/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.15/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.15/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.15/x64/lib
Installing DVC from: https://dvc.org/download/linux-deb/dvc-undefined
DVC Download Failed, trying from GitHub Releases
Installing DVC from: https://github.com/iterative/dvc/releases/download/undefined/dvc_undefined_amd64.deb
Error: Not Found

We've had this action working as-is for ages and haven't changed anything about it, and only just started seeing this issue pop up last week, but it looks the same as what @PeterJCLaw previously reported.
@shcheklein I see that you merged a PR attempting to address this--any thoughts on what might be missing or additional changes that would help make this more robust? Do you think switching to v2 might help?
Thanks!

@0x2b3bfa0
Copy link
Member

Unable to reproduce; this works on my computer: ™

on: push
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: iterative/setup-dvc@v1
        with:
          version: latest
      - run: dvc doctor

@0x2b3bfa0
Copy link
Member

@BotScutters, please create a new issue if your issue persists, and share your runs-on and container configurations if applicable.

@shcheklein
Copy link
Member

Do you think switching to v2 might help?

if you can do that - I think it makes sense. I doubt it can help with this issue, but at least it will be using a non-deprecated Node version.

@BotScutters
Copy link

Unable to reproduce; this works on my computer: ™

on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: iterative/setup-dvc@v1
with:
version: latest
- run: dvc doctor

It mysteriously worked on fifth try of rerunning the action, so... even I can't reliably reproduce it. Seems like the root cause is likely upstream of you. I'll create a new ticket / investigate this further if/when it happens again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting-response Waiting for user feedback triage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants