Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Requesting AllTargets in a plugin fails if an entry in [python.resolves] points to a nonexistent file #18933

Closed
danxmoran opened this issue May 7, 2023 · 2 comments · Fixed by #18940
Closed

Requesting AllTargets in a plugin fails if an entry in [python.resolves] points to a nonexistent file #18933

danxmoran opened this issue May 7, 2023 · 2 comments · Fixed by #18940
Assignees
@kaos
Labels
backend: Python Python backend-related issues bug
Milestone
2.16.x

Comments

@danxmoran
Copy link
Contributor

Describe the bug

In one of our in-repo plugins, we request:

await Get(AllTargets, AllTargetsRequest())

This typically works fine, but if I add a new [python.resolves] entry:

[python.resolves]
new = "path/to/new.lockfile"

Then the goal logic explodes with:

Traceback (most recent call last):
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/color/build/pants/plugins/lockfile/pex/register.py", line 304, in lock_goal
    all_targets = await Get(AllTargets, AllTargetsRequest())
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/graph.py", line 455, in find_all_targets
    tgts = await Get(
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/specs_rules.py", line 175, in addresses_from_raw_specs_without_file_owners
    target_parametrizations_list = await MultiGet(
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 358, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/graph.py", line 283, in resolve_target_parametrizations
    all_generated = await MultiGet(
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 358, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/graph.py", line 1418, in generate_file_targets
    sources_paths = await Get(
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/graph.py", line 1031, in resolve_source_paths
    paths = await Get(Paths, PathGlobs, path_globs)
  File "/Users/dan.moran/Library/Caches/nce/56153e74337dc4e840948775afc9384890c3683383a5da29460a7aa1ccb965f6/bindings/venvs/2.16.0rc0+git06aaef7f/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
native_engine.IntrinsicError: Unmatched glob from path/to:_new_lockfile's `sources` field: "path/to/new.lockfile"

Pants version

PANTS_SHA=06aaef7f21092a7bd76ba157cd221266b8444653 (on the 2.16.x branch)

OS

MacOS but would presumably see it on Linux too if I pushed my branch to CI.

Additional info

I believe the error is happening because the rule that provides AllTargets here wraps around:

await Get(
    Targets,
    RawSpecsWithoutFileOwners(
        recursive_globs=(RecursiveGlobSpec(""),),
        description_of_origin="the `AllTargets` rule"
    ),
)

RawSpecsWithoutFileOwners has an unmatched_glob_behavior field here that defaults to GlobMatchErrorBehavior.error.
@danxmoran danxmoran added the bug label May 7, 2023
@danxmoran
Copy link
Contributor Author

Hrmmm although - it looks like that unmatched_glob_behavior field is only used in the RawSpecsWithoutFileOwners.to_build_file_path_globs_tuple method. If I replace my AllTargets request with this:

all_targets = await Get(
    Targets,
    RawSpecsWithoutFileOwners(
        recursive_globs=(RecursiveGlobSpec(""),),
        description_of_origin="the `lock` goal",
        unmatched_glob_behavior=GlobMatchErrorBehavior.ignore,
    ),
)

I still get the Unmatched glob intrinsic error 🤔

@danxmoran
Copy link
Contributor Author

This also seems to bite pants check when mypy is enabled, because its logic includes an await Get(AllTargets, AllTargetsRequest())

@tdyas tdyas added the backend: Python Python backend-related issues label May 8, 2023
@danxmoran danxmoran mentioned this issue May 8, 2023
Merged
@kaos kaos added this to the 2.16.x milestone May 8, 2023
@kaos kaos self-assigned this May 8, 2023
kaos added a commit to kaos/pants that referenced this issue May 8, 2023
@kaos
Do not choke on missing lockfiles.
2df7b5f 
Fixes pantsbuild#18933 (and pantsbuild#18404). Original attempt at fixing this but failed
was in pantsbuild#18406 due to an oversight that it was the `_lockfiles` target
generator with a `sources` field that was being synthesized, not the
`_lockfile` target with a single `source` file only.

Either use should work equally well, and now they do.
@kaos kaos mentioned this issue May 8, 2023
Merged
kaos added a commit that referenced this issue May 9, 2023
@kaos
Do not choke on missing lockfiles. (#18940)
fa7ea81 
Fixes #18933 (and #18404). Original attempt at fixing this but failed
was in #18406 due to an oversight that it was the `_lockfiles` target
generator with a `sources` field that was being synthesized, not the
`_lockfile` target with a single `source` file only.

Either use should work equally well, and now they do.
@kaos kaos mentioned this issue May 9, 2023
Merged
kaos added a commit to kaos/pants that referenced this issue May 9, 2023
@kaos
Do not choke on missing lockfiles. (pantsbuild#18940)
69be43c 
Fixes pantsbuild#18933 (and pantsbuild#18404). Original attempt at fixing this but failed
was in pantsbuild#18406 due to an oversight that it was the `_lockfiles` target
generator with a `sources` field that was being synthesized, not the
`_lockfile` target with a single `source` file only.

Either use should work equally well, and now they do.
@kaos kaos mentioned this issue May 9, 2023
Merged
kaos added a commit to kaos/pants that referenced this issue May 9, 2023
@kaos
Do not choke on missing lockfiles. (pantsbuild#18940)
d8b18ae 
Fixes pantsbuild#18933 (and pantsbuild#18404). Original attempt at fixing this but failed
was in pantsbuild#18406 due to an oversight that it was the `_lockfiles` target
generator with a `sources` field that was being synthesized, not the
`_lockfile` target with a single `source` file only.

Either use should work equally well, and now they do.
kaos added a commit that referenced this issue May 9, 2023
@kaos
Do not choke on missing lockfiles. (Cherry-pick of #18940) (#18948)
f1b0ce0 
Fixes #18933 (and #18404). Original attempt at fixing this but failed
was in #18406 due to an oversight that it was the `_lockfiles` target
generator with a `sources` field that was being synthesized, not the
`_lockfile` target with a single `source` file only.

Either use should work equally well, and now they do.
kaos added a commit that referenced this issue May 9, 2023
@kaos
Do not choke on missing lockfiles. (Cherry-pick of #18940) (#18947)
d858b9d 
Fixes #18933 (and #18404). Original attempt at fixing this but failed
was in #18406 due to an oversight that it was the `_lockfiles` target
generator with a `sources` field that was being synthesized, not the
`_lockfile` target with a single `source` file only.

Either use should work equally well, and now they do.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kaos kaos

Labels
backend: Python Python backend-related issues bug
Projects
None yet
Milestone
2.16.x
Development

Successfully merging a pull request may close this issue.

Do not choke on missing lockfiles. kaos/pants
3 participants
@kaos @tdyas @danxmoran