Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More refactoring for Resolve logic #2511

Merged
merged 20 commits into from
May 8, 2023
Merged

Conversation

lihaoyi
Copy link
Member

@lihaoyi lihaoyi commented May 7, 2023

Major Changes

  1. Break out main/resolve/ from main/, depending on main/define/ but not depending on main/eval/.

    1. Resolving tasks should not depend on Evaluator, and the core logic of Evaluator should not depend on task resolution (even if individual tasks might require it).
    2. Breaking it up into a separate modules enforces that, and helps simplify each module so the complex logic within each one can be reasoned about independently.
  2. Minimize the amount of actual instantiations we perform in ResolveCore

    1. We can postpone instantiating Targets and Commands until resolution is complete, removing a whole bunch of messy mainargs logic for the already-messy recursive resolution code. A lot of it ends up in Resolve.scala, which is somewhat more straightforward and better able to absorb the messiness
    2. We only need to instantiate Modules when trying to resolve tasks within a Cross module, since that requires us to instantiate the Cross object to see it's cross keys. Most resolution can take place purely by reflecting on java.lang.Class[_]s without needing the modules to actually be instantiated
    3. ResolveCore now only returns the segments: Segments for the things it resolves, leaving instantiation to Resolve.Tasks
    4. This significantly increases the laziness of ResolveCore, letting mill resolve work faster and work even when the modules cannot instantiate, unless there is a Cross module we need to instantiate
  3. Refactored ResolveNonEmpty to ResolveNotFoundHandler, to only be called by Resolve if ResolveCore returns a ResolveCore.NotFound. That helps avoid deep call hierarchies, and allows us to ensure that ResolveCore and ResolveNotFoundHandler independent of each other (rather than one wrapping the other)

  4. Extracted mill.define.Module.Internal into a separate mill.define.Reflect object. This helps clean up Module.scala. Ideally I wanted to move Reflect to mill.resolve, but it's difficult to do that while still providing nice helper methods on mill.define.Module objects e.g. millModuleDirectChildren while allowing them to be overriden etc.

  5. ResolveTasks is now Resolve.Tasks, ResolveSegments is now Resolve.Segments, ResolveMetadata is removed since Resolve.Segments does more or less the same thing anyway

  6. Make Cross#items a List[Item], so we can publicize it while continuing to evolve it in a binary-compatible manner

Testing

  1. Added more asserts to the existing ResolveTests.scala unit tests to check that ResolveSegments can still succeed even when ResolveTasks fails due to module initialization errors

  2. Added additional test cases to check the behavior of cross-module failures in more scenarios: when _ and __ is used, when the cross module itself fails to initialize (previously we only checked for children failing to initialize), and when the cross module's parent module fails to initialize

  3. Added a few test cases to integration/failure/module-init-error/ exercise the "resolve works even when module cannot initialize" behavior end-to-end

  4. Existing test cases in ResolveTests.scala and downstream tests that exercise task resolution all pass

@lihaoyi lihaoyi changed the title [WIP] Break out resolve from main into its own module [WIP] More re-structuring for Resolve logic May 7, 2023
@lihaoyi lihaoyi changed the title [WIP] More re-structuring for Resolve logic More re-structuring for Resolve logic May 8, 2023
@lihaoyi lihaoyi requested a review from lefou May 8, 2023 06:50
@lihaoyi lihaoyi changed the title More re-structuring for Resolve logic More refactoring for Resolve logic May 8, 2023
@lihaoyi lihaoyi marked this pull request as ready for review May 8, 2023 06:50
Copy link
Member

@lefou lefou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. I think, we should provide some @deprecated forwarders for the classes moved to mill.resolve in the mill.main package object. This should improve migration convenience and cross compilation.

@lihaoyi lihaoyi merged commit c12c5c5 into com-lihaoyi:main May 8, 2023
@lefou lefou added this to the 0.11.0-M9 milestone May 8, 2023
@lefou
Copy link
Member

lefou commented May 9, 2023

@lihaoyi We have one failing test in main and it seems related to this PR. Could you please look at it?

X mill.resolve.ResolveTests.overridenModule.0 5ms 
  utest.AssertionError: resolvedTasks.map(_.map(_.toString).toSet[String]) ==
  resolvedTasks: Either[String,List[mill.define.NamedTask[Any]]] = Left(Cannot resolve sub.inner.subTarget. Try `mill resolve sub.inner._` to see what's available.)
  expected: scala.util.Either[String,scala.collection.immutable.Set[mill.define.NamedTask[Any]]] = Right(Set(sub.BaseInnerModule.subTarget))
    utest.asserts.Asserts$.assertImpl(Asserts.scala:30)
    mill.resolve.ResolveTests$Checker.checkSeq(ResolveTests.scala:24)
    mill.resolve.ResolveTests$Checker.apply(ResolveTests.scala:14)
    mill.resolve.ResolveTests$.$anonfun$tests$257(ResolveTests.scala:754)

@lihaoyi
Copy link
Member Author

lihaoyi commented May 9, 2023

will take a look

@lihaoyi
Copy link
Member Author

lihaoyi commented May 9, 2023

CI is green now, but I do not understand what caused the breakage, nor whay fixed it, so we'll have to see if the issue reappears. If it turns out the test is just flaky then we may be forced to disable it

1 similar comment
@lihaoyi
Copy link
Member Author

lihaoyi commented May 9, 2023

CI is green now, but I do not understand what caused the breakage, nor whay fixed it, so we'll have to see if the issue reappears. If it turns out the test is just flaky then we may be forced to disable it

lefou added a commit that referenced this pull request Feb 4, 2024
This PR add support for new selector pattern `_:Type`.

In addition to `_` and `__`, which select arbitrary segments, the
`_:MyType` and `__:MyType` patterns can select modules of the specified
type.

The type is matched by it's name and optionally by it's enclosing types
and packages, separated by a `.` sign. Since this is also used to
separate target path segments, a type selector segment containing a `.`
needs to be enclosed in parenthesis. A full qualified type can be
enforced with the `_root_` package.

Example: Find all test jars

```sh
> mill resolve __:TestModule.jar
> mill resolve "(__:scalalib.TestModule).jar"
> mill resolve "(__:mill.scalalib.TestModule).jar"
> mill resolve "(__:_root_.mill.scalalib.TestModule).jar"
```

If a `^` or `!` is preceding the type pattern, it only matches segments
not an instance of that specified type. Please note that in some shells
like `bash`, you need to mask the `!` character.

Example: Find all jars not in test modules

```sh
> mill resolve __:^TestModule.jar
```

You can also provide more than one type pattern, separated with `:`. 

Example: Find all `JavaModule`s which are not `ScalaModule`s or
`TestModule`s:

```sh
> mill resolve "__:JavaModule:^ScalaModule:^TestModule.jar"
```

Remarks:

* Kudos to @lihaoyi who refactored the resolver in
#2511 and made this PR possible.
I tried to implement it multiple times before, and ever got bitten by
the old gnarly resolver code.
* It's currently not possible to match task/target types. It might be
possible, but due to `Task` being a parametrized type, it might not be
as easy to implement and use.

Fix #1550

Pull request: #2997
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants