Generate Rules/Types from subsystems

[thejcannon]

In several places we hand-write a rule and a type to get a single option from a susbsytem (usually the global subsystem).

We can and should make this generic so this is much easier to do.

[thejcannon]

This is intended to generically handle classes like

pants/src/python/pants/option/global_options.py

Line 1983 in 2307d75

class ProcessCleanupOption:

and

pants/src/python/pants/option/global_options.py

Line 1993 in 2307d75

class NamedCachesDirOption:

[gauthamnair]

@thejcannon was pondering this, and if you still think the problem would be worth the weight of adding some tooling, here is a proposal.

We could avoid having to manually define the rule with an API like (details to be worked out in names and types, treat as pseudocode!):

@dataclass(frozen=True)
class NamedCachesDirOption(ExtractedOption):
    subsystem = GlobalOptions  #ClassVar
    attribute_name = 'named_caches_dir' 
    val: PurePath

We could arrange for the superclass to automatically construct the rule:

class ExtractedOption
    subsystem: ClassVar[Type[Subsystem]]
    attribute_name: ClassVar[str]
    val: any   # will have to work out how to get typing to work, but there's probably some nice way to do it.

    @classmethod
    def rule(cls):
        # do something like what Subsystem._construct_subsystem_rule() does to craft rules
        # but the actual getting will be some flavor of getattr on the instance

and we can have collect_rules automatically register the rule if any rule that consumes the type gets collected, much as it currently does to register the Subsystem rules:

# def collect_rules(...):
    if isinstance(rule, TaskRule):
        for input in rule.input_selectors:
            if issubclass(input, Subsystem):
                yield from input.rules()
            if issubclass(input, Subsystem.EnvironmentAware):
                yield from input.subsystem.rules()
            if issubclass(input, ExtractedOption):  # this would be new
                yield input.rule()
        if issubclass(rule.output_type, Goal):
            yield from rule.output_type.subsystem_cls.rules()
        yield rule

So the user would just have to define the option type and then:

@rule(desc="Prepare environment for running PEXes", level=LogLevel.DEBUG)
async def find_pex_python(
    python_bootstrap: PythonBootstrap,
    # ... 
    named_caches_dir: NamedCachesDirOption,
) -> PexEnvironment:

We can probably notch up the magic in the class declaration if it feels worth it, and some things need to be worked out regarding preserving type information.

[thejcannon]

I actually have a working prototype of using a different syntax that's much more intuitive, but also mypy doesn't understand (pyrigjt does though)