Synthetic targets API

[kaos]

This is a post-fact kind of discussion, as the PR has already landed #16998.

Design doc: https://docs.google.com/document/d/1PPnXzUedm5mpPVuUdEm7WJppyiECHn28jIWOdp_uO3E/edit?usp=sharing

My motivation for this feature, as spelled out in #16979 was to address the desire to track changes to lockfiles in order to rebuild targets that depend on them rather than on changes to the requirements file, as there may not be a 1-to-1 mapping between requirements and lockfiles due to multiple resolves.

Another aspect of this is a slight shift nudging in the direction of a possible future having fewer targets explicitly declared in BUILD files as placeholders (that is, empty targets with no added configuration such as you get with tailor on a Python source tree sprinkling BUILD files with python_sources() in them..)

@thejcannon have two questions to address in particular:

I want to know how this might help a few things:

• 1. hierarchical target overrides (top level recursive glob for a target generator, then instead of the overrides dict, I define the metadata at the file subdir, and it augments the generated target)
• 2. reducing macros by allowing plugins to define many targets. Which AFAIK isn't supported well today.

For 1) I don't see the synthetic targets affecting this scenario at all as it stands today. It could potentially evolve in that direction somehow, let's have a thread in Slack about that.

2. Using this API plugins may populate any number of targets of any kind at any place in the build graph. There is a comprehensive docstring in https://github.com/kaos/pants/blob/00d2dbc8eb225eccd849ab654fe2e53955d184ce/src/python/pants/engine/internals/synthetic_targets.py that shows what this may look like, and an actual use case for Python lockfiles here:

pants/src/python/pants/backend/python/goals/lockfile.py

Lines 278 to 305 in 085e892

	@rule
	async def python_lockfile_synthetic_targets(
	request: PythonSyntheticLockfileTargetsRequest,
	python_setup: PythonSetup,
	) -> SyntheticAddressMaps:
	if (
	not python_setup.enable_resolves
	or request.path != SyntheticTargetsRequest.SINGLE_REQUEST_FOR_ALL_TARGETS
	):
	return SyntheticAddressMaps()
	resolves = [
	(os.path.dirname(lockfile), os.path.basename(lockfile), name)
	for name, lockfile in python_setup.resolves.items()
	]
	return SyntheticAddressMaps.for_targets_request(
	request,
	[
	(
	os.path.join(spec_path, "BUILD.python-lockfiles"),
	tuple(
	TargetAdaptor("lockfiles", name=name, sources=[lockfile])
	for _, lockfile, name in lockfiles
	),
	)
	for spec_path, lockfiles in itertools.groupby(sorted(resolves), key=itemgetter(0))
	],
	)

There are two primary modes of operation to populate the build graph with synthetic targets. A) if you know all your targets up-front without having to look for files in the project source tree you can get a single request and return all your targets with a single response, or for all other cases B) you get a request for each directory where a BUILD file is being parsed so there is a opportunity to add any synthetic targets that should live in that directory (presented as declared in a dedicated BUILD file for the plugin)

pants/src/python/pants/backend/python/goals/lockfile.py

Line 297 in 085e892

os.path.join(spec_path, "BUILD.python-lockfiles"),

The synthetic targets are loaded by the "parse address family" rule, so works the same for both BUILD file discovery as well as looking up a TargetAdaptor from an Address.

[thejcannon]

Specifically, the targets I wanna generate stem from https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/ (so generate a handful of PEXs and Docker images)

[kaos]

Yeah, that should work just fine, and it would all be peekable too ;)

[tdyas]

1. Would synthetic targets be able to live in a different "namespace" then on-disk targets? I.e., some form of "engine-addressed" targets: engine://jvm/scala2.12 could be a synthetic target provided by the JVM rules as a default scala requirement.
2. Any thoughts on the risk of user confusion if they do not find synthetic targets at the path specified in the targets' addresses if they were to look in BUILD files?

[kaos]

1. Would synthetic targets be able to live in a different "namespace" then on-disk targets? I.e., some form of "engine-addressed" targets: engine://jvm/scala2.12 could be a synthetic target provided by the JVM rules as a default scala requirement.

Part of the idea here is that you can override/replace a synthesized target, and as far as possible should be transparent to the user that it isn't a regular target. The only difference from a regular target is its source, where it originates from.

2. Any thoughts on the risk of user confusion if they do not find synthetic targets at the path specified in the targets' addresses if they were to look in BUILD files?

I don't think this will be a big problem. I did contemplate adding some marker to synthetic targets in order to discern them from other targets, but it was discouraged at this point in time. We can always revisit later if needed.

What could be useful though, along these lines in general, is to keep track of the source path to the BUILD file that a target originates from. I haven't checked but I believe there may be some assumptions in the Pants code base that all targets are declared in a single BUILD file per directory, but we do support parsing multiple build files per directory according to the build files pattern, and as such it could make sense to preserve this information in error messages for instance, when it could be useful to point back at where it was defined. In those cases synthetic targets would have a fictivious name but that would hint of its origin (as in the above example, for python lockfiles targets it would point back to a .../BUILD.python-lockfiles file)