[Feature Request]: Expand --extra_package option to allow passing a directory containing the package and deprecate the --setup_file option

[tvalentyn]

What would you like to happen?

The --setup_file option is becoming a misnomer now that many packages are defined with pyproject.toml. I think we can simplify the API and the documentation by sticking with the --extra_package option and allow it to accept a source directory of any python package in addition to sdists/bdist, which is already allowed.

As a bonus point, users could specify many pipeline packages as sources. The pipeline package is no different from any other packages being installed . In fact, some users build an sdist with their pipeline code and pass it via --extra_package.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

cc: @nitobuendia FYI

[lgeiger]

From quickly scanning the code to me it looks like the current code would work as expected just by also allowing pyproject.toml in this check:

beam/sdks/python/apache_beam/runners/portability/stager.py

Lines 285 to 288 in d24d838

	if os.path.basename(setup_options.setup_file) != 'setup.py':
	raise RuntimeError(
	'The --setup_file option expects the full path to a file named '
	'setup.py instead of %s' % setup_options.setup_file)

Or am I missing something?

[tvalentyn]

yes, it will work; the naming is not great but overall we can add this simple change.

[tvalentyn]

@lgeiger feel free to send a PR if you are interested to make this change.