Remove setuptools from core dependencies

[ZiruiXuQB]

Description

I'm always frustrated when pip install kedro upgrades the versions of setuptools, wheel and pip.

Context

Usually I would not expect pip install to upgrade these packages unless I explicitly run python3 -m pip install --upgrade pip setuptools wheel.

The behaviour is even more surprising as kedro does not give a first impression of a package that is related to packaging or dependency management.

Possible Implementation

• wheel and pip dependencies will probably be excluded when pip-tools is removed from the core dependencies (by deprecating kedro build-reqs)
• setuptools can be moved to an optional dependency, kedro[package] to support the functionality of kedro package

Possible Alternatives

n/a

[datajoely]

super reasonable suggestion @ZiruiXuQB - I'm actually amazed this hasn't been suggested before!

[deepyaman]

I wouldn't move setuptools to an optional dependency, but rather remove it altogether. Instead of invoking setup.py directly, it's preferable to use python -m build (as of PEP 517, PEP 518). However, I suppose then build becomes a dependency...

Alternatively, you could just deprecate the kedro package command, because all it is is a thin wrapper on top of a setup.py invocation. I think I'd recommend this, since Kedro went through the exercise of evaluating CLI command usage and deprecating other thin wrappers; not sure why kedro package stats don't show up in that report.

Somewhat relevant (but not a solution): #2280

[noklam]

kedro package is just updated with the packaged config, which may improve the usage(?) @deepyaman

I am adding this to the inbox to make sure we discussed it in backlog grooming. kedro[pacakge] sounds reasonable to me, does micropkg also requires this? I haven't checked but probably yes?

[noklam]

Pinging @astrojuanlu

[astrojuanlu]

Now that I've been summoned 👹 let me just note that in the next pip release, if wheel is not installed and the project does not have a pyproject.toml, pip will not try setup.py install anymore. See pypa/pip#11874 (and related: pypa/pip#11860)

So, I think completing https://github.com/kedro-org/kedro/milestone/36 is becoming more urgent, and after that we could consider this issue.

[datajoely]

Fantastic push astro-👹

[astrojuanlu]

This is blocked by migrating to pyproject.toml (see https://github.com/kedro-org/kedro/milestone/36)

[noklam]

@astrojuanlu Can we close this already? I think for kedro it is move to pyproject.toml so these are only build requirements now?

[astrojuanlu]

kedro still has a dependency on pip-tools and setuptools:

kedro/pyproject.toml

Lines 31 to 36 in 664411d

	"pip-tools>=6.5,<8",
	"pluggy~=1.0",
	"PyYAML>=4.2, <7.0",
	"rich>=12.0, <14.0",
	"rope>=0.21, <2.0", # subject to LGPLv3 license
	"setuptools>=65.5.1",

IIUC, when we remove kedro build-reqs part of this issue will fix itself.

OTOH, kedro package now does not depend on setuptools anymore, and build does not depend on it either:

https://github.com/pypa/build/blob/381ce789e1662e2e16b0ca4be520cdcf7464f5a3/pyproject.toml#L34

I was under the impression that we needed to wait on #2853 to remove our direct dependency on setuptools, but maybe we can proceed already. That would fix the other half of the issue.

[astrojuanlu]

Double check micropkg though, I recall we use setuptools package discovery. The structure of kedro micropkg package could be changed into an src-layout to avoid this, but it would be a breaking change. If that's needed to close this issue though, maybe let's do it ahead of 0.19.

[astrojuanlu]

This is the code that uses setuptools on runtime:

kedro/kedro/framework/cli/micropkg.py

Lines 220 to 231 in ae5f6ff

	# Project name will be `my-pipeline` even if `pyproject.toml` says `my_pipeline`
	# because standards mandate normalization of names for comparison,
	# see https://packaging.python.org/en/latest/specifications/core-metadata/#name
	# The proper way to get it would be
	# project_name = library_meta.get("Name")
	# However, the rest of the code expects the non-normalized package name,
	# so we have to find it.
	packages = [
	package
	for package in FlatLayoutPackageFinder().find(project_root_dir)
	if "." not in package
	]

[astrojuanlu]

Ideas to remove dependency on setuptools:

• In pyproject.toml, explicitly declare the packages ("packages = ...") instead of using the autodiscovery ("tool.setuptools.packages.find").
• Alternatively, in the pyproject.toml generated by micropkg, add a [tool.kedro] section that contains package_name, so we don't have to guess it.

[merelcht]

Only removing setuptools is left to do.

[astrojuanlu]

🎉