PEP518 - setup.py is deprecated

[philschwartzsonos]

Looks like newer versions of pip3 and several updated packages have adopted PEP518 that has deprecated the use of setup.py in favor of projecttoml.py

OS: ubuntu 20.04
Python version: 3.10
Pip3 version: 22.1.2

$ fpm -s python -t deb --python-bin=python3 --python-pip=pip3 --python-package-name-prefix=python3 --python-install-lib=/usr/lib/python3/dist-packages --python-install-bin=/usr/bin --deb-no-default-config-files chardet
{:timestamp=>"2023-01-21T11:17:07.978577+0000", :message=>"Could not find 'setup.py'", :path=>"/tmp/package-python-build-7031c224a18e2eadd5ce547d7798236fcd57971f78cd71741a3246c33a30/chardet/chardet-5.1.0/setup.py", :level=>:error}
error occurred building package: chardet
re-running with verbose output: fpm -s python -t deb --python-bin=python3 --python-pip=pip3 --python-package-name-prefix=python3 --python-install-lib=/usr/lib/python3/dist-packages --python-install-bin=/usr/bin --deb-no-default-config-files --verbose chardet
{:timestamp=>"2023-01-21T11:17:08.246296+0000", :message=>"Setting workdir", :workdir=>"/tmp", :level=>:info}
{:timestamp=>"2023-01-21T11:17:08.249694+0000", :message=>"Trying to download", :package=>"chardet", :level=>:info}
{:timestamp=>"2023-01-21T11:17:08.568979+0000", :message=>"Looking in indexes: https://pypi.python.org/simple", :level=>:info}
{:timestamp=>"2023-01-21T11:17:08.734182+0000", :message=>"Collecting chardet", :level=>:info}
{:timestamp=>"2023-01-21T11:17:08.740048+0000", :message=>" Using cached chardet-5.1.0.tar.gz (2.1 MB)", :level=>:info}
{:timestamp=>"2023-01-21T11:17:09.055844+0000", :message=>" Installing build dependencies: started", :level=>:info}
{:timestamp=>"2023-01-21T11:17:15.553840+0000", :message=>" Installing build dependencies: finished with status 'done'", :level=>:info}
{:timestamp=>"2023-01-21T11:17:15.556456+0000", :message=>" Getting requirements to build wheel: started", :level=>:info}
{:timestamp=>"2023-01-21T11:17:15.907519+0000", :message=>" Getting requirements to build wheel: finished with status 'done'", :level=>:info}
{:timestamp=>"2023-01-21T11:17:16.036982+0000", :message=>" Installing backend dependencies: started", :level=>:info}
{:timestamp=>"2023-01-21T11:17:18.464475+0000", :message=>" Installing backend dependencies: finished with status 'done'", :level=>:info}
{:timestamp=>"2023-01-21T11:17:18.467011+0000", :message=>" Preparing metadata (pyproject.toml): started", :level=>:info}
{:timestamp=>"2023-01-21T11:17:18.820436+0000", :message=>" Preparing metadata (pyproject.toml): finished with status 'done'", :level=>:info}
{:timestamp=>"2023-01-21T11:17:18.823612+0000", :message=>"Saved /tmp/package-python-build-2ba88f12b4bda6cf08c776540dcddbaf55a4669d695e925c3c0cca7a11ed/chardet-5.1.0.tar.gz", :level=>:info}
{:timestamp=>"2023-01-21T11:17:18.823790+0000", :message=>"Successfully downloaded chardet", :level=>:info}
{:timestamp=>"2023-01-21T11:17:18.929421+0000", :message=>"Could not find 'setup.py'", :path=>"/tmp/package-python-build-2ba88f12b4bda6cf08c776540dcddbaf55a4669d695e925c3c0cca7a11ed/chardet/chardet-5.1.0/setup.py", :level=>:error}
/var/lib/gems/3.0.0/gems/fpm-1.13.1/lib/fpm/package/python.rb:108:in input': Unable to find python package; tried /tmp/package-python-build-2ba88f12b4bda6cf08c776540dcddbaf55a4669d695e925c3c0cca7a11ed/chardet/chardet-5.1.0/setup.py (RuntimeError) from /var/lib/gems/3.0.0/gems/fpm-1.13.1/lib/fpm/command.rb:352:in block in execute'
from /var/lib/gems/3.0.0/gems/fpm-1.13.1/lib/fpm/command.rb:351:in each' from /var/lib/gems/3.0.0/gems/fpm-1.13.1/lib/fpm/command.rb:351:in execute'
from /var/lib/gems/3.0.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in run' from /var/lib/gems/3.0.0/gems/fpm-1.13.1/lib/fpm/command.rb:574:in run'
from /var/lib/gems/3.0.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in run' from /var/lib/gems/3.0.0/gems/fpm-1.13.1/bin/fpm:7:in <top (required)>'
from /usr/local/bin/fpm:25:in load' from /usr/local/bin/fpm:25:in

'

In addition to the "chardet" package, I see the same issue with "filelock" and "prettytable"

In looking into it a bit, I saw a post that you can shim a setup.py file into the build, something like this

import setuptools

if name == "main":
setuptools.setup()

Looks like I can workaround the issue for now by downgrading pip3:
$ pip3 install pip==21.1.2
$ fpm -s python -t deb --python-bin=python3 --python-pip=pip3 --python-package-name-prefix=python3 --python-install-lib=/usr/lib/python3/dist-packages --python-install-bin=/usr/bin --deb-no-default-config-files chardet
Created package {:path=>"python3-chardet_5.0.0_all.deb"}

However, this maybe more difficult in the future as more packages drop setup.py support for newer versions. In this case, pip3 21.1.2 installed chardet 5.0.0 whereas fpm/pip3 22.1.2 tried to build 5.1.0.

[bugfood]

I ran into a module which doesn't have setup.py:
https://pypi.org/project/pika/

There are two parts in the fpm code which rely on setup.py:

fpm/lib/fpm/package/python.rb

Lines 111 to 112 in ad6b18a

	load_package_info(setup_py)
	install_to_staging(setup_py)

I haven't looked at install_to_staging() at all, but I attempted to write a replacement for load_package_info(). Loading the toml data is straightforward, given a suitable ruby gem (tomlrb), but I'm really uncertain how to handle the file specification.
https://peps.python.org/pep-0621/
Specifically, the parts about "synonyms". The PEP seems to have grandfathered in existing uses from other tools. Or something--I'm not familiar with python packaging.
It would be feasible (though tedious) to support all listed synonyms, but it really seems like there should be a better way.

The existing way to handle setup.py is to use some helper python code. That approach could be better, but I didn't see anything in setuptools for parsing pyproject.toml that isn't explicitly labeled as private. Maybe there's something higher-level I didn't see.

I posted my work here:
main...bugfood:fpm:python-modules-new
I'm uncertain about it, so not submitting a PR at this time.
The first three commits are for a separate PR, with the last two commits being relevant to this issue.

-Corey

[jordansissel]

Thanks for the reports! I figured this day would come eventually when setup.py would go away.

I’m in favor of adding support for the new style and will do some reading about the newer toml project configs and see what we can do.

@philschwartzsonos i saw you have a bunch of open issues. I am available for a short term contract to get my energy focused on this and the other issues you have open. Happy to discuss further :)

[philschwartzsonos]

Hey Jordan, that sounds like it could work for us but I'd have to run it through management. It would be helpful if you could provide your rate and estimate of hours it would take you to implement all of my features/issues I've submitted. Also, not sure if there is a more direct way to communicate with you either via GitHub or personal email, etc... Thanks.

[bugfood]

I didn't see this earlier:
#1982
That approach seems better than my attempt, if it can be finished.

This particular issue also duplicates #1873.

[jordansissel]

@philschwartzsonos I’ll work on a project proposal scoped to those issues to send your way. You can reach me at [email protected] :)