Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.2.2: codespell script installed in wrong location #2592

Closed
kloczek opened this issue Nov 12, 2022 · 44 comments · Fixed by #2595
Closed

2.2.2: codespell script installed in wrong location #2592

kloczek opened this issue Nov 12, 2022 · 44 comments · Fixed by #2595
Labels
packaging packaging related issue

Comments

@kloczek
Copy link

kloczek commented Nov 12, 2022

During package 2.2.2 I found that codespell script is not installed in bindir.

Provides: codespell = 2.2.2-3.fc35 python3.8dist(codespell) = 2.2.2 python3dist(codespell) = 2.2.2
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires: /usr/bin/python3 python(abi) = 3.8
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/tkloczko/rpmbuild/BUILDROOT/codespell-2.2.2-3.fc35.x86_64
error: Installed (but unpackaged) file(s) found:
   /usr/lib/python3.8/site-packages/bin/codespell
@kloczek
Copy link
Author

kloczek commented Nov 12, 2022

Looks as well like test suite is installed as well (conten of the codespell_lib/tests/)

@kloczek
Copy link
Author

kloczek commented Nov 12, 2022

Because in pyproject.toml already is [project.scripts] section first issue can be solved by delete bin/ directiry and include bin/codespell from MANIFEST.in.

@DimitriPapadopoulos
Copy link
Collaborator

Is this an issue with codespell itself, or the Fedora package of codespell?

@kloczek
Copy link
Author

kloczek commented Nov 12, 2022

I'm not using Fedora.
Fedora package is not using pep517 build procedure which I'm using to generate .whl archive.
If you want to test that procedure please use python3 -sBm build -w --no-isolation command to testwhat lands in generated .whl archove.

@kloczek
Copy link
Author

kloczek commented Nov 12, 2022

Just checked and in Fedora spec file in %install is line which removes test suite from buildroot.
https://src.fedoraproject.org/rpms/codespell/blob/rawhide/f/codespell.spec#_33

@DimitriPapadopoulos
Copy link
Collaborator

DimitriPapadopoulos commented Nov 14, 2022

Sorry, I noticed the fc35 in Provides: codespell = 2.2.2-3.fc35 and assumed this was somehow occurring while packaging on or for Fedora 35, or something similar. My wrong.

I am not sure how to reproduce this issue, or where to find documentation on duplicates in MANIFEST.in and pyproject.toml. Hopefully, someone else will review this.

@kloczek
Copy link
Author

kloczek commented Nov 14, 2022

Sorry, I noticed the fc35 in Provides: codespell = 2.2.2-3.fc35 and assumed this was somehow occurring while packaging on or for Fedora 35, or something similar. My wrong.

And my wrong as well. I shoud change %dist macro value as well to not confuse peopel like you 😋

I am not sure how to reproduce this issue, or where to find documentation on duplicates in MANIFEST.in and pyproject.toml. Hopefully, someone else will review this.

My typical build proceduyte which uses pap517 consist from:

  • python3 -sBm build -w --no-isolation
  • because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
  • install .whl file in </install/prefix>
  • run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

@DimitriPapadopoulos
Copy link
Collaborator

DimitriPapadopoulos commented Nov 15, 2022

The command line option -s is not an option on my machine:

Logs of "python3.8 -Bm build -w --no-isolation", directly on Ubuntu 18.04
$ python3.8 -sBm build -w --no-isolation
/usr/bin/python3.8: No module named build
$ 
$ python3.8 -Bm build -w --no-isolation
[...]
adding 'bin/codespell'
adding 'codespell_lib/__init__.py'
adding 'codespell_lib/__main__.py'
adding 'codespell_lib/_codespell.py'
adding 'codespell_lib/_version.py'
adding 'codespell_lib/data/__init__.py'
adding 'codespell_lib/data/dictionary.txt'
adding 'codespell_lib/data/dictionary_code.txt'
adding 'codespell_lib/data/dictionary_en-GB_to_en-US.txt'
adding 'codespell_lib/data/dictionary_informal.txt'
adding 'codespell_lib/data/dictionary_names.txt'
adding 'codespell_lib/data/dictionary_rare.txt'
adding 'codespell_lib/data/dictionary_usage.txt'
adding 'codespell_lib/data/linux-kernel.exclude'
adding 'codespell_lib/tests/__init__.py'
adding 'codespell_lib/tests/test_basic.py'
adding 'codespell_lib/tests/test_dictionary.py'
adding 'codespell-2.2.3.dev65+g3f05fd62.dist-info/METADATA'
adding 'codespell-2.2.3.dev65+g3f05fd62.dist-info/WHEEL'
adding 'codespell-2.2.3.dev65+g3f05fd62.dist-info/entry_points.txt'
adding 'codespell-2.2.3.dev65+g3f05fd62.dist-info/top_level.txt'
adding 'codespell-2.2.3.dev65+g3f05fd62.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built codespell-2.2.3.dev65+g3f05fd62-py3-none-any.whl
$ 
$ unzip -t dist/codespell-2.2.3.dev65+g3f05fd62-py3-none-any.whl 
Archive:  dist/codespell-2.2.3.dev65+g3f05fd62-py3-none-any.whl
    testing: bin/codespell            OK
    testing: codespell_lib/__init__.py   OK
    testing: codespell_lib/__main__.py   OK
    testing: codespell_lib/_codespell.py   OK
    testing: codespell_lib/_version.py   OK
    testing: codespell_lib/data/__init__.py   OK
    testing: codespell_lib/data/dictionary.txt   OK
    testing: codespell_lib/data/dictionary_code.txt   OK
    testing: codespell_lib/data/dictionary_en-GB_to_en-US.txt   OK
    testing: codespell_lib/data/dictionary_informal.txt   OK
    testing: codespell_lib/data/dictionary_names.txt   OK
    testing: codespell_lib/data/dictionary_rare.txt   OK
    testing: codespell_lib/data/dictionary_usage.txt   OK
    testing: codespell_lib/data/linux-kernel.exclude   OK
    testing: codespell_lib/tests/__init__.py   OK
    testing: codespell_lib/tests/test_basic.py   OK
    testing: codespell_lib/tests/test_dictionary.py   OK
    testing: codespell-2.2.3.dev65+g3f05fd62.dist-info/METADATA   OK
    testing: codespell-2.2.3.dev65+g3f05fd62.dist-info/WHEEL   OK
    testing: codespell-2.2.3.dev65+g3f05fd62.dist-info/entry_points.txt   OK
    testing: codespell-2.2.3.dev65+g3f05fd62.dist-info/top_level.txt   OK
    testing: codespell-2.2.3.dev65+g3f05fd62.dist-info/RECORD   OK
No errors detected in compressed data of dist/codespell-2.2.3.dev65+g3f05fd62-py3-none-any.whl.
$ 

I do not see bin/codespell missing from the resulting wheel.

I do see the tests in the resulting wheel, but I don't know whether they should be excluded or not from binary distributions such as wheels. The Python Packaging User Guide seems to imply they should not be excluded from source distributions at least:

The following files are included in a source distribution by default:

  • ...
  • all files matching the pattern test/test*.py

I am surprised changing MANIFEST.in could affect binary distributions such as wheels, because the Python Packaging User Guide states:

Note: MANIFEST.in does not affect binary distributions such as wheels.

I cannot keep the current pace of changes in Python packaging, it is a poorly documented mess. Do you know of any documentation on migrating from setup.py to pyproject.toml and how MANIFEST.in is supposed to evolve in this context?

@DimitriPapadopoulos
Copy link
Collaborator

To clarify, is this a regression, meaning you can reproduce this issue with version 2.2.2 but not earlier versions?

@kloczek
Copy link
Author

kloczek commented Nov 15, 2022

To clarify, is this a regression, meaning you can reproduce this issue with version 2.2.2 but not earlier versions?

Yes. Prev version had no such issues.

@DimitriPapadopoulos
Copy link
Collaborator

Have you made sure pip, setuptools and wheel are updated to the latest versions before packaging?

As I wrote, I cannot easily reproduce this issue on my machine (maybe because I cannot use the -s option), and the documentation suggests MANIFEST.in does not affect binary distributions such as wheels.

@kloczek
Copy link
Author

kloczek commented Nov 15, 2022

Have you made sure pip, setuptools and wheel are updated to the latest versions before packaging?

Here is list of modules with versions installed in build env

Package              Version
-------------------- --------------
appdirs              1.4.4
attrs                22.1.0
Brlapi               0.8.3
build                0.9.0
chardet              5.0.0
contourpy            1.0.6
cssselect            1.1.0
cycler               0.11.0
distro               1.8.0
dnspython            2.2.1
extras               1.0.0
fixtures             4.0.0
fonttools            4.38.0
gpg                  1.17.1-unknown
importlib-resources  5.9.0
iniconfig            1.1.1
kiwisolver           1.4.4
libcomps             0.1.19
louis                3.23.0
lxml                 4.9.1
matplotlib           3.6.2
numpy                1.23.1
olefile              0.46
packaging            21.3
pbr                  5.9.0
pep517               0.13.0
Pillow               9.3.0
pip                  22.2.2
pkgutil_resolve_name 1.3.10
platformdirs         2.5.2
pluggy               1.0.0
py                   1.11.0
PyGObject            3.42.2
pyparsing            3.0.9
pyrsistent           0.19.2
pytest               7.1.3
python-dateutil      2.8.2
rpm                  4.17.0
scour                0.38.2
setuptools           65.5.1
setuptools-scm       7.0.5
six                  1.16.0
testtools            2.5.0
tomli                2.0.1
traitlets            5.4.0
typing_extensions    4.3.0
wheel                0.37.1
zipp                 3.9.0

@DimitriPapadopoulos DimitriPapadopoulos added the packaging packaging related issue label Nov 16, 2022
@DimitriPapadopoulos
Copy link
Collaborator

Version 2.2.2 moved from setup.py to setuptools.toml, see 754b9a5 / #2523. Chances are this issue is related. Unfortunately, I cannot reproduce it.

@DimitriPapadopoulos
Copy link
Collaborator

Again, my suggestion is to update Python packages. In your case, pip can be upgraded from 22.2.2 to 22.3.1, and
wheel from 0.37.1 to 0.38.4. Please do try these suggestions before reporting back.

See Python Packaging User Guide:

Ensure pip, setuptools, and wheel are up to date

While pip alone is sufficient to install from pre-built binary archives, up to date copies of the setuptools and wheel projects are useful to ensure you can also install from source archives:

python3 -m pip install --upgrade pip setuptools wheel

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

Again, my suggestion is to update Python packages. In your case, pip can be upgraded from 22.2.2 to 22.3.1, and
wheel from 0.37.1 to 0.38.4. Please do try these suggestions before reporting back.

Did you truy try to use pep517 build procedure?

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

I want only to say that none of those modiules are involved in pep517 based build procedure.

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

FYI: I've updated pip and wheel and as expected pep517 based still fails with the same error message.

@DimitriPapadopoulos
Copy link
Collaborator

We really need a way to reproduce this, not abstract things like Did you try try to use pep517 build procedure? I have done what you suggested and am unable to reproduce the issue at hand:

  • python3 -sBm build -w --no-isolation in a venv on Ubuntu 18.04 with python3.8
  • python3 -Bm build -w --no-isolation directly on Ubuntu 18.04 with python3.8

Speaking for myself here, I cannot spend more time trying to find a way to reproduce this. I don't know enough about Python packing, and cannot spend time learning more in the short term, to be able to infer the root cause of this issue.

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

Could you pelase show me results of what you done?

@DimitriPapadopoulos
Copy link
Collaborator

See details in #2592 (comment).

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

Just FTR .. Debian actibvely is switching now to pep517 and Fedora is in that porocess more than year.

I have 100% of my packages with python modules (+1.05k total) switched to pep517.
Amongs all those modules anly handfull have problems with pep517 builds

[tkloczko@devel-g2v SPECS]$ grep BUG: python-*spec| grep pep517
python-flexmock.spec:# BUG: pep517 build issue https://github.com/flexmock/flexmock/issues/120
python-frozenlist.spec:# BUG: pep517 build fails https://github.com/aio-libs/frozenlist/issues/296 temporary stay on sdist source tar ball
python-hidapi.spec:# BUG: pep517 buld is broken https://github.com/trezor/cython-hidapi/issues/133
python-jupyterlite.spec:# BUG: pep517 build fails https://github.com/jupyterlite/jupyterlite/issues/652
python-kiwisolver.spec:# BUG: pep517 warnings https://github.com/nucleic/kiwi/issues/139
python-multipledispatch.spec:# BUG: missing six in venv for pep517 based build https://github.com/mrocklin/multipledispatch/issues/120
python-openstackdocstheme.spec:# BUG: pep517 warnings
python-pep517.spec:# BUG: flake8 tests are failing https://github.com/pypa/pep517/issues/127
python-plotly.spec:# BUG: pep517 build fails when used VCS tar ball https://github.com/plotly/plotly.py/issues/3669
python-pydantic.spec:# BUG: pep517 build is broken https://github.com/samuelcolvin/pydantic/issues/3577
python-pyjson5.spec:# BUG: pep517 build fails https://github.com/Kijewski/pyjson5/issues/29
python-pytest-toolbox.spec:# BUG: pep517 build fails https://github.com/samuelcolvin/pytest-toolbox/issues/53

@DimitriPapadopoulos
Copy link
Collaborator

I understand that, and I wish I could help, but I have absolutely no clue how to reproduce this issue.

I cannot find codespell in the above log, I don't understand how it relates to codespell and how it helps solve the issue at hand.

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

I understand that, and I wish I could help, but I have absolutely no clue how to reproduce this issue.

All what you need to do is:

  • install build module
  • execute python3 -sBm build -w --no-isolation

All that needs to be bone in git tree where is (redundant) bin/ directory.

@DimitriPapadopoulos
Copy link
Collaborator

DimitriPapadopoulos commented Nov 16, 2022

Please help understand what you mean exactly by a “git tree where is (redundant) bin/ directory”. Should we remove bin/codespell from the Git tree?

  • I have installed build and run python3.8 -Bm build -w --no-isolation directly on my Ubuntu 18.04 machine, without -s because I cannot use it on my machine. See details in 2.2.2: codespell script installed in wrong location #2592 (comment).
  • I have installed build and run python -sBm build -w --no-isolation inside a venv built with python3.8 on my Ubuntu 18.04 machine. See details below.
Logs of "python -sBm build -w --no-isolation", inside a python3.8 venv on Ubuntu 18.04
$ python3.8 -m venv /path/pep517
$ 
$ 
$ . /path/pep517/bin/activate
(pep517) $ 
(pep517) $ 
(pep517) $ pip install --upgrade pip setuptools setuptools_scm wheel build
Collecting pip
  Using cached https://files.pythonhosted.org/packages/09/bd/2410905c76ee14c62baf69e3f4aa780226c1bbfc9485731ad018e35b0cb5/pip-22.3.1-py3-none-any.whl
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/b6/40/353c051f77ee5618adaf1fd96f4f6bae9714ed0a22c7142c01c24eb77fe4/setuptools-65.5.1-py3-none-any.whl
Collecting setuptools_scm
  Using cached https://files.pythonhosted.org/packages/01/ed/75a20e7b075e8ecb1f84e8debf833917905d8790b78008915bd68dddd5c4/setuptools_scm-7.0.5-py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/bd/7c/d38a0b30ce22fc26ed7dbc087c6d00851fb3395e9d0dac40bec1f905030c/wheel-0.38.4-py3-none-any.whl
Collecting build
  Using cached https://files.pythonhosted.org/packages/03/97/f58c723ff036a8d8b4d3115377c0a37ed05c1f68dd9a0d66dab5e82c5c1c/build-0.9.0-py3-none-any.whl
Collecting typing-extensions (from setuptools_scm)
  Using cached https://files.pythonhosted.org/packages/0b/8e/f1a0a5a76cfef77e1eb6004cb49e5f8d72634da638420b9ea492ce8305e8/typing_extensions-4.4.0-py3-none-any.whl
Collecting packaging>=20.0 (from setuptools_scm)
  Using cached https://files.pythonhosted.org/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl
Collecting tomli>=1.0.0 (from setuptools_scm)
  Using cached https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl
Collecting pep517>=0.9.1 (from build)
  Using cached https://files.pythonhosted.org/packages/ee/2f/ef63e64e9429111e73d3d6cbee80591672d16f2725e648ebc52096f3d323/pep517-0.13.0-py3-none-any.whl
Collecting pyparsing!=3.0.5,>=2.0.2 (from packaging>=20.0->setuptools_scm)
  Using cached https://files.pythonhosted.org/packages/6c/10/a7d0fa5baea8fe7b50f448ab742f26f52b80bfca85ac2be9d35cdd9a3246/pyparsing-3.0.9-py3-none-any.whl
Installing collected packages: pip, setuptools, typing-extensions, pyparsing, packaging, tomli, setuptools-scm, wheel, pep517, build
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
  Found existing installation: setuptools 39.0.1
    Uninstalling setuptools-39.0.1:
      Successfully uninstalled setuptools-39.0.1
Successfully installed build-0.9.0 packaging-21.3 pep517-0.13.0 pip-22.3.1 pyparsing-3.0.9 setuptools-65.5.1 setuptools-scm-7.0.5 tomli-2.0.1 typing-extensions-4.4.0 wheel-0.38.4
(pep517) $ 
(pep517) $ 
(pep517) $ python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
/path/pep517/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)
running egg_info
creating codespell.egg-info
writing codespell.egg-info/PKG-INFO
writing dependency_links to codespell.egg-info/dependency_links.txt
writing entry points to codespell.egg-info/entry_points.txt
writing requirements to codespell.egg-info/requires.txt
writing top-level names to codespell.egg-info/top_level.txt
writing manifest file 'codespell.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'example'
warning: no previously-included files found matching 'snap'
warning: no previously-included files found matching 'tools'
warning: no previously-included files found matching 'tools/*'
writing manifest file 'codespell.egg-info/SOURCES.txt'
* Building wheel...
/path/pep517/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  warnings.warn(msg, _BetaConfiguration)
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/codespell_lib
copying codespell_lib/__init__.py -> build/lib/codespell_lib
copying codespell_lib/__main__.py -> build/lib/codespell_lib
copying codespell_lib/_version.py -> build/lib/codespell_lib
copying codespell_lib/_codespell.py -> build/lib/codespell_lib
creating build/lib/codespell_lib/tests
copying codespell_lib/tests/test_basic.py -> build/lib/codespell_lib/tests
copying codespell_lib/tests/__init__.py -> build/lib/codespell_lib/tests
copying codespell_lib/tests/test_dictionary.py -> build/lib/codespell_lib/tests
creating build/lib/codespell_lib/data
copying codespell_lib/data/__init__.py -> build/lib/codespell_lib/data
running egg_info
writing codespell.egg-info/PKG-INFO
writing dependency_links to codespell.egg-info/dependency_links.txt
writing entry points to codespell.egg-info/entry_points.txt
writing requirements to codespell.egg-info/requires.txt
writing top-level names to codespell.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'example'
warning: no previously-included files found matching 'snap'
warning: no previously-included files found matching 'tools'
warning: no previously-included files found matching 'tools/*'
writing manifest file 'codespell.egg-info/SOURCES.txt'
copying codespell_lib/py.typed -> build/lib/codespell_lib
copying codespell_lib/data/dictionary_en-GB_to_en-US.txt -> build/lib/codespell_lib/data
copying codespell_lib/data/dictionary_code.txt -> build/lib/codespell_lib/data
copying codespell_lib/data/dictionary.txt -> build/lib/codespell_lib/data
copying codespell_lib/data/dictionary_informal.txt -> build/lib/codespell_lib/data
copying codespell_lib/data/dictionary_names.txt -> build/lib/codespell_lib/data
copying codespell_lib/data/dictionary_rare.txt -> build/lib/codespell_lib/data
copying codespell_lib/data/dictionary_usage.txt -> build/lib/codespell_lib/data
copying codespell_lib/data/linux-kernel.exclude -> build/lib/codespell_lib/data
creating build/lib/bin
copying bin/codespell -> build/lib/bin
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/codespell_lib
creating build/bdist.linux-x86_64/wheel/codespell_lib/tests
copying build/lib/codespell_lib/tests/test_basic.py -> build/bdist.linux-x86_64/wheel/codespell_lib/tests
copying build/lib/codespell_lib/tests/__init__.py -> build/bdist.linux-x86_64/wheel/codespell_lib/tests
copying build/lib/codespell_lib/tests/test_dictionary.py -> build/bdist.linux-x86_64/wheel/codespell_lib/tests
copying build/lib/codespell_lib/__init__.py -> build/bdist.linux-x86_64/wheel/codespell_lib
copying build/lib/codespell_lib/py.typed -> build/bdist.linux-x86_64/wheel/codespell_lib
copying build/lib/codespell_lib/__main__.py -> build/bdist.linux-x86_64/wheel/codespell_lib
copying build/lib/codespell_lib/_version.py -> build/bdist.linux-x86_64/wheel/codespell_lib
creating build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/data/linux-kernel.exclude -> build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/data/__init__.py -> build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/data/dictionary_en-GB_to_en-US.txt -> build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/data/dictionary_code.txt -> build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/data/dictionary.txt -> build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/data/dictionary_informal.txt -> build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/data/dictionary_names.txt -> build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/data/dictionary_rare.txt -> build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/data/dictionary_usage.txt -> build/bdist.linux-x86_64/wheel/codespell_lib/data
copying build/lib/codespell_lib/_codespell.py -> build/bdist.linux-x86_64/wheel/codespell_lib
creating build/bdist.linux-x86_64/wheel/bin
copying build/lib/bin/codespell -> build/bdist.linux-x86_64/wheel/bin
running install_egg_info
Copying codespell.egg-info to build/bdist.linux-x86_64/wheel/codespell-2.2.3.dev81+g1302d797-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/codespell-2.2.3.dev81+g1302d797.dist-info/WHEEL
creating '/path/src/codespell/dist/tmp4ssu29zo/codespell-2.2.3.dev81+g1302d797-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'bin/codespell'
adding 'codespell_lib/__init__.py'
adding 'codespell_lib/__main__.py'
adding 'codespell_lib/_codespell.py'
adding 'codespell_lib/_version.py'
adding 'codespell_lib/py.typed'
adding 'codespell_lib/data/__init__.py'
adding 'codespell_lib/data/dictionary.txt'
adding 'codespell_lib/data/dictionary_code.txt'
adding 'codespell_lib/data/dictionary_en-GB_to_en-US.txt'
adding 'codespell_lib/data/dictionary_informal.txt'
adding 'codespell_lib/data/dictionary_names.txt'
adding 'codespell_lib/data/dictionary_rare.txt'
adding 'codespell_lib/data/dictionary_usage.txt'
adding 'codespell_lib/data/linux-kernel.exclude'
adding 'codespell_lib/tests/__init__.py'
adding 'codespell_lib/tests/test_basic.py'
adding 'codespell_lib/tests/test_dictionary.py'
adding 'codespell-2.2.3.dev81+g1302d797.dist-info/METADATA'
adding 'codespell-2.2.3.dev81+g1302d797.dist-info/WHEEL'
adding 'codespell-2.2.3.dev81+g1302d797.dist-info/entry_points.txt'
adding 'codespell-2.2.3.dev81+g1302d797.dist-info/top_level.txt'
adding 'codespell-2.2.3.dev81+g1302d797.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built codespell-2.2.3.dev81+g1302d797-py3-none-any.whl
(pep517) $ 
(pep517) $ 
(pep517) $ unzip -t dist/codespell-2.2.3.dev81+g1302d797-py3-none-any.whl 
Archive:  dist/codespell-2.2.3.dev81+g1302d797-py3-none-any.whl
    testing: bin/codespell            OK
    testing: codespell_lib/__init__.py   OK
    testing: codespell_lib/__main__.py   OK
    testing: codespell_lib/_codespell.py   OK
    testing: codespell_lib/_version.py   OK
    testing: codespell_lib/py.typed   OK
    testing: codespell_lib/data/__init__.py   OK
    testing: codespell_lib/data/dictionary.txt   OK
    testing: codespell_lib/data/dictionary_code.txt   OK
    testing: codespell_lib/data/dictionary_en-GB_to_en-US.txt   OK
    testing: codespell_lib/data/dictionary_informal.txt   OK
    testing: codespell_lib/data/dictionary_names.txt   OK
    testing: codespell_lib/data/dictionary_rare.txt   OK
    testing: codespell_lib/data/dictionary_usage.txt   OK
    testing: codespell_lib/data/linux-kernel.exclude   OK
    testing: codespell_lib/tests/__init__.py   OK
    testing: codespell_lib/tests/test_basic.py   OK
    testing: codespell_lib/tests/test_dictionary.py   OK
    testing: codespell-2.2.3.dev81+g1302d797.dist-info/METADATA   OK
    testing: codespell-2.2.3.dev81+g1302d797.dist-info/WHEEL   OK
    testing: codespell-2.2.3.dev81+g1302d797.dist-info/entry_points.txt   OK
    testing: codespell-2.2.3.dev81+g1302d797.dist-info/top_level.txt   OK
    testing: codespell-2.2.3.dev81+g1302d797.dist-info/RECORD   OK
No errors detected in compressed data of dist/codespell-2.2.3.dev81+g1302d797-py3-none-any.whl.
(pep517) $ 

@DimitriPapadopoulos
Copy link
Collaborator

Actually, if you have a patch or a clue how to fix this issue, why not create a merge request? It might be easier for me to understand.

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

creating build/bdist.linux-x86_64/wheel/bin
copying build/lib/bin/codespell -> build/bdist.linux-x86_64/wheel/bin

So as you see still existin in git repo bin/ content is not needed.
Just try to remove bin/ an drepeat build and you will see that this script will be generated.

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

Actually, if you have a patch or a clue how to fix this issue, why not create a merge request? It might be easier for me to understand.

All what you need to do is remove from git repo bin/.

DimitriPapadopoulos added a commit to DimitriPapadopoulos/codespell that referenced this issue Nov 16, 2022
DimitriPapadopoulos added a commit to DimitriPapadopoulos/codespell that referenced this issue Nov 16, 2022
@DimitriPapadopoulos
Copy link
Collaborator

DimitriPapadopoulos commented Nov 16, 2022

So as you see still existin in git repo bin/ content is not needed. Just try to remove bin/ an drepeat build and you will see that this script will be generated.

Yes, that is exactly what I had understood from your explanation at some point, and I had already tried removing bin/codespell from the Git repository. Unfortunately, that had broken our tests. I had probably missed something, trying again in #2595.

I suspect I might need to fix this:

codespell/Makefile

Lines 15 to 17 in 1302d79

codespell.1: codespell.1.include bin/codespell Makefile
PYTHONPATH=. help2man ./bin/codespell --include codespell.1.include --no-info --output codespell.1
sed -i '/\.SS \"Usage/,+2d' codespell.1

Providing a merge request and a link to documentation would have been so much easier for both of us ☹️

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

Again .. you have already in pyproject.toml

codespell/pyproject.toml

Lines 50 to 51 in 1302d79

[project.scripts]
codespell = "codespell_lib:_script_main"

This part is responsble for generating that script.
Everything elese can be removed.

@DimitriPapadopoulos
Copy link
Collaborator

Not sure about the test suite, though. Should I remove it?

The Python Packaging User Guide seems to imply they should not be excluded, at least from source distributions:

The following files are included in a source distribution by default:

  • ...
  • all files matching the pattern test/test*.py

@kloczek
Copy link
Author

kloczek commented Nov 16, 2022

I've mentioned about pyprojecto.toml entry first time here:
#2592 (comment)

@DimitriPapadopoulos
Copy link
Collaborator

What about the test suite?

@kloczek
Copy link
Author

kloczek commented Nov 17, 2022

Please do not remove it. Just move from codespell_lib/tests/ to tests/ (that may require some test suite adjustments in test suite is fyou are using relative imports).
tests/ content is automatically skipped on generate .whl archive.

@kloczek
Copy link
Author

kloczek commented Nov 17, 2022

Looks like test suite in current form does not contain any relative paths import lines so just move codespell_lib/tests/ to tests/ shoud skip include test suite files in .whl.

@DimitriPapadopoulos
Copy link
Collaborator

DimitriPapadopoulos commented Nov 17, 2022

If I understand correctly:

  • the Python Packaging User Guide implies tests should be included in source distributions,
  • tests should be removed from binary distributions – any documentation on the subject?

I think we want to keep tests under codespell_lib, see #2381. Therefore, we need an explicit way to remove tests from binary distributions. I will have a look at numpy or scikit-learn.

@DimitriPapadopoulos
Copy link
Collaborator

pypa/setuptools-scm#190 might be of interest.

@kloczek
Copy link
Author

kloczek commented Nov 17, 2022

Coud you at least try to move test suite files to tests/ in your workig directory to see result?

@DimitriPapadopoulos
Copy link
Collaborator

Sure, I can do that. It is worth trying, but in the end, we need an alternative solution.

@DimitriPapadopoulos
Copy link
Collaborator

The issue remains that all these changes break some targets in the Makefile:

  • codespell.1: Not sure how to modify help2man ./bin/codespell so that it still works after deleting bin/codespell.
  • check-distutils: See make check-distutils #2599, this one seems easy, we can move from setup.py check to twine check.

@kloczek
Copy link
Author

kloczek commented Nov 17, 2022

Using setup.py <foo> is now deprecated.
Please notice that instead should be used any pep517 based procedure.

DimitriPapadopoulos added a commit to DimitriPapadopoulos/codespell that referenced this issue Nov 17, 2022
Source distributions such as sdist must include test directories.
Note that all files matching the pattern `test/test*.py` (or perhaps
`tests/test*.py` with an `s`?) are included implicitly:
	https://packaging.python.org/guides/using-manifest-in/#how-files-are-included-in-an-sdist

Binary distributions such as wheel must exclude test directories.
This seems to be done implicitly for top-level `tests/` directories
only. I do not know how to explicitly exclude files from binary
distributions. This setuptools issue might be of interest:
	pypa/setuptools-scm#190

I change `codespell_lib/tests/` to a top-level `tests/` directory as
a workaround. Not that I like it, but I currently lack an alternative.

Fixes (partially) codespell-project#2592.
@DimitriPapadopoulos
Copy link
Collaborator

Yes, we know, this is exactly what I have suggested: setup.py checktwine check

@DimitriPapadopoulos
Copy link
Collaborator

DimitriPapadopoulos commented Nov 17, 2022

Back to tests:

I have created a separate branch tests for your codespell_lib/tests/tests/ suggestion. My commit should be (almost) identical to previous merge request #2381, which has been rejected. Indeed, maintainers do not necessarily agree:

@DimitriPapadopoulos
Copy link
Collaborator

And finally back to removing bin/codespell:

Help is welcome to fix the codespell.1 target in the Makefile:

codespell/Makefile

Lines 15 to 17 in 1302d79

codespell.1: codespell.1.include bin/codespell Makefile
PYTHONPATH=. help2man ./bin/codespell --include codespell.1.include --no-info --output codespell.1
sed -i '/\.SS \"Usage/,+2d' codespell.1

DimitriPapadopoulos added a commit to DimitriPapadopoulos/codespell that referenced this issue Nov 17, 2022
DimitriPapadopoulos added a commit to DimitriPapadopoulos/codespell that referenced this issue Nov 20, 2022
@kloczek
Copy link
Author

kloczek commented Nov 29, 2022

It is +100 commits since last release and it would be a bit hard to integrate that with my build procedure.
I'll test tat after make new release.
If it will be no more comments here it will mean that everything is fine.

Thank you 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
packaging packaging related issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants