PyPI package is missing Grammar.txt

[moubctez]

yapf --help

FileNotFoundError: [Errno 2] No such file or directory: '/MY/PATH/lib/python3.11/site-packages/yapf_third_party/_ylib2to3/Grammar.txt'

[bwendling]

Strange. Are the other files present in that directory? or are just the *.txt files gone?

[Spitfire1900]

I have not reproduced this on GitHub Codespaces, pipx (Ubuntu and Windows) or poetry virtualenv (Ubuntu and Windows).

What is the repro environment (including versions and package manager)?

[hartwork]

Linux + Python 3.10 + pip install + wheel yapf-0.40.0-py3-none-any.whl works fine.
I'd like to second @Spitfire1900's question about environment details.

[Spitfire1900]

From #1108 looks like this is affecting Homebrew, but that's insufficient information to determine a repro.

[moubctez]

I am trying to package v0.40.0 for PkgSrc (http://pkgsrc.org) and I need the source distfile, but it does not contain Grammar.txt. Here's the proof:

curl -L https://files.pythonhosted.org/packages/7a/cb/7675e1d2788ce93246f8c2e0e6ed00019c86853f92dc9226a90e0e1a1e95/yapf-0.40.0.tar.gz | tar ztf - | grep Grammar.txt

[Spitfire1900]

Confirmed, the sdist on pypi.org does not have the Grammer files.

Local testing shows that with setuptools==67.7.2 this does not repro though. #1108 (comment)

[Spitfire1900]

@bwendling , what is the reproduction environment for building the sdist for publishing to pypi? I'm getting a different output locally.

Looks like on pypi.org it's built against metadata-version 1.2 and for myself locally it's 2.1.

pypi-vs-setuptools==67.7.2.diff.txt

[bwendling]

I create it on a MacBook Pro. Nothing interesting about the environment. Just execute python3.9 setup.py sdist and let it fly.

[Spitfire1900]

@bwendling What version of python and setup tools?

[bwendling]

Python 3.9 and here's the pip version list:

% pip list
Package            Version
------------------ -------
attrs              21.2.0
bleach             4.1.0
coverage           6.1.1
docutils           0.18.1
flake8             4.0.1
importlib-metadata 6.6.0
iniconfig          1.1.1
mccabe             0.6.1
packaging          21.2
pip                23.1.2
platformdirs       3.5.3
pluggy             1.0.0
py                 1.10.0
pycodestyle        2.8.0
pyflakes           2.4.0
Pygments           2.10.0
pyparsing          2.4.7
pytest             6.2.5
pytest-cov         3.0.0
readme-renderer    32.0
setuptools         49.2.1
six                1.16.0
toml               0.10.2
tomli              2.0.1
webencodings       0.5.1
wheel              0.37.0
yapf               0.40.0
zipp               3.15.0

[bwendling]

I updated things and pushed a v0.40.1. Could you check it out please?

[hartwork]

I updated things and pushed a v0.40.1. Could you check it out please?

@bwendling I think that worked:

# cd "$(mktemp -d)"
# wget https://files.pythonhosted.org/packages/7a/cb/7675e1d2788ce93246f8c2e0e6ed00019c86853f92dc9226a90e0e1a1e95/yapf-0.40.0.tar.gz
# wget https://files.pythonhosted.org/packages/e0/7a/9020bfa17d294b5d0d8bf26bb175ad4c90d1e3ad4039001f621ef046cb06/yapf-0.40.1.tar.gz
# tar xf yapf-0.40.0.tar.gz
# tar xf yapf-0.40.1.tar.gz
# diff -r -q yapf-0.40.{0,1} | grep Only
Only in yapf-0.40.1/third_party/yapf_third_party/yapf_diff: LICENSE
Only in yapf-0.40.1/third_party/yapf_third_party/_ylib2to3: Grammar.txt
Only in yapf-0.40.1/third_party/yapf_third_party/_ylib2to3: LICENSE
Only in yapf-0.40.1/third_party/yapf_third_party/_ylib2to3: PatternGrammar.txt

What was the fix — upgrading setuptools?

[chenrui333]

What should I do to get this patch merged? We still need it for 0.40.1 release

[hartwork]

@chenrui333 cannot confirm for setuptools 67.8.0 on Linux (proof below). Could you elaborate what "we still need" means in detail?

For proof on Linux:

# cd "$(mktemp -d)"

# git clone --depth 1 https://github.com/google/yapf

# cd yapf/

# git rev-parse HEAD
54c679d4060a75d733bc96bacaab78c48710b785

# python3.10 -m venv venv

# source venv/bin/activate

# python -c 'import setuptools; print(setuptools.__version__)'
67.8.0

# rm -Rf build/ *.egg-info/ && python setup.py sdist

# tar tf dist/yapf-0.40.1.tar.gz | grep -i grammar
yapf-0.40.1/third_party/yapf_third_party/_ylib2to3/Grammar.txt
yapf-0.40.1/third_party/yapf_third_party/_ylib2to3/PatternGrammar.txt
yapf-0.40.1/third_party/yapf_third_party/_ylib2to3/pgen2/grammar.py

[chenrui333]

@hartwork Looks like you have them now in the 0.41.1 release, if possible, can you also point us to the related changes?

[hartwork]

@chenrui333 I didn't do the upload and there are no relevant changes in Git according to v.40.0...v0.40.1 .
I believe that @bwendling has the answers in what he changed about his environment. I'm hoping for him to enlighten us about that in detail.

[chenrui333]

ah ok, then you might still with my patch PR, but I will wait for the confirmation from bwendling first. Thanks!

[hartwork]

I played with different versions of setuptools now and found that it takes setuptools >=58.5.0 to work. I have created a pull request #1115 to make sure that future releases will use at least that version so that this issue cannot happen again. @bwendling can you confirm?

[bwendling]

What was the fix — upgrading setuptools?

Yeah. :-)

[bwendling]

@chenrui333 The fix was to upgrade my setuptools to the newest version. I'm very sorry about the trouble with the v0.40.0 release.