python: "Unexpected directory layout ..."

[jordansissel]

Forked this issue from #1780

Relevant comments for this issue:

• fpm 1.12.0, Debian 11: python to deb: easy_install executable not available, but required. #1780 (comment)
• fpm 1.12.0, Debian 11: python to deb: easy_install executable not available, but required. #1780 (comment)

Summary: Converting python to deb is failing when using pip.

I'd like to figure out how to reproduce this, or at least dig a little deeper to help me understand where the problem in fpm might be.

Reported cases:

• fpm --python-bin python3 --python-package-name-prefix python3 -s python -t deb --python-dis able-dependency py-radix --iteration $CI_PIPELINE_ID aggregate6
• fpm -s python -t deb --python-package-name-prefix python3 --python-bin=python3 argparse

[jordansissel]

@devicenull and @amdei, can y'all help provide any additional info? I do believe this is a bug somewhere in fpm, and learning more about how/why it fails will help in fixing it.

Both of the reported cases work for me, unfortunately:

% CI_PIPELINE_ID=1; fpm --python-bin python3 --python-package-name-prefix python3 -s python -t deb --python-dis
able-dependency py-radix --iteration $CI_PIPELINE_ID aggregate6
Created package {:path=>"python3-aggregate6_1.0.12-1_all.deb"}

% fpm -s python -t deb --python-package-name-prefix python3  --python-bin=python3 argparse
Created package {:path=>"python3-argparse_1.4.0_all.deb"}

Python version info. I'm testing on Ubuntu 20.04. fpm 1.14.1

% python3 --version
Python 3.8.10
% python3 -m pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

[devicenull]

We're building on Debian 11 - we build in a docker image built like this

FROM debian:bullseye
RUN apt-get update
RUN apt-get install -y apt-utils apt-transport-https gnupg gnupg2
RUN apt-get update
RUN apt-get install -y ruby rubygems-integration ruby-dev gcc make patch curl  git wget quilt dpkg-dev devscripts ninja-build build-essential libc6-dev
RUN gem install fpm

And then our build script does

apt-get install -y python3-dev python3-pip
fpm --python-bin python3 --python-package-name-prefix python3 -s python -t deb --python-disable-dependency py-radix --iteration $CI_PIPELINE_ID aggregate6

Versions:

# python3 -m pip --version
pip 20.3.4 from /usr/lib/python3/dist-packages/pip (python 3.9)

[jordansissel]

@devicenull Oh nice! This made it easy to reproduce. Thank you :)

root@b3a818affb14:/# CI_PIPELINE_ID=1
root@b3a818affb14:/# fpm --python-bin python3 --python-package-name-prefix python3 -s python -t deb --python-disable-dependency py-radix --iteration $CI_PIPELINE_ID aggregate6
Traceback (most recent call last):
...
         1: from /var/lib/gems/2.7.0/gems/fpm-1.14.1/lib/fpm/package/python.rb:98:in `input'
/var/lib/gems/2.7.0/gems/fpm-1.14.1/lib/fpm/package/python.rb:184:in `download_if_necessary': Unexpected directory layout after easy_install. Maybe file a bug? The directory is /tmp/package-python-build-6c433fb19f55bb5aacd75846cf042aa0480f54fb6d656705b18a5717063a (RuntimeError)

[jordansissel]

I ran this with --debug-workspace which tells fpm to keep any temporary directories around, and I found this:

root@b3a818affb14:/tmp/package-python-build-f715004495b75a4f098048dde4802ced7c3d8ca8ccf051be4bd14b991209# ls -lR
.:
total 12
drwxr-xr-x 2 root root 4096 Nov 11 18:34 aggregate6
-rw-r--r-- 1 root root 6026 Nov 11 18:34 aggregate6-1.0.12.tar.gz

./aggregate6:
total 0

the aggregate6 directory is empty, unexpectedly. I think I hvae enough info now to start looking into the issue and hopefully finding a fix. Thank you!

[fujexo]

Ran into this today when building:

podman run tenzer/fpm:latest -s python -t deb hvac

/usr/local/bundle/gems/fpm-1.14.1/lib/fpm/package/python.rb:184:in `download_if_necessary': Unexpected directory layout after easy_install. Maybe file a bug? The directory is /tmp/package-python-build-9f1e03e30436dc08831ef73fd95eb4bf4c14f8e8a1b33aa6212a7fa3465f (RuntimeError)
        from /usr/local/bundle/gems/fpm-1.14.1/lib/fpm/package/python.rb:98:in `input'
        from /usr/local/bundle/gems/fpm-1.14.1/lib/fpm/command.rb:352:in `block in execute'
        from /usr/local/bundle/gems/fpm-1.14.1/lib/fpm/command.rb:351:in `each'
        from /usr/local/bundle/gems/fpm-1.14.1/lib/fpm/command.rb:351:in `execute'
        from /usr/local/bundle/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'
        from /usr/local/bundle/gems/fpm-1.14.1/lib/fpm/command.rb:574:in `run'
        from /usr/local/bundle/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'
        from /usr/local/bundle/gems/fpm-1.14.1/bin/fpm:7:in `<top (required)>'
        from /usr/local/bundle/bin/fpm:23:in `load'
        from /usr/local/bundle/bin/fpm:23:in `<main>'

[ulricvbs]

If it helps, I'm also seeing this same issue:
/var/lib/gems/2.7.0/gems/fpm-1.14.1/lib/fpm/package/python.rb:184:in 'download_if_necessary': Unexpected directory layout after easy_install. Maybe file a bug?

Additionally, with verbose on, pip is returning the following message in regards to upcoming version:
DEPRECATION: The -b/--build/--build-dir/--build-directory option is deprecated and has no effect anymore. pip 21.1 will remove support for this functionality. A possible replacement is use the TMPDIR/TEMP/TMP environment variable, possibly combined with --no-clean. You can find discussion regarding this at https://github.com/pypa/pip/issues/8333. {:level=>:info}

[ObjatieGroba]

FROM debian:bullseye-slim
RUN apt update && apt install -y python3-pip ruby-dev
RUN gem install fpm

root@bf66ff626882:/# fpm --verbose -s python -t deb --python-package-name-prefix python3 --python-pip=pip3 --python-bin=python3 argparse
Setting workdir {:workdir=>"/tmp", :level=>:info}
Trying to download {:package=>"argparse", :level=>:info}
DEPRECATION: The -b/--build/--build-dir/--build-directory option is deprecated and has no effect anymore. pip 21.1 will remove support for this functionality. A possible replacement is use the TMPDIR/TEMP/TMP environment variable, possibly combined with --no-clean. You can find discussion regarding this at https://github.com/pypa/pip/issues/8333. {:level=>:info}
Looking in indexes: https://pypi.python.org/simple {:level=>:info}
Collecting argparse {:level=>:info}
  Using cached argparse-1.4.0.tar.gz (70 kB) {:level=>:info}
Saved /tmp/package-python-build-7625aa650f5ef7e23a2a11b7ff49a29cbb5599fc581c562881da9f77b946/argparse-1.4.0.tar.gz {:level=>:info}
Successfully downloaded argparse {:level=>:info}
Traceback (most recent call last):
	10: from /usr/local/bin/fpm:23:in `<main>'
	 9: from /usr/local/bin/fpm:23:in `load'
	 8: from /var/lib/gems/2.7.0/gems/fpm-1.14.1/bin/fpm:7:in `<top (required)>'
	 7: from /var/lib/gems/2.7.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'
	 6: from /var/lib/gems/2.7.0/gems/fpm-1.14.1/lib/fpm/command.rb:574:in `run'
	 5: from /var/lib/gems/2.7.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'
	 4: from /var/lib/gems/2.7.0/gems/fpm-1.14.1/lib/fpm/command.rb:351:in `execute'
	 3: from /var/lib/gems/2.7.0/gems/fpm-1.14.1/lib/fpm/command.rb:351:in `each'
	 2: from /var/lib/gems/2.7.0/gems/fpm-1.14.1/lib/fpm/command.rb:352:in `block in execute'
	 1: from /var/lib/gems/2.7.0/gems/fpm-1.14.1/lib/fpm/package/python.rb:98:in `input'
/var/lib/gems/2.7.0/gems/fpm-1.14.1/lib/fpm/package/python.rb:184:in `download_if_necessary': Unexpected directory layout after easy_install. Maybe file a bug? The directory is /tmp/package-python-build-7625aa650f5ef7e23a2a11b7ff49a29cbb5599fc581c562881da9f77b946 (RuntimeError)

root@:/# fpm --version
1.14.1

root@:/# dpkg -l | grep python3-pip
python3-pip                            20.3.4-4

Downgrading setuptools to 50 does not affect...

This means that there are no solution today for generate deb packages(

[ObjatieGroba]

Oh, I found solution

1. Downgrade setuptools to 51
2. Downgrade fpm to 1.12.0
3. DO NOT USE arg --python-pip=pip3

The last two points confuse...

[ObjatieGroba]

In my opinion the best way is remove easy_install and use pip3 for generating package in folder as issue #1873 suggest

It is --target option that can help a lot.

pip install package --target /tmp/folder/folder/folder

https://pip.pypa.io/en/stable/cli/pip_install/?highlight=target#cmdoption-t

[devicenull]

fwiw - this is what I ended up doing

apt-get install -y python3-dev ruby-dev
gem install fpm --version "1.12.0"
gem uninstall fpm --version "1.14.1"
apt-get remove -y python3-setuptools python3-pip
wget https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py "pip < 20.0" "setuptools < 51.0"
# debian of course named py-radix just radix
fpm --python-bin python3 --python-package-name-prefix python3 -s python -t deb --python-easyinstall /usr/local/bin/easy_install --python-disable-dependency py-radix --iteration $CI_PIPELINE_ID aggregate6

Which is dumb, but at least I have a working deb (and I've done it in a container so I don't care about the python mess it leaves behind)

[AutisticShark]

Can confirm the same issue still exists in the latest release of fpm.

OS: RHEL 9 x86_64
Python: 3.9.10
pip: 21.2.3
fpm: 1.14.2

[WilliamDEdwards]

This bit me too. Here's my solution for using a Python source with a deb target on Debian 11.

Due to this issue, fpm does not work with pip. Therefore, we have to use easy_install. Installing easy_install on Debian 11 is not easy (pun intended). Notes:

• Starting from fpm 1.14.0, pip is used by default (By default, use pip for fetching and installing python packages. #1821). It is possible to use easy_install though, with --no-python-internal-pip. It is NOT needed to downgrade fpm. Just install the latest version (1.14.2 at the time of writing).
• easy_install was removed from setuptools in version 52.0.0 (Remove easy_install script and module. pypa/setuptools#2544). Bullseye comes with setuptools 52.0.0, so easy_install is no longer installed.
• Stretch came with setuptools 33.1.1, and Buster came with 40.8.0. I was thinking of using Buster as easy_install is still installed on those systems, but fpm doesn't install on it anymore due to Dependency Error public_suffix requires Ruby version >= 2.6 and breaks installing on Debian 10 / Buster #1923.

So, we have to install an older version of setuptools. I'll use pip to install the last version before the one in which easy_install was removed. Only do this on volatile systems, like a build container. It could severely f%#* your system.

apt install python3-pip
pip3 install setuptools==51.3.3

I am now able to build a package with the following command line:

$ fpm --version 1.2.0 --python-bin /usr/bin/python3.9 --no-python-internal-pip -s python -t deb pika
Created package {:path=>"python-pika_1.2.0_all.deb"}

Off-topic:

--python-bin is required, because on Debian 11 systems, /usr/bin/python no longer exists. As fpm defaults to this path, it returns the following error message:

Your python environment is missing json support (either json or simplejson python module). I cannot continue without this. {:python=>"python", :error=>#<FPM::Util::ProcessFailed: /bin/bash failed (exit code 127). Full command was:["/bin/bash", "-c", "python -c 'try:\n  import json\nexcept ImportError:\n  import simplejson as json'"]>, :level=>:error}
Process failed: Python (python) is missing simplejson or json modules. {:level=>:error}

This error message is incorrect. The issue is that python does not exist, not that it's missing modules.

[jordansissel]

This error message is incorrect. The issue is that python does not exist, not that it's missing modules.

Agreed! FPM could do some more detailed checking to help identify the problem -- checking if it's missing python, or missing a module, etc

I wonder, separately, if fpm could try a few different known-default python executable names if "python" doesn't exist, as you noted, only "python3" or "python2" might exist as program names.

Thanks for all these extra details. I hope we can resolve this soon :)