rosdep install python3 dependencies when a package has manifest.xml and package.xml

[mikaelarguedas]

Issue description

It appears that wrongly rosdep installs python3 version of packages, removing needed python2 packages in the process.

How to reproduce

Trying to build ros_comm from source for Melodic:

To make the example clearer we'll install python-catkin-tools and build the workspace with that Python2 package

docker run -it --rm ubuntu:bionic
apt -qq update && apt -qq install -y lsb-release gnupg2 dirmngr
sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
apt update && apt-get install python-rosdep python-rosinstall-generator python-wstool python-rosinstall build-essential python-catkin-tools
rosdep init && rosdep update
mkdir ~/ros_catkin_ws && cd ~/ros_catkin_ws
rosinstall_generator ros_comm --rosdistro melodic --deps --tar > melodic-ros_comm.rosinstall
wstool init -j8 src melodic-ros_comm.rosinstall
rosdep install --from-paths src --ignore-src --rosdistro melodic -y
catkin config --install

This fails with

root@083919565a76:~/ros_catkin_ws# catkin config --install
bash: catkin: command not found

catkin tools has been uninstall by rosdep

executing command [apt-get install -y python3-catkin-pkg]
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  javascript-common libjs-jquery libjs-sphinxdoc libjs-underscore python-catkin-pkg-modules python-concurrent.futures python-osrf-pycommon python-pyparsing python-rosdistro python-rosdistro-modules
  python-rospkg python-rospkg-modules python-setuptools python-trollius sudo
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  python3-catkin-pkg-modules python3-dateutil python3-docutils python3-olefile python3-pil python3-pkg-resources python3-pygments python3-pyparsing python3-roman python3-six
Suggested packages:
  fonts-linuxlibertine | ttf-linux-libertine texlive-lang-french texlive-latex-base texlive-latex-recommended python-pil-doc python3-pil-dbg python3-setuptools ttf-bitstream-vera python-pyparsing-doc
The following packages will be REMOVED:
  python-catkin-pkg python-catkin-tools python-rosdep python-rosdep-modules python-rosinstall python-rosinstall-generator
The following NEW packages will be installed:
  python3-catkin-pkg python3-catkin-pkg-modules python3-dateutil python3-docutils python3-olefile python3-pil python3-pkg-resources python3-pygments python3-pyparsing python3-roman python3-six
0 upgraded, 11 newly installed, 6 to remove and 0 not upgraded.

Additional info

This issue does not happen if the manifest.xml from roslisp is not present

Possible reason: roslisp's manifest.xml depends on rospack, rosdep installs all dependencies of rospack, regardless of their ROS python version

[dirk-thomas]

This prevents the install-from-source instructions for Noetic to work.

@sloretz FYI

[sloretz]

Possible reason: roslisp's manifest.xml depends on rospack, rosdep installs all dependencies of rospack, regardless of their ROS python version

The package.xml also lists rospack. It seems odd that it coming from a manifest.xml vs a package.xml would change how it's dependencies are evaluated.

[dirk-thomas]

It seems odd that it coming from a manifest.xml vs a package.xml would change how it's dependencies are evaluated.

Not really, for the manifest.xml there are no conditional dependencies which only exist in format 3 in package.xml files.

[sloretz]

Not really, for the manifest.xml there are no conditional dependencies which only exist in format 3 in package.xml files.

But should that affect rospack which doesn't have a manifest.xml? rospack isn't a conditional dependency of roslisp, though it's package.xml has conditional dependencies. Regardless it seems like it must be something else. The ticket has apt-get install -y python3-catkin-pkg, but rospack doesn't depend on that (it depends on python3-catkin-pkg-modules). edit: nevermind, this is building the latest release of rospack which does not have this commit.

[dirk-thomas]

In my case I was using the latest code from the development branch.

[meyerj]

I think #353 (Prefer package.xml in hybrid packages?) is related.

Maybe it is time to prefer wet over dry in the meantime, which would probably solve most issues with features introduced in newer package.xml formats ;-)