Merge the latest master so that this can proceed. (#1)

* Add Suite3 option with Ubuntu Focal (ros-infrastructure#734)

Signed-off-by: Shane Loretz <[email protected]>

* [update] Allow to process single ROS distro, fix 723 (ros-infrastructure#738)

* [update] skip other distro if --rosdistro passed

Signed-off-by: Mikael Arguedas <[email protected]>

* ignore argument if specified distro doesnt exist

* address review comments

* update help message for rosdistro

Signed-off-by: Mikael Arguedas <[email protected]>

* Fix CI and reduce CI time (ros-infrastructure#739)

* test newer python
* use yaml.safe_load
* pin PyYAML version for Python 3.4
* pass user flag to pip when needed
* move slow jobs to the top to reduce CI time

Signed-off-by: Mikael Arguedas <[email protected]>

* Strip Alpine's patch version from OS codename (ros-infrastructure#716)

* Resolve Alpine os_version_type using OsDetect
* Reduced Alpine OS VERSION to Major.Minor

* improve support for null entries (ros-infrastructure#726)

* support null for entire OS and not only OS version

syntax like 'ubuntu: null' will now be raising ResolutionError with
relevant error message instead of InvalidData with obscure error message
will also cover syntax like:
"
ubuntu:
  '*': null
  bionic: [foobar]
"

Signed-off-by: Mikael Arguedas <[email protected]>

Co-authored-by: Scott K Logan <[email protected]>

* Use DNF installer on RHEL 8 and newer (ros-infrastructure#713)

RHEL/CentOS 8 uses DNF by default.

* Updates to YUM and DNF (ros-infrastructure#640)

Output YUM, DNF and RPM versions with `--all-versions` and fix the format guide's default PM for Fedora.

* tests: don't assume euid != 0 (ros-infrastructure#703)

If tests rely on the environment being a certain way, they should ensure
it meets its expectations. Update the tests to ensure they're testing
multiple euid conditions, and setting it explicitly where required.

Fix ros-infrastructure#702

Signed-off-by: Kyle Fazzari <[email protected]>

* openSUSE package query  and install enhancements (ros-infrastructure#729)

* Enable PIP installer for openSUSE

* openSUSE package detection with RPM capabilities

Packages sometimes get renamed and their old name is kept as an rpm capability (like an alias), so the additional flag `--whatprovides` is passed to the `rpm` query.

* Fix conditional dependencies when one package uses manifest.xml (ros-infrastructure#737)

* Fix conditional dependencies when one package uses manifest.xml

Signed-off-by: Shane Loretz <[email protected]>

* Return rosdeps as a list to avoid breaking the interface.

Co-authored-by: Steven! Ragnarök <[email protected]>

* Guard next(inter) (ros-infrastructure#701)

* guard next(inter)

  Fix ros-infrastructure#691

Signed-off-by: artivis <[email protected]>

* Handle StopIteration with slightly less line noise.

Co-authored-by: Steven! Ragnarök <[email protected]>

* [Windows] Add console script entry point (ros-infrastructure#656)

* Add console script entry point

Add console script entry point for platforms (e.g. Windows) not supporting shebang.

* remove scripts entry since we are using console_scripts.

* fix bad merge.

* fix SKIP_PYTHON_SCRIPTS case.

Co-authored-by: Lou Amadio <[email protected]>

* Depend on modules packages only to allow co-installability. (ros-infrastructure#750)

When the rosdep modules package was split in [ros-infrastructure#731] the module
dependencies weren't updated to only depend on modules packages
themselves which prevents rosdep modules for python2 and python3 from
actually being co-installable.

I haven't yet audited to make sure there's no cli usage of these tools
in the modules but it's Friday afternoon and I wanted to get this inked
before I walked away.

When I return to it I'll check for that before proceeding for reviews.

* 0.19.0

* also install buildtool_export_depends (ros-infrastructure#753)

not adding 'exec_depends' or 'build_export_depends' are both are included inside 'run_depends'

Signed-off-by: Mikael Arguedas <[email protected]>

* Drop shebang from a non-executable file (ros-infrastructure#755)

* add alias for Pop! OS (ros-infrastructure#757)

* add alias for Pop! OS

Signed-off-by: Mikael Arguedas <[email protected]>

Co-authored-by: Shane Loretz <[email protected]>
Co-authored-by: Mikael Arguedas <[email protected]>
Co-authored-by: Mark Hedley Jones <[email protected]>
Co-authored-by: Scott K Logan <[email protected]>
Co-authored-by: Kyle Fazzari <[email protected]>
Co-authored-by: Bjar Ne <[email protected]>
Co-authored-by: Jeremie Deray <[email protected]>
Co-authored-by: Sean Yen <[email protected]>
Co-authored-by: Lou Amadio <[email protected]>

.travis.yml

@@ -1,31 +1,38 @@
 language: python
 matrix:
   include:
+    - os: osx
+      language: generic
+      env: PYTHON_INSTALLER=pyenv TRAVIS_PYTHON_VERSION=2.7.14
+    - os: osx
+      language: generic
+      env: PYTHON_INSTALLER=pyenv TRAVIS_PYTHON_VERSION=3.6.4
+    - os: osx
+      language: generic
+      env:
+        - PYTHON_INSTALLER=brew
+        - PIP_USER_FLAG="--user"
     - os: linux
       python: 2.7
     - os: linux
-      python: 3.4
+      python: 3.4   # When support for 3.4 is removed unpin the PyYAML version below.
     - os: linux
       python: 3.5
     - os: linux
       python: 3.6
-    - os: osx
-      language: generic
-      env: PYTHON_INSTALLER=brew
-    - os: osx
-      language: generic
-      env: PYTHON_INSTALLER=pyenv TRAVIS_PYTHON_VERSION=2.7.14
-    - os: osx
-      language: generic
-      env: PYTHON_INSTALLER=pyenv TRAVIS_PYTHON_VERSION=3.6.4
+    - os: linux
+      python: 3.7
+    - os: linux
+      python: 3.8
 
 # command to install dependencies
 install:
   - source .travis/install.sh
   - python --version
-  - python -m pip install PyYAML argparse rospkg vcstools catkin_pkg python-dateutil rosdistro
-  - python -m pip install -e .
-  - python -m pip install nose coverage flake8 mock codecov
+  - if [ $TRAVIS_PYTHON_VERSION == "3.4" ]; then python -m pip install PyYAML==5.2; fi # Forcing PyYAML 5.2 while we retain Python 3.4 support PyYAML 5.3 and higher does not support Python 3.4
+  - python -m pip install $PIP_USER_FLAG PyYAML argparse rospkg vcstools catkin_pkg python-dateutil rosdistro
+  - python -m pip install $PIP_USER_FLAG -e .
+  - python -m pip install $PIP_USER_FLAG nose coverage flake8 mock codecov
 # command to run tests
 script:
   - python -m nose --with-coverage --cover-package=rosdep2 --with-xunit test

CHANGELOG.rst

@@ -1,3 +1,35 @@
+0.19.0 (2020-04-03)
+-------------------
+- Only release for Python3 into focal
+  - https://github.com/ros-infrastructure/rosdep/pull/734
+- Added --rosdistro argument to rosdep-update to scope update to one rosdistro
+  - https://github.com/ros-infrastructure/rosdep/pull/738
+  - Fixes https://github.com/ros-infrastructure/rosdep/pull/723
+- Fix CI for Python 3.4 and run slower CI jobs first
+  - https://github.com/ros-infrastructure/rosdep/pull/739
+- Strip Alpine's patch version from OS codename
+  - https://github.com/ros-infrastructure/rosdep/pull/716
+  - Fixes https://github.com/ros-infrastructure/rosdep/issues/715
+- Raise a clear and specific error message for null entries
+  - https://github.com/ros-infrastructure/rosdep/pull/726
+- Use DNF as the default installer on RHEL 8 and newer
+  - https://github.com/ros-infrastructure/rosdep/pull/713
+- Updates to YUM and DNF handling
+  - https://github.com/ros-infrastructure/rosdep/pull/640
+- Fix tests so they don't assume euid != 0
+  - https://github.com/ros-infrastructure/rosdep/pull/703
+- Update openSUSE package query function and enable pip installer
+  - https://github.com/ros-infrastructure/rosdep/pull/729
+- Fix conditional dependencies when one package uses manifest.xml
+  - https://github.com/ros-infrastructure/rosdep/pull/737
+- Handle StopIteration when querying in debian platform
+  - https://github.com/ros-infrastructure/rosdep/pull/701
+- Use entry points rather than console scripts to enable usage on Windows
+  - https://github.com/ros-infrastructure/rosdep/pull/656
+- Depend on modules packages only to allow modules packages to be co-installable.
+  - https://github.com/ros-infrastructure/rosdep/pull/750
+
+
 0.18.0 (2019-11-20)
 -------------------
 - split -modules into separate Debian package

doc/rosdep_yaml_format.rst

@@ -87,7 +87,8 @@ OS name identifiers and supported package managers
 
  * ``fedora``: Fedora Project
 
-   * ``yum`` (default)
+   * ``dnf`` (default)
+   * ``yum``
    * ``source``
 
  * ``freebsd``: FreeBSD

setup.py

@@ -6,17 +6,22 @@
     # same version as in:
     # - src/rosdep2/__init__.py
     # - stdeb.cfg
-    'version': '0.18.0',
+    'version': '0.19.0',
     'packages': ['rosdep2', 'rosdep2.ament_packages', 'rosdep2.platforms'],
     'package_dir': {'': 'src'},
-    'install_requires': ['catkin_pkg >= 0.4.0', 'rospkg >= 1.2.0', 'rosdistro >= 0.7.5', 'PyYAML >= 3.1'],
+    'install_requires': ['catkin_pkg >= 0.4.0', 'rospkg >= 1.2.7', 'rosdistro >= 0.7.5', 'PyYAML >= 3.1'],
     'test_suite': 'nose.collector',
     'test_requires': ['mock', 'nose >= 1.0'],
-    'scripts': ['scripts/rosdep', 'scripts/rosdep-source'],
     'author': 'Tully Foote, Ken Conley',
     'author_email': '[email protected]',
     'url': 'http://wiki.ros.org/rosdep',
     'keywords': ['ROS'],
+    'entry_points': {
+        'console_scripts': [
+            'rosdep = rosdep2.main:rosdep_main',
+            'rosdep-source = rosdep2.install:install_main'
+        ]
+    },
     'classifiers': [
         'Programming Language :: Python',
         'License :: OSI Approved :: BSD License'],
@@ -30,6 +35,6 @@
     kwargs['package_dir'] = {}
 if 'SKIP_PYTHON_SCRIPTS' in os.environ:
     kwargs['name'] += '_modules'
-    kwargs['scripts'] = {}
+    kwargs['entry_points'] = {}
 
 setup(**kwargs)

src/rosdep2/_version.py

@@ -1,4 +1,4 @@
 # same version as in:
 # - setup.py
 # - stdeb.cfg
-__version__ = '0.18.0'
+__version__ = '0.19.0'

scripts/rosdep-source → src/rosdep2/install.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python
 # Copyright (c) 2011, Willow Garage, Inc.
 # All rights reserved.
 #
@@ -39,7 +38,7 @@
 from rosdep2 import InstallFailed
 from rosdep2.platforms import source
 
-NAME='rosdep-source'
+NAME = 'rosdep-source'
 
 
 def install_main():
@@ -49,16 +48,12 @@ def install_main():
         parser.error("please specify one and only one rdmanifest url")
     if args[0] != 'install':
         parser.error("currently only support the 'install' command")
-    rdmanifest_url= args[1]
+    rdmanifest_url = args[1]
     try:
         if os.path.isfile(rdmanifest_url):
             source.install_from_file(rdmanifest_url)
         else:
             source.install_from_url(rdmanifest_url)
     except InstallFailed as e:
-        print("ERROR: installation failed:\n%s"%e, file=sys.stderr)
+        print("ERROR: installation failed:\n%s" % e, file=sys.stderr)
         sys.exit(1)
-
-
-if __name__ == '__main__':
-    install_main()

src/rosdep2/lookup.py

@@ -144,6 +144,9 @@ def get_rule_for_platform(self, os_name, os_version, installer_keys, default_ins
                                 return_key = installer_key
                                 break
 
+        # Check if the rule is null
+        if data is None:
+            raise ResolutionError(rosdep_key, self.data, os_name, os_version, '[%s] defined as "not available" for OS version [%s]' % (rosdep_key, os_version))
         if type(data) not in (dict, list, type('str')):
             raise InvalidData('rosdep OS definition for [%s:%s] must be a dictionary, string, or list: %s' % (self.rosdep_key, os_name, data), origin=self.origin)
 
@@ -171,7 +174,7 @@ def __str__(self):
 \trosdep key : %s
 \tOS name    : %s
 \tOS version : %s
-\tData: %s""" % (self.args[0], self.rosdep_key, self.os_name, self.os_version, pretty_data)
+\tData:\n%s""" % (self.args[0], self.rosdep_key, self.os_name, self.os_version, pretty_data.replace('\n', '\n\t\t'))
 
 
 class RosdepView(object):

src/rosdep2/main.py

@@ -269,7 +269,7 @@ def setup_environment_variables(ros_distro):
     """
     Set environment variables needed to find ROS packages and evaluate conditional dependencies.
 
-    :param rosdistro: The requested ROS distro passed on the CLI, or None
+    :param ros_distro: The requested ROS distro passed on the CLI, or None
     """
     if ros_distro is not None:
         if 'ROS_DISTRO' in os.environ and os.environ['ROS_DISTRO'] != ros_distro:
@@ -355,7 +355,9 @@ def _rosdep_main(args):
     parser.add_option('--rosdistro', dest='ros_distro', default=None,
                       help='Explicitly sets the ROS distro to use, overriding '
                            'the normal method of detecting the ROS distro '
-                           'using the ROS_DISTRO environment variable.')
+                           'using the ROS_DISTRO environment variable. '
+                           "When used with the 'update' verb, "
+                           'only the specified distro will be updated.')
     parser.add_option('--as-root', default=[], action='append',
                       metavar='INSTALLER_KEY:<bool>', help='Override '
                       'whether sudo is used for a specific installer, '
@@ -643,14 +645,18 @@ def update_error_handler(data_source, exc):
             pass
         update_sources_list(success_handler=update_success_handler,
                             error_handler=update_error_handler,
-                            skip_eol_distros=not options.include_eol_distros)
+                            skip_eol_distros=not options.include_eol_distros,
+                            ros_distro=options.ros_distro)
         print('updated cache in %s' % (sources_cache_dir))
     except InvalidData as e:
         print('ERROR: invalid sources list file:\n\t%s' % (e), file=sys.stderr)
         return 1
     except IOError as e:
         print('ERROR: error loading sources list:\n\t%s' % (e), file=sys.stderr)
         return 1
+    except ValueError as e:
+        print('ERROR: invalid argument value provided:\n\t%s' % (e), file=sys.stderr)
+        return 1
     if error_occured:
         print('ERROR: Not all sources were able to be updated.\n[[[')
         for e in error_occured:

src/rosdep2/platforms/alpine.py

@@ -48,6 +48,7 @@ def register_platforms(context):
     context.add_os_installer_key(OS_ALPINE, PIP_INSTALLER)
     context.add_os_installer_key(OS_ALPINE, SOURCE_INSTALLER)
     context.set_default_os_installer_key(OS_ALPINE, lambda self: APK_INSTALLER)
+    context.set_os_version_type(OS_ALPINE, lambda self: ".".join(self.get_version().split('.')[:2]))
 
 
 def apk_detect(pkgs, exec_fn=read_stdout):

src/rosdep2/platforms/debian.py

@@ -32,7 +32,17 @@
 import sys
 
 from rospkg.os_detect import OS_DEBIAN, OS_LINARO, OS_UBUNTU, OS_ELEMENTARY, OS_MX, OS_ZORIN, OsDetect, read_os_release
-
+from rospkg.os_detect import (
+    OS_DEBIAN,
+    OS_LINARO,
+    OS_UBUNTU,
+    OS_ELEMENTARY,
+    OS_MX,
+    OS_POP,
+    OS_ZORIN,
+    OsDetect,
+    read_os_release
+)
 from .pip import PIP_INSTALLER
 from .gem import GEM_INSTALLER
 from .source import SOURCE_INSTALLER
@@ -55,6 +65,7 @@ def register_platforms(context):
     register_elementary(context)
     register_linaro(context)
     register_mx(context)
+    register_pop(context)
     register_zorin(context)
 
 
@@ -99,6 +110,16 @@ def register_mx(context):
         context.set_os_override(OS_DEBIAN, version[version.find("(") + 1:version.find(")")])
 
 
+def register_pop(context):
+    # Pop! OS is an alias for Ubuntu. If Pop! is detected and it's
+    # not set as an override force ubuntu.
+    (os_name, os_version) = context.get_os_name_and_version()
+    if os_name == OS_POP and not context.os_override:
+        print('rosdep detected OS: [%s] aliasing it to: [%s]' %
+              (OS_POP, OS_UBUNTU), file=sys.stderr)
+        context.set_os_override(OS_UBUNTU, context.os_detect.get_codename())
+
+
 def register_zorin(context):
     # Zorin is an alias for Ubuntu. If Zorin is detected and it's
     # not set as an override force ubuntu.
@@ -151,20 +172,32 @@ def _read_apt_cache_showpkg(packages, exec_fn=None):
 
         header = 'Package: %s' % p
         # proceed to Package header
-        while next(lines) != header:
+        try:
+            while next(lines) != header:
+                pass
+        except StopIteration:
             pass
 
         # proceed to versions section
-        while next(lines) != 'Versions: ':
+        try:
+            while next(lines) != 'Versions: ':
+                pass
+        except StopIteration:
             pass
 
         # virtual packages don't have versions
-        if next(lines) != '':
-            yield p, False, None
-            continue
+        try:
+            if next(lines) != '':
+                yield p, False, None
+                continue
+        except StopIteration:
+            break
 
         # proceed to reserve provides section
-        while next(lines) != 'Reverse Provides: ':
+        try:
+            while next(lines) != 'Reverse Provides: ':
+                pass
+        except StopIteration:
             pass
 
         pr = [line.split(' ', 2)[0] for line in lines]

src/rosdep2/platforms/opensuse.py

@@ -31,6 +31,7 @@
 
 from rospkg.os_detect import OS_OPENSUSE
 
+from .pip import PIP_INSTALLER
 from .source import SOURCE_INSTALLER
 from ..installers import PackageManagerInstaller
 
@@ -44,14 +45,15 @@ def register_installers(context):
 
 def register_platforms(context):
     context.add_os_installer_key(OS_OPENSUSE, SOURCE_INSTALLER)
+    context.add_os_installer_key(OS_OPENSUSE, PIP_INSTALLER)
     context.add_os_installer_key(OS_OPENSUSE, ZYPPER_INSTALLER)
     context.set_default_os_installer_key(OS_OPENSUSE, lambda self: ZYPPER_INSTALLER)
 
 
 def rpm_detect(packages):
     installed = []
     for p in packages:
-        if not subprocess.call(['rpm', '-q', p]):
+        if not subprocess.call(['rpm', '-q', '--whatprovides', p]):
             installed.append(p)
     return installed
 

src/rosdep2/platforms/osx.py

@@ -303,7 +303,7 @@ def handle_options(options):
 
             # make sure options is a list of list of strings
             try:
-                valid = all([isinstance(x, _basestring) for l in options for x in l])
+                valid = all([isinstance(x, _basestring) for option in options for x in option])
             except Exception as e:
                 raise InvalidData("Invalid list of options '%s', error: %s" % (options, e))
             else: