Rclpy deployment as part of Python3.8 fails

[sdoerr20]

After the successful deployment of my first individual C++ ROS2 applications
on the TurtleBot3, I also target the integration of the python3.8 libraries
for Python-based applications, such as ros2 providing comfortable
debugging possibilities on the target.

During the development process the integration of Python3.8 on a ROMFS
overall worked, whereas ros2 is not capable of using the rclpy
client. The abbreviated trace below shows the produced error and the
structure of my ROMFS. The full trace is also attached.

vxworks_trace.txt

-> cmd
[vxWorks *]# set env LD_LIBRARY_PATH=/romfs/lib
[vxWorks *]# cd /romfs/lib
[vxWorks *]# ls
.
..

### All C++ libraries

libexample_interfaces__rosidl_typesupport_c.so
python3.8
[vxWorks *]# cd ../
[vxWorks *]# ls
.
..
ros2_utils
python_test
own_applications
lib
turtlebot3
[vxWorks *]# cd python_test/
[vxWorks *]# ls
.
..
hello_world.py
[vxWorks *]# cd ../ros2_utils/
[vxWorks *]# ls
.
..
ros2
python3
[vxWorks *]# rtp exec -u 0x20000 python3 /romfs/python_test/hello_world.py
Launching process 'python3' ...
Process 'python3' (process Id = 0xffff800000290010) launched.
Hello, world!
[vxWorks *]# rtp exec -u 0x20000 python3 ros2
Launching process 'python3' ...
Process 'python3' (process Id = 0xffff8000003dbd40) launched.
Failed to load entry point 'info': _rtld_exclusive_exit :: called

The C extension '/romfs/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
Failed to load entry point 'list': _rtld_exclusive_exit :: called

### Many error messages

The C extension '/romfs/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
usage: ros2 [-h] Call `ros2 <command> -h` for more detailed usage. ...

ros2 is an extensible command-line tool for ROS 2.

optional arguments:
  -h, --help            show this help message and exit

Commands:
  action     Various action related sub-commands
  component  Various component related sub-commands
  daemon     Various daemon related sub-commands
  lifecycle  Various lifecycle related sub-commands
  msg        Various msg related sub-commands
  multicast  Various multicast related sub-commands
  node       Various node related sub-commands
  param      Various param related sub-commands
  pkg        Various package related sub-commands
  run        Run a package specific executable
  service    Various service related sub-commands
  srv        Various srv related sub-commands
  topic      Various topic related sub-commands

  Call `ros2 <command> -h` for more detailed usage.

Referring to the ROS2 documentation from the
ROS2 index page,
the error is described in the following manner:

Import failing without library present on the system¶

Sometimes rclpy fails to be imported because the expected C extension
libraries are not found. If so, compare the libraries present in the
directory with the one mentioned in the error message.
Assuming a file with a similar name exists (same prefix like _rclpy. and same
suffix like .so but a different Python version / architecture) you are using
a different Python interpreter than which was used to build the C extension.
Be sure to use the same Python interpreter as the one used to build the binary.

For example, such a mismatch can crop up after an update of the OS.
Then, rebuilding the workspace may fix the issue.

But when I look at the C-extensions as part of rclpy libraries,
I could not recognize an architecture missmatch, as the C-extensions
are build for the RPI3 architecture.

# cd export/deploy/lib/python3.8/site-packages/rclpy/

# file _rclpy_action.cpython-38-vxworks.so
_rclpy_action.cpython-38-vxworks.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

# file _rclpy.cpython-38-vxworks.so
_rclpy.cpython-38-vxworks.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

# file _rclpy_logging.cpython-38-vxworks.so
_rclpy_logging.cpython-38-vxworks.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

# file _rclpy_pycapsule.cpython-38-vxworks.so
_rclpy_pycapsule.cpython-38-vxworks.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

# file _rclpy_signal_handler.cpython-38-vxworks.so
_rclpy_signal_handler.cpython-38-vxworks.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

According to this set of dependencies my questions are,
for which python interpreter rclpy is actually build, whether it can be
configured somehow and which adjustments may have to be made for the Vxwork
configuration.

[razr]

@sdoerr20, could you please run errno after this error occurs?
[vxWorks *]# errno

@mdragusu does ROS2 python works on your side with the RPI3?

[mdragusu]

@razr I currently don't have access to a RPI3, only a RPI4.

@sdoerr20 Here is how I have things working:
Because I have other packages depending on the /usr/lib path, I copy the files from export (from the build) in /usr on my sdcard.
you should also have the same kind of links:
[vxWorks *]# devs
drv refs name
1 [ 3] /
2 [ 3] /bin ==> /sd0a/sysroot/bin
2 [ 3] /boot ==> /sd0a/sysroot/boot
2 [ 3] /dev ==> /
2 [ 3] /etc ==> /sd0a/sysroot/etc
2 [ 3] /lib ==> /sd0a/sysroot/lib
2 [ 3] /sd0:1 ==> /sd0a
2 [ 3] /usr ==> /sd0a/sysroot/usr

then I run:
[vxWorks *]# C putenv "LD_LIBRARY_PATH=/sd0a/sysroot/usr/lib"

[vxWorks *]# rtp exec -u 0x80000 python3
Launching process 'python3' ...
Process 'python3' (process Id = 0xffff800001e55f90) launched.
Python 3.8.0 (default, Nov 11 2020, 08:22:16)
[Clang 9.0.1.1 (ssh://[email protected]:7999/llvm/clang.git 67f1ee998a3a1d on vxworks
Type "help", "copyright", "credits" or "license" for more information.

import os, sys
import select
import rclpy
from geometry_msgs.msg import Twist
from std_msgs.msg import String
from rclpy.node import Node
from rclpy.qos import QoSProfile
from sensor_msgs.msg import LaserScan
from rclpy.qos import qos_profile_default, qos_profile_sensor_data
from rclpy.qos import QoSReliabilityPolicy

[sdoerr20]

To answer from @razr, the output for errno is the following
after excuting the previous rtp command.

[vxWorks *]# rtp exec -u 0x20000 python3 ros2
Launching process 'python3' ...
Process 'python3' (process Id = 0xffff80000069f3c0) launched.
Failed to load entry point 'info': _rtld_exclusive_exit :: called

The C extension '/romfs/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions

### A lot of import errors

The C extension '/romfs/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
usage: ros2 [-h] Call `ros2 <command> -h` for more detailed usage. ...

ros2 is an extensible command-line tool for ROS 2.

optional arguments:
  -h, --help            show this help message and exit

Commands:
  action     Various action related sub-commands
  component  Various component related sub-commands
  daemon     Various daemon related sub-commands
  lifecycle  Various lifecycle related sub-commands
  msg        Various msg related sub-commands
  multicast  Various multicast related sub-commands
  node       Various node related sub-commands
  param      Various param related sub-commands
  pkg        Various package related sub-commands
  run        Run a package specific executable
  service    Various service related sub-commands
  srv        Various srv related sub-commands
  topic      Various topic related sub-commands

  Call `ros2 <command> -h` for more detailed usage.
[vxWorks *]# errno
errno = 0xb20003 : S_rtpLib_ACCESS_DENIED.

Referring to the Help pages from the WindRiver Workbench, this error
represents an RTP state where the process can not be accessed for meta
information. At the point of writing I could not draw a conclusion from this.

In order to proceed, I tested my python installation at /romfs/lib
following the suggestions of @mdragusu. There only occur error traces in
case of accessing parts within rclpy. The complete trace is given in the
following.

-> cmd
[vxWorks *]# set env LD_LIBRARY_PATH=/romfs/lib
[vxWorks *]# cd /romfs/ros2_utils
[vxWorks *]# rtp exec -u 0x20000 python3
Launching process 'python3' ...
Process 'python3' (process Id = 0xffff8000003fe010) launched.
Python 3.8.0 (default, Feb 17 2021, 12:13:31)
[Clang 9.0.1.1 (ssh://[email protected]:7999/llvm/clang.git 67f1ee998a3a1d on vxworks
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> import select
>>> import rclpy
>>> from geometry_msgs.msg import Twist
>>> from std_msgs.msg import String
>>> from rclpy.node import Node
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/romfs/lib/python3.8/site-packages/rclpy/node.py", line 39, in <module>
    from rclpy.client import Client
  File "/romfs/lib/python3.8/site-packages/rclpy/client.py", line 22, in <module>
    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
  File "/romfs/lib/python3.8/site-packages/rclpy/impl/implementation_singleton.py", line 31, in <module>
    rclpy_implementation = _import('._rclpy')
  File "/romfs/lib/python3.8/site-packages/rclpy/impl/__init__.py", line 22, in _import
    return importlib.import_module(name, package='rclpy')
  File "/romfs/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: _rtld_exclusive_exit :: called

The C extension '/romfs/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
>>> from rclpy.qos import QoSProfile
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/romfs/lib/python3.8/site-packages/rclpy/qos.py", line 20, in <module>
    from rclpy.duration import Duration
  File "/romfs/lib/python3.8/site-packages/rclpy/duration.py", line 16, in <module>
    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
  File "/romfs/lib/python3.8/site-packages/rclpy/impl/implementation_singleton.py", line 31, in <module>
    rclpy_implementation = _import('._rclpy')
  File "/romfs/lib/python3.8/site-packages/rclpy/impl/__init__.py", line 22, in _import
    return importlib.import_module(name, package='rclpy')
  File "/romfs/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: _rtld_exclusive_exit :: called

The C extension '/romfs/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
>>> from sensor_msgs.msg import LaserScan
>>> from rclpy.qos import qos_profile_default, qos_profile_sensor_data
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/romfs/lib/python3.8/site-packages/rclpy/qos.py", line 20, in <module>
    from rclpy.duration import Duration
  File "/romfs/lib/python3.8/site-packages/rclpy/duration.py", line 16, in <module>
    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
  File "/romfs/lib/python3.8/site-packages/rclpy/impl/implementation_singleton.py", line 31, in <module>
    rclpy_implementation = _import('._rclpy')
  File "/romfs/lib/python3.8/site-packages/rclpy/impl/__init__.py", line 22, in _import
    return importlib.import_module(name, package='rclpy')
  File "/romfs/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: _rtld_exclusive_exit :: called

The C extension '/romfs/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
>>> from rclpy.qos import QoSReliabilityPolicy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/romfs/lib/python3.8/site-packages/rclpy/qos.py", line 20, in <module>
    from rclpy.duration import Duration
  File "/romfs/lib/python3.8/site-packages/rclpy/duration.py", line 16, in <module>
    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
  File "/romfs/lib/python3.8/site-packages/rclpy/impl/implementation_singleton.py", line 31, in <module>
    rclpy_implementation = _import('._rclpy')
  File "/romfs/lib/python3.8/site-packages/rclpy/impl/__init__.py", line 22, in _import
    return importlib.import_module(name, package='rclpy')
  File "/romfs/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: _rtld_exclusive_exit :: called

The C extension '/romfs/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
>>> quit()

In my development setup, I usually use a network boot with no sysroot
folder present. For Debugging purposes I created an SD-Card with the
following contents:

├── bcm2710-rpi-3-b.dtb
├── bcm2710-rpi-3-b-plus.dtb
├── bcm2710-rpi-cm3.dtb
├── bcm2837-rpi-3-b.dtb
├── bootcode.bin
├── boot.scr
├── broadcom/
├── cmdline.txt
├── config.txt
├── fixup_cd.dat
├── fixup.dat
├── fixup_db.dat
├── fixup_x.dat
├── initrd.img
├── meta-data
├── network-config
├── overlays/
├── rpi-3b-plus.dtb
├── start_cd.elf
├── start_db.elf
├── start.elf
├── start_x.elf
├── sysroot
│   └── usr
│       ├── bin
│       │   ├── __pycache__
│       │   ├── pydoc3
│       │   ├── pydoc3.8
│       │   ├── python3
│       │   ├── python3.8
│       │   ├── python3.8-config
│       │   ├── python3-config
│       │   ├── ros2
│       │   ├── _ros2_daemon
│       └── lib
│           ├── ### C++ libraries
│           └── python3.8/
├── u-boot.bin
├── user-data
└── uVxWorks

In order to deploy the libraries and executables separately, I manually
created a sysroot folder on the SD Card.
Now, the U-BOOT boots the uVxworks image from the SD Card instead of
performing a tftp download. With the completion of the boot sequence
I am able to observe the same device structure, but /usr seems not
to reference any SD card content. I tried to run python according to
this commands.

-> cmd
[vxWorks *]# devs
drv refs name
  1 [ 3] /
  2 [ 3] /bin  ==>  /sd0a/sysroot/bin
  2 [ 3] /boot  ==>  /sd0a/sysroot/boot
  2 [ 3] /dev  ==>  /
  2 [ 3] /etc  ==>  /sd0a/sysroot/etc
 10 [ 3] /fifos
 13 [ 3] /input/event
  2 [ 3] /lib  ==>  /sd0a/sysroot/lib
  0 [ 3] /null
 12 [ 3] /random
  9 [ 3] /romfs
  6 [ 3] /tmp
  3 [ 3] /ttyS0
  2 [ 3] /tyCo/0  ==>  /ttyS0
 12 [ 3] /urandom
 16 [ 3] /usb2ttyS/0
 16 [ 3] /usb2ttyS/1
  2 [ 3] /usr  ==>  /sd0a/sysroot/usr
 11 [ 3] /zero
 15 [ 3] host:
[vxWorks *]# C putenv "LD_LIBRARY_PATH=/sd0a/sysroot/usr/lib"
value = 0 = 0x0
[vxWorks *]# cd /romfs/ros2_utils/
[vxWorks *]# ls
. 
.. 
ros2 
python3 
[vxWorks *]# rtp exec -u 0x80000 python3
Launching process 'python3' ...
Process 'python3' (process Id = 0xffff800000290290) launched.
Shared object "libpython3.8.so.1.0" not found
[vxWorks *]# cd /usr
cd: error = 0x2.

[razr]

@sdoerr20 I'm wondering whether you have a read-only filesystem as your /romfs?

and it looks to me that your VxWorks image does not recognize an SDCard, I do not see it in the devs
this line is missing
2 [ 3] /sd0:1 ==> /sd0a

[sdoerr20]

@razr Yes, I do have a ROMFS mounted at "/romfs". After loading the Workbench
tools, setting the Workspace and navigating to the VIP project, I check
the config for the mounting point in the following manner:

/path/to/Robot_vip# wrtool prj vip parameter value ROMFS_DEV_NAME
ROMFS_DEV_NAME = "/romfs"

In order to specify the content of the ROMFS, a ROMFS project is linked to the
VIP project as a subproject:

wrtool prj subproject add Robot_romfs Robot_vip

According to the missing /sd0a device I started to collect the drivers from my
Image in the first instance and yes, the device is missing, because no
appropriate driver is available.

As a second stage, I tried to use the uVxWorks images from the RPI3 SDKs in
version 1.4 and 1.5, but I was unable to find a /sd0a device. Based on this
circumstance, I included further drivers for the SD card as mentioned in
the following:

wrtool prj vip component add DRV_MMCSTORAGE_CARD
wrtool prj vip component add DRV_PCI_SDHC_CTRL
wrtool prj vip component add INCLUDE_SD_BUS
wrtool prj vip component add DRV_SDSTORAGE_CARD

In the Kernel Configuration menu, these components are all located at
Storage -> Block Device Management -> SDMMC. Unfortunately, the addition of
these drivers do not led to the abstraction of a /sd0a device.

Is there an RPI3 specific driver for SD/MMC devices available (comparable to
OpenCR/ LiDAR)? Looking at the contents of the RPI3 BSP I could not identify an
RPi3 specific driver for SD/MMC.

[razr]

@sdoerr20 I apologize for the troubles you still have. I do not see here https://labs.windriver.com/downloads/wrsdk-vxworks7-docs/650/README-raspberrypi3b.html an explicit description of how to use the SDCard.
As an option have you tried to use a USB stick instead?
I guess (not sure) since you are using a read-only filesystem, python fails to create pyc files, that's why you could get this error message.

@danmilea do we have SDCard support on the RPI3 in the VxWorks image we deliver with the SDK?

[danmilea]

The VxWorks kernels provided with the RPi3 VxWorks SDKs do not include SD card support.

[razr]

@danmilea thanks for the clarification, do we support USB sticks?

[danmilea]

FAT32-formatted USB drives should work. Please check if the list of files to copy includes symbolic links. If this is the case, use -L (always follow symbolic links in SOURCE) when using cp.

[razr]

@sdoerr20 it looks like that we have narrowed down a solution. Could you please try to use a USB stick instead? I'll update my README accordingly to mention what kind of storage shall be used.

[sdoerr20]

@razr What I can confirm so far, is the fact that using a ROMFS is definitely
not usable to create files. As I tried so analogously to the following commands,
it produces an error. However this error may not fit to RCLPY error.

[vxWorks *]# pwd
/romfs/lib
[vxWorks *]# file copy librclpy_common.so ../
copying file librclpy_common.so -> ..//librclpy_common.so
Can't write to "..//librclpy_common.so", errno 0x      1e
[vxWorks *]# file copy librclpy_common.so /romfs
copying file librclpy_common.so -> /romfs/librclpy_common.so
Can't write to "/romfs/librclpy_common.so", errno 0x      1e

Further I created a VIP image with DOSFS libraries and without an integrated
ROMFS project. Therefore I put all ROMFS contents on a FAT32 USB-Stick (GEN2)
which is abstracted as /bd0a device. Setting the LD_LIBRARY_PATH
appropriately and starting with the hello_world.py programm and the
import sequence from mdragusu, I was able to reproduce the errors from
mentioned above which is shown in the following trace.

-> cmd
[vxWorks *]# devs
drv refs name
  1 [ 3] /
  2 [ 3] /bd0:1  ==>  /bd0a
  6 [ 3] /bd0a
  2 [ 3] /bin  ==>  /sd0a/sysroot/bin
  2 [ 3] /boot  ==>  /sd0a/sysroot/boot
 14 [ 3] /console
  2 [ 3] /dev  ==>  /
  2 [ 3] /etc  ==>  /sd0a/sysroot/etc
 11 [ 3] /fifos
 15 [ 3] /input/event
  2 [ 3] /lib  ==>  /sd0a/sysroot/lib
  0 [ 3] /null
  7 [ 3] /ram0
 13 [ 3] /random
  9 [ 3] /stdio_pty_0xffff8000002b15a0.M
  8 [ 3] /stdio_pty_0xffff8000002b15a0.S
  7 [ 3] /tmp
  3 [ 3] /ttyS0
  2 [ 3] /tyCo/0  ==>  /ttyS0
 13 [ 3] /urandom
 18 [ 3] /usb2ttyS/0
 18 [ 3] /usb2ttyS/1
  2 [ 3] /usr  ==>  /sd0a/sysroot/usr
  4 [ 3] /vxbus
 12 [ 3] /zero
 17 [ 3] host:
[vxWorks *]# cd /bd0a
[vxWorks *]# ls
lib 
own_applications 
python_test 
ros2_utils 
turtlebot3 
[vxWorks *]# set env LD_LIBRARY_PATH=/bd0a/lib
[vxWorks *]# cd python_test/
[vxWorks *]# ls
. 
.. 
hello_world.py 
[vxWorks *]# rtp exec -u 0x20000 /bd0a/ros2_utils/python3 hello_world.py
Launching process '/bd0a/ros2_utils/python3' ...
Process '/bd0a/ros2_utils/python3' (process Id = 0xffff80000043d010) launched.
Hello, world!
[vxWorks *]# rtp exec -u 0x20000 /bd0a/ros2_utils/python3
Launching process '/bd0a/ros2_utils/python3' ...
Process '/bd0a/ros2_utils/python3' (process Id = 0xffff8000006b8470) launched.
Python 3.8.0 (default, Feb 17 2021, 12:13:31) 
[Clang 9.0.1.1 (ssh://[email protected]:7999/llvm/clang.git 67f1ee998a3a1d on vxworks
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> import select
>>> import rclpy
>>> from geometry_msgs.msg import Twist
>>> from std_msgs.msg import String
>>> from rclpy.node import Node
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/bd0a/lib/python3.8/site-packages/rclpy/node.py", line 39, in <module>
    from rclpy.client import Client
  File "/bd0a/lib/python3.8/site-packages/rclpy/client.py", line 22, in <module>
    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
  File "/bd0a/lib/python3.8/site-packages/rclpy/impl/implementation_singleton.py", line 31, in <module>
    rclpy_implementation = _import('._rclpy')
  File "/bd0a/lib/python3.8/site-packages/rclpy/impl/__init__.py", line 22, in _import
    return importlib.import_module(name, package='rclpy')
  File "/bd0a/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: _rtld_exclusive_exit :: called 

The C extension '/bd0a/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
>>> from rclpy.qos import QoSProfile
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/bd0a/lib/python3.8/site-packages/rclpy/qos.py", line 20, in <module>
    from rclpy.duration import Duration
  File "/bd0a/lib/python3.8/site-packages/rclpy/duration.py", line 16, in <module>
    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
  File "/bd0a/lib/python3.8/site-packages/rclpy/impl/implementation_singleton.py", line 31, in <module>
    rclpy_implementation = _import('._rclpy')
  File "/bd0a/lib/python3.8/site-packages/rclpy/impl/__init__.py", line 22, in _import
    return importlib.import_module(name, package='rclpy')
  File "/bd0a/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: _rtld_exclusive_exit :: called 

The C extension '/bd0a/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
>>> from sensor_msgs.msg import LaserScan
>>> from rclpy.qos import qos_profile_default, qos_profile_sensor_data
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/bd0a/lib/python3.8/site-packages/rclpy/qos.py", line 20, in <module>
    from rclpy.duration import Duration
  File "/bd0a/lib/python3.8/site-packages/rclpy/duration.py", line 16, in <module>
    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
  File "/bd0a/lib/python3.8/site-packages/rclpy/impl/implementation_singleton.py", line 31, in <module>
    rclpy_implementation = _import('._rclpy')
  File "/bd0a/lib/python3.8/site-packages/rclpy/impl/__init__.py", line 22, in _import
    return importlib.import_module(name, package='rclpy')
  File "/bd0a/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: _rtld_exclusive_exit :: called 

The C extension '/bd0a/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
>>> from rclpy.qos import QoSReliabilityPolicy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/bd0a/lib/python3.8/site-packages/rclpy/qos.py", line 20, in <module>
    from rclpy.duration import Duration
  File "/bd0a/lib/python3.8/site-packages/rclpy/duration.py", line 16, in <module>
    from rclpy.impl.implementation_singleton import rclpy_implementation as _rclpy
  File "/bd0a/lib/python3.8/site-packages/rclpy/impl/implementation_singleton.py", line 31, in <module>
    rclpy_implementation = _import('._rclpy')
  File "/bd0a/lib/python3.8/site-packages/rclpy/impl/__init__.py", line 22, in _import
    return importlib.import_module(name, package='rclpy')
  File "/bd0a/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: _rtld_exclusive_exit :: called 

The C extension '/bd0a/lib/python3.8/site-packages/rclpy/_rclpy.cpython-38-vxworks.so' failed to be imported while being present on the system. Please refer to 'https://index.ros.org/doc/ros2/Troubleshooting/Installation-Troubleshooting/#import-failing-even-with-library-present-on-the-system' for possible solutions
>>> quit()

For completeness I also run rtp exec -u 0x20000 python3 ros2 which took
quite a while to execute. Here I was also able to reproduce the same error
messages from above multiple times.

Beside the lack of function, I would be still interested in an experimental
driver for an SD card (~25 MB/s), as the usage of an USB (~10 MB/s) slows
down the overall performance of the robot. Alternatively I would also have
a look at the SD card driver for the RPI4 model.

[razr]

@sdoerr20 I don't think we'll provide support for SDcard on RPI3, we do have it on RPI4

[razr]

@sdoerr20, do you copy files from the export directory to your target with cp -L ? as @mdragusu has mentioned VxWorks does not support symbolic links. It could be the reason why you have this error.

[sdoerr20]

@razr Retrying the build of the ROS2 libraries and copying the library artifacts
from export/deploy/lib to a FAT32 USB stick with cp -LR finally worked.

Further, I noticed the fact that actually only the
export/root/lib directory within the ROS2 build contains symbolic links
as listed in the following:

# find export/root -type l
./lib/libfastcdr.so
./lib/libfastrtps.so
./lib/libfastcdr.so.1
./lib/libtinyxml2.so
./lib/libfastrtps.so.1
./lib/libconsole_bridge.so
./lib/libtinyxml2.so.6

#find export/deploy -type l
<empty>

Comparing the filesystem attributes of the contents from root/ and deploy/
with the file command, the following differences can be outlined:

file	export/root/lib	export/deploy/lib
libfastcdr.so	symbolic link to libfastcdr.so.1	ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped
libfastrtps.so	symbolic link to libfastrtps.so.1	ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped
libfastcdr.so.1	symbolic link to libfastcdr.so.1.0.13	ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped
libtinyxml2.so	symbolic link to libtinyxml2.so.6	ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped
libfastrtps.so.1	symbolic link to libfastrtps.so.1.8.2	ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped
libconsole_bridge.so	symbolic link to libconsole_bridge.so.0.4	ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped
libtinyxml2.so.6	symbolic link to libtinyxml2.so.6.0.0	ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, with debug_info, not stripped

Comparing this behavior with the toolchain from the SR0640-dashing branch
(only the folder root exists),
the deploy/ folder seems to already contain all files needed with resolved
links. Therefore cp -LR actually does not resolve any symbolic link.

The question that remains for me, which impact the -L option has so that it fixes the issue.