User dependent gazebo installation

[stefanbuettner]

Hey there,

I want to provide Gazebo via the conan package manager to some colleagues. Building all the components worked like a charm using conan's cmake build helpers and git tool. Also gazebo starts without problems for my user and I can compile plugins against the Gazebo C++ API using cmake.

However, if I upload the built package to our Artifactory server, and another user installs the pre-built binaries, Gazebo doesn't start and find_package cannot find Gazebo anymore.

We were able to start Gazebo under the other user by providing the environment variable GAZEBO_RESOURCE_PATH (see PX4/PX4-Avoidance#177).

But in general it seems that user-specific paths are compiled into the generated packages, both in cmake config files and so-files.
Is this the case? And if so, can Gazebo and its dependencies be compiled in a user-independent manner?

I could not find cmake options in any of the packages. I also didn't find any pointers in the documentation or READMEs.

Cheers,
Stefan

[traversaro]

But in general it seems that user-specific paths are compiled into the generated packages, both in cmake config files and so-files.
Is this the case? And if so, can Gazebo and its dependencies be compiled in a user-independent manner?

At least for the CMake aspect, this problem is also tracked in #2755, even if I do not think that anyone is actively working on it.

For the paths that contained the hardcoded installation prefix in them, in general the suggestion is always to source the <prefix>/share/gazebo/setup.sh script before running Gazebo, see http://gazebosim.org/tutorials?tut=components&cat=get_started#EnvironmentVariables . The main problem is that the <prefix>/share/gazebo/setup.sh script itself at the moment is not relocatable. However, fixing that is relatively simple, it is necessary to refactor it to compute the installation prefix relatively to the location of the setup.sh path. I have a similar script working in this way in https://github.com/robotology/robotology-superbuild-dependencies-vcpkg/blob/master/scripts/setup-gazebo.sh .

[stefanbuettner]

Cool, thanks. I'll have a look. Would such a change be merge request worthy?

[traversaro]

Would such a change be merge request worthy?

In my opinion definitely, and I would be happy to provide feedback as I would be happy to not have to mantain my own scripts for that. However let's see what the Gazebo maintainers think about this.

Unrelated comment: do you have any plan to open source your Gazebo conan recipes?

[stefanbuettner]

No plans so far. They're also rather rudimentary at this point. Just to compile stuff. No options. But could serve as a starting point for the community.

[traversaro]

I had forgot about this issue, but as it is related to conan I guess @joxoby could also be interested in this.

[joxoby]

However, if I upload the built package to our Artifactory server, and another user installs the pre-built binaries, Gazebo doesn't start and find_package cannot find Gazebo anymore.

This sounds confusing to me. Conan itself will generate the required Find<package>.cmake files that will point to the local Conan packages. This step will normally happen when you call conan_cmake_run inside your CMake scripts that want to include Gazebo, so it shouldn't really matter whether the Conan package was built locally or downloaded from Artifactory.

[stefanbuettner]

This is only true, if you use the cmake_find_package generator which generates the FindXXX.config scripts. But I don't see a reason to use them when the package itself provides CMake config scripts.
So for all packages which provide CMake config scripts we use the cmake_paths generator which only points the CMAKE_PREFIX_PATH to the appropriate locations.

[joxoby]

Normally, you wouldn't package the *.cmake scripts: https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#kb-h016-cmake-modules-config-files

More info: https://blog.conan.io/2018/06/11/Transparent-CMake-Integration.html

[traversaro]

Normally, you wouldn't package the *.cmake scripts: https://github.com/conan-io/conan-center-index/blob/master/docs/error_knowledge_base.md#kb-h016-cmake-modules-config-files

More info: https://blog.conan.io/2018/06/11/Transparent-CMake-Integration.html

I am probably missing something, but it seems that cmake_find_package generator, can't work for all libraries, i.e. as the say in the blog post:

If you are consuming packages and you have high restrictions to change your CMakeLists.txt, probably the cmake_paths is the best choice.

So if I have an open source codebase that supports consuming dependencies via multiple package managers (apt/conda/homebrew/vcpkg), I guess I can't use the cmake_find_package generator.

Furthermore, how can this work with complex libraries such as Qt that also need to install CMake macros?

[joxoby]

So if I have an open source codebase that supports consuming dependencies via multiple package managers (apt/conda/homebrew/vcpkg), I guess I can't use the cmake_find_package generator.

I might need to see a concrete example.

Furthermore, how can this work with complex libraries such as Qt that also need to install CMake macros?

Great question. To be clear, the hook restriction is trying to avoid that you don't package the pacakge's own Find<package>.cmake scripts. If the package includes cmake scripts to find other packages, like ignition-cmake (and you want to upload it to conan-center-index), you have to add an exception to that hook check. Example. Anyways, this is somewhat tangential to the main issue.

@stefanbuettner, if you'd like, you can try making a PR to upload your recipes to conan-center-index. Besides making them available to other people, the main Conan reviewers will provide you with great feedback on best practices.

[traversaro]

So if I have an open source codebase that supports consuming dependencies via multiple package managers (apt/conda/homebrew/vcpkg), I guess I can't use the cmake_find_package generator.

I might need to see a concrete example.

So if I have an open source codebase that supports consuming dependencies via multiple package managers (apt/conda/homebrew/vcpkg), I guess I can't use the cmake_find_package generator.

I might need to see a concrete example.

Quoting the blogpost, if I have a project that uses CURL and I want my project to work with all package managers, I will use it in the command find_package(CURL), and so I need to use the cmake_paths generator. To use the cmake_find_package generator, you would need to use find_package(libcurl), and either break compatibility with all others package manager, or add package manager-specific logic, that is something that is not desirable. But I never used Conan, so I could get something wrong.

[joxoby]

Quoting the blogpost, if I have a project that uses CURL and I want my project to work with all package managers, I will use it in the command find_package(CURL), and so I need to use the cmake_paths generator. To use the cmake_find_package generator, you would need to use find_package(libcurl), and either break compatibility with all others package manager, or add package manager-specific logic, that is something that is not desirable. But I never used Conan, so I could get something wrong.

The blogpost can be misleading in that specific example. For any Conan package, you can set the name that the generator will use to create the Find<name>.cmake script. https://github.com/conan-io/conan-center-index/blob/7d73c4e97052080a712028c082d6da4a78dd7179/recipes/libcurl/all/conanfile.py#L536

[traversaro]

Quoting the blogpost, if I have a project that uses CURL and I want my project to work with all package managers, I will use it in the command find_package(CURL), and so I need to use the cmake_paths generator. To use the cmake_find_package generator, you would need to use find_package(libcurl), and either break compatibility with all others package manager, or add package manager-specific logic, that is something that is not desirable. But I never used Conan, so I could get something wrong.

The blogpost can be misleading in that specific example. For any Conan package, you can set the name that the generator will use to create the Find<name>.cmake script. https://github.com/conan-io/conan-center-index/blob/7d73c4e97052080a712028c082d6da4a78dd7179/recipes/libcurl/all/conanfile.py#L536

Cool, that is much more clear, I just have two more doubts:

• Gazebo's CMake config files do not use imported targets, but just CMake variables, can you generate also those with the cmake_find_package generator? This is similar to the problem of packages that install CMake macros (that are not CMake's Find**.cmake scripts, but rather CMake functions such as qt5_generate_moc.
• I wonder why conan generates Find<package>.cmake files, that in CMake are meant to inspect the system to find a dependency, instead of generating <package>-config.cmake files, that are the ones that are generated by packages to make sure that that they are found by downstream projects, but I suspect the reason is that if you call find_package without specific options Find<package>.cmake have the precedence over <package>-config.cmake .

In any case, a personal note: even if the approach of trying to avoid build system-specific formats is quite interesting, replicating all the information of the imported target in manually maintained (and package manager-specif) python scripts (see for example https://github.com/conan-io/conan-center-index/pull/3215/files#diff-68a42dbaa991102b8ab0fa45fa4ed20dae5b48e127822865644e071974f6be07R126) may be error-prone in some cases. A possible solution for this would be a real cross-build-system format to express C++ library information, instead of relying on CMake config files and .pc files, and on that there is an interesting discussion between CMake and meson developers in https://gitlab.kitware.com/cmake/cmake/-/issues/20106 and in the issues of https://github.com/mwoehlke/cps .

[traversaro]

A possible solution for this would be a real cross-build-system format to express C++ library information, instead of relying on CMake config files and .pc files, and on that there is an interesting discussion between CMake and meson developers in https://gitlab.kitware.com/cmake/cmake/-/issues/20106 and in the issues of https://github.com/mwoehlke/cps .

Indeed, I found a related conan discussion in conan-io/conan#4878 .

[traversaro]

#2879 makes CMake config packages relocatable, but some other files in Gazebo (in particular the setup.sh script) is not relocatable, not sure if it make sense to open a new issue for tracking that or it make sense to re-open this one.

[traversaro]

fyi @mboisson

[scpeters]

#2879 makes CMake config packages relocatable, but some other files in Gazebo (in particular the setup.sh script) is not relocatable, not sure if it make sense to open a new issue for tracking that or it make sense to re-open this one.

I've opened #3056 to track the relocatability of setup.sh