Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-35804: [CI][Packaging][Conan] Synchronize upstream conan #39729

Merged
merged 11 commits into from
Apr 15, 2024

Conversation

wgtmac
Copy link
Member

@wgtmac wgtmac commented Jan 22, 2024

What changes are included in this PR?

Merged upstream conan using ci/conan/merge_upstream.sh ../conan-center-index/

Are these changes tested?

Will be tested on CI.

Are there any user-facing changes?

No

@github-actions github-actions bot added the awaiting review Awaiting review label Jan 22, 2024
ci/conan/all/conanfile.py Outdated Show resolved Hide resolved
@github-actions github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jan 22, 2024
@kou
Copy link
Member

kou commented Jan 22, 2024

@github-actions crossbow submit -g conan

This comment was marked as outdated.

@wgtmac

This comment was marked as outdated.

This comment was marked as outdated.

@kou
Copy link
Member

kou commented Jan 22, 2024

Could you make your visibility of the "Members" group in apache organization to "Public"?
https://github.com/orgs/apache/people?query=wgtmac

@wgtmac
Copy link
Member Author

wgtmac commented Jan 22, 2024

@github-actions crossbow submit -g conan

@wgtmac
Copy link
Member Author

wgtmac commented Jan 22, 2024

Could you make your visibility of the "Members" group in apache organization to "Public"? https://github.com/orgs/apache/people?query=wgtmac

Thank you for the suggestion! @kou

@apache apache deleted a comment from github-actions bot Jan 22, 2024
@apache apache deleted a comment from github-actions bot Jan 22, 2024
@apache apache deleted a comment from github-actions bot Jan 22, 2024
@wgtmac

This comment was marked as outdated.

This comment was marked as outdated.

@wgtmac

This comment was marked as outdated.

This comment was marked as outdated.

@wgtmac
Copy link
Member Author

wgtmac commented Jan 22, 2024

conan-maximum failed to find boost:

-- [ /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:1905 ] Boost_LIB_PREFIX = ""
-- [ /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e[862](https://github.com/ursacomputing/crossbow/actions/runs/7613109791/job/20732245591#step:6:863)fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:1906 ] Boost_NAMESPACE = "boost"
-- [ /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:953 ] _boost_COMPILER = "-gcc" (guessed)
-- [ /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:1942 ] _boost_MULTITHREADED = "-mt"
-- [ /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:2020 ] _boost_ARCHITECTURE_TAG = "" (detected)
-- [ /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:2024 ] _boost_RELEASE_ABI_TAG = "-"
-- [ /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:2025 ] _boost_DEBUG_ABI_TAG = "-d"
-- [ /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:2085 ] _boost_LIBRARY_SEARCH_DIRS_RELEASE = "Boost_INCLUDE_DIR-NOTFOUND/lib;Boost_INCLUDE_DIR-NOTFOUND/../lib;Boost_INCLUDE_DIR-NOTFOUND/stage/lib;PATHS;C:/boost/lib;C:/boost;/sw/local/lib"
CMake Error at /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Boost (missing: Boost_INCLUDE_DIR) (Required is at least
  version "1.58")
Call Stack (most recent call first):
  /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:2392 (find_package_handle_standard_args)
  cmake_modules/ThirdpartyToolchain.cmake:307 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:1274 (resolve_dependency)
  CMakeLists.txt:542 (include)


-- [ /root/.conan/data/cmake/3.28.1/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/share/cmake-3.28/Modules/FindBoost.cmake:2086 ] _boost_LIBRARY_SEARCH_DIRS_DEBUG = "Boost_INCLUDE_DIR-NOTFOUND/lib;Boost_INCLUDE_DIR-NOTFOUND/../lib;Boost_INCLUDE_DIR-NOTFOUND/stage/lib;PATHS;C:/boost/lib;C:/boost;/sw/local/lib"
-- Configuring incomplete, errors occurred!
arrow/16.0.0: 
WARN: Remotes registry file missing, creating default one in /root/.conan/remotes.json
snappy/1.1.9: WARN: Using the new toolchains and generators without specifying a build profile (e.g: -pr:b=default) is discouraged and might cause failures and unexpected behavior
arrow/16.0.0: WARN: Using the new toolchains and generators without specifying a build profile (e.g: -pr:b=default) is discouraged and might cause failures and unexpected behavior
arrow/16.0.0: WARN: Using the new toolchains and generators without specifying a build profile (e.g: -pr:b=default) is discouraged and might cause failures and unexpected behavior
arrow/16.0.0: ERROR: Package '12cbe[865](https://github.com/ursacomputing/crossbow/actions/runs/7613109791/job/20732245591#step:6:866)a3377d69c44f700e62742dabee33402d' build failed
arrow/16.0.0: WARN: Build folder /root/.conan/data/arrow/16.0.0/_/_/build/12cbe865a3377d69c44f700e62742dabee33402d/build/Release
ERROR: arrow/16.0.0: Error in build() method, line 576
	cmake.configure(build_script_folder=os.path.join(self.source_folder, "cpp"))
	ConanException: Error 1 while executing cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="/root/.conan/data/arrow/16.0.0/_/_/build/12cbe865a3377d69c44f700e62742dabee33402d/build/Release/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/.conan/data/arrow/16.0.0/_/_/package/12cbe865a3377d69c44f700e62742dabee33402d" -DCMAKE_PROJECT_arrow_INCLUDE="/root/.conan/data/arrow/16.0.0/_/_/build/12cbe865a3377d69c44f700e62742dabee33402d/src/conan_cmake_project_include.cmake" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/.conan/data/arrow/16.0.0/_/_/build/12cbe865a3377d69c44f700e62742dabee33402d/src/cpp"
1
Error: `docker-compose --file /home/runner/work/crossbow/crossbow/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=16.0.0.dev5 -e ARROW_CONAN_PARQUET=True -e ARROW_CONAN_WITH_BROTLI=True -e ARROW_CONAN_WITH_BZ2=True -e ARROW_CONAN_WITH_FLIGHT_RPC=True -e ARROW_CONAN_WITH_GLOG=True -e ARROW_CONAN_WITH_JEMALLOC=True -e ARROW_CONAN_WITH_JSON=True -e ARROW_CONAN_WITH_LZ4=True -e ARROW_CONAN_WITH_SNAPPY=True -e ARROW_CONAN_WITH_ZSTD=True conan` exited with a non-zero exit code 1, see the process log above.

I tried to debug it locally with command conan create . --profile:host default --profile:build default --build=missing --version 14.0.2 --options arrow:parquet=True --options arrow:with_brotli=True --options arrow:with_bz2=True --options arrow:with_flight_rpc=True --options arrow:with_glog=True --options arrow:with_jemalloc=True --options arrow:with_json=True --options arrow:with_lz4=True --options arrow:with_snappy=True --options arrow:with_zstd=True, which is copied from conan-maximum except that I used version 14.0.2. It complained ERROR: Error while initializing options. The usage of package names arrow:parquet in options is deprecated, use a pattern like arrow/*:parquet instead.

Replacing the options with --options parquet=True --options with_brotli=True --options with_bz2=True --options with_flight_rpc=True --options with_glog=True --options with_jemalloc=True --options with_json=True --options with_lz4=True --options with_snappy=True --options with_zstd=True can move forward to the following error on my laptop:

-- Checking for module 'grpc++'
--   No package 'grpc++' found
CMake Error at /Users/gangwu/.conan2/p/cmakefc220a4e56e94/p/CMake.app/Contents/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find gRPCAlt (missing: GRPCPP_IMPORTED_LOCATION GRPC_CPP_PLUGIN)
  (Required is at least version "1.30.0")
Call Stack (most recent call first):
  /Users/gangwu/.conan2/p/cmakefc220a4e56e94/p/CMake.app/Contents/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  cmake_modules/FindgRPCAlt.cmake:66 (find_package_handle_standard_args)
  cmake_modules/ThirdpartyToolchain.cmake:296 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:4020 (resolve_dependency)
  CMakeLists.txt:542 (include)

Then I appended --options with_grpc=True to the command to include gprc as well. It failed at the install step:

-- Installing: /Users/gangwu/.conan2/p/b/arrow38f78ba52e915/p/include/parquet/parquet_version.h
-- Installing: /Users/gangwu/.conan2/p/b/arrow38f78ba52e915/p/share/doc/arrow/LICENSE.txt
-- Installing: /Users/gangwu/.conan2/p/b/arrow38f78ba52e915/p/share/doc/arrow/NOTICE.txt
-- Installing: /Users/gangwu/.conan2/p/b/arrow38f78ba52e915/p/share/doc/arrow/README.md
-- Installing: /Users/gangwu/.conan2/p/b/arrow38f78ba52e915/p/share/arrow/gdb/gdb_arrow.py

arrow/14.0.2: package(): Packaged 2 '.txt' files: NOTICE.txt, LICENSE.txt
arrow/14.0.2: package(): Packaged 336 '.h' files
arrow/14.0.2: package(): Packaged 3 '.hpp' files: pcg_extras.hpp, pcg_uint128.hpp, pcg_random.hpp
arrow/14.0.2: package(): Packaged 3 '.a' files: libparquet.a, libarrow_flight.a, libarrow.a
arrow/14.0.2: Created package revision aa1d1310d053af9a589be8cb70f55552
arrow/14.0.2: Package '8dd229cff99fc16b43426c82edde907cc028a61b' created
arrow/14.0.2: Full package reference: arrow/14.0.2#62816e9907b36768dee58f6b71cc3399:8dd229cff99fc16b43426c82edde907cc028a61b#aa1d1310d053af9a589be8cb70f55552
arrow/14.0.2: Package folder /Users/gangwu/.conan2/p/b/arrow38f78ba52e915/p
ERROR: arrow/14.0.2: required component package 'protobuf::' not in dependencies

Appending --options with_protobuf=True to the command ended up with following error:

ERROR: Version conflict: Conflict between protobuf/3.21.12 and protobuf/3.21.9 in the graph.
Conflict originates from grpc/1.50.0

Run conan graph info with your recipe and add --format=html to inspect the graph errors in an easier to visualize way.

Finally I changed protobuf requirement from 3.21.9 to 3.21.12 resolved the issue:

arrow/14.0.2: package(): Packaged 2 '.txt' files: NOTICE.txt, LICENSE.txt
arrow/14.0.2: package(): Packaged 336 '.h' files
arrow/14.0.2: package(): Packaged 3 '.hpp' files: pcg_extras.hpp, pcg_uint128.hpp, pcg_random.hpp
arrow/14.0.2: package(): Packaged 3 '.a' files: libparquet.a, libarrow_flight.a, libarrow.a
arrow/14.0.2: Created package revision efbf450e954429d0b03483be6f34286f
arrow/14.0.2: Package 'eb5b71dec6425480023b196b58b1283cc1b1969d' created
arrow/14.0.2: Full package reference: arrow/14.0.2#6da1ed12f8e53fdb2432af2768057820:eb5b71dec6425480023b196b58b1283cc1b1969d#efbf450e954429d0b03483be6f34286f
arrow/14.0.2: Package folder /Users/gangwu/.conan2/p/b/arrowccacf653c6c19/p

Perhaps we need to update the build script? @kou @raulcd

@kou
Copy link
Member

kou commented Jan 22, 2024

Does this work?

diff --git a/ci/scripts/conan_build.sh b/ci/scripts/conan_build.sh
index b1ee0a8fc2..c1862c16f1 100755
--- a/ci/scripts/conan_build.sh
+++ b/ci/scripts/conan_build.sh
@@ -30,34 +30,34 @@ export CONAN_HOOK_ERROR_LEVEL=40
 conan_args=()
 conan_args+=(--build=missing)
 if [ -n "${ARROW_CONAN_PARQUET:-}" ]; then
-  conan_args+=(--options arrow:parquet=${ARROW_CONAN_PARQUET})
+  conan_args+=(--options "arrow/*:parquet=${ARROW_CONAN_PARQUET}")
 fi
 if [ -n "${ARROW_CONAN_WITH_BROTLI:-}" ]; then
-  conan_args+=(--options arrow:with_brotli=${ARROW_CONAN_WITH_BROTLI})
+  conan_args+=(--options "arrow/*:with_brotli=${ARROW_CONAN_WITH_BROTLI}")
 fi
 if [ -n "${ARROW_CONAN_WITH_BZ2:-}" ]; then
-  conan_args+=(--options arrow:with_bz2=${ARROW_CONAN_WITH_BZ2})
+  conan_args+=(--options "arrow/*:with_bz2=${ARROW_CONAN_WITH_BZ2}")
 fi
 if [ -n "${ARROW_CONAN_WITH_FLIGHT_RPC:-}" ]; then
-  conan_args+=(--options arrow:with_flight_rpc=${ARROW_CONAN_WITH_FLIGHT_RPC})
+  conan_args+=(--options "arrow/*:with_flight_rpc=${ARROW_CONAN_WITH_FLIGHT_RPC}")
 fi
 if [ -n "${ARROW_CONAN_WITH_GLOG:-}" ]; then
-  conan_args+=(--options arrow:with_glog=${ARROW_CONAN_WITH_GLOG})
+  conan_args+=(--options "arrow/*:with_glog=${ARROW_CONAN_WITH_GLOG}")
 fi
 if [ -n "${ARROW_CONAN_WITH_JEMALLOC:-}" ]; then
-  conan_args+=(--options arrow:with_jemalloc=${ARROW_CONAN_WITH_JEMALLOC})
+  conan_args+=(--options "arrow/*:with_jemalloc=${ARROW_CONAN_WITH_JEMALLOC}")
 fi
 if [ -n "${ARROW_CONAN_WITH_JSON:-}" ]; then
-  conan_args+=(--options arrow:with_json=${ARROW_CONAN_WITH_JSON})
+  conan_args+=(--options "arrow/*:with_json=${ARROW_CONAN_WITH_JSON}")
 fi
 if [ -n "${ARROW_CONAN_WITH_LZ4:-}" ]; then
-  conan_args+=(--options arrow:with_lz4=${ARROW_CONAN_WITH_LZ4})
+  conan_args+=(--options "arrow/*:with_lz4=${ARROW_CONAN_WITH_LZ4}")
 fi
 if [ -n "${ARROW_CONAN_WITH_SNAPPY:-}" ]; then
-  conan_args+=(--options arrow:with_snappy=${ARROW_CONAN_WITH_SNAPPY})
+  conan_args+=(--options "arrow/*:with_snappy=${ARROW_CONAN_WITH_SNAPPY}")
 fi
 if [ -n "${ARROW_CONAN_WITH_ZSTD:-}" ]; then
-  conan_args+=(--options arrow:with_zstd=${ARROW_CONAN_WITH_ZSTD})
+  conan_args+=(--options "arrow/*:with_zstd=${ARROW_CONAN_WITH_ZSTD}")
 fi
 
 version=$(grep '^set(ARROW_VERSION ' ${ARROW_HOME}/cpp/CMakeLists.txt | \

@wgtmac

This comment was marked as outdated.

This comment was marked as outdated.

@wgtmac
Copy link
Member Author

wgtmac commented Jan 23, 2024

@kou I found the problem: https://github.com/conan-io/conan-center-index/pull/19380/files#diff-8bccf85b8738fe9963099a68c25d98d0160fb1ed06657221ea2daebea2c685f0L273. We cannot replace self._with_xxx() with self.options.with_xxx, otherwise it only checks user-provided options, not the auto detect logic. If you can confirm this, we have to upstream our change anyway.

@github-actions github-actions bot added Component: C++ awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Apr 11, 2024
@wgtmac
Copy link
Member Author

wgtmac commented Apr 11, 2024

@github-actions crossbow submit -g conan

Copy link

Command '['git', 'clone', 'https://github.com/ursacomputing/crossbow', '/tmp/tmpnvcdle_b/crossbow']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/8648947115

@wgtmac
Copy link
Member Author

wgtmac commented Apr 11, 2024

@github-actions crossbow submit -g conan

Copy link

Revision: 8c062fb

Submitted crossbow builds: ursacomputing/crossbow @ actions-4b0ee017a9

Task Status
conan-maximum GitHub Actions
conan-minimum GitHub Actions

@wgtmac
Copy link
Member Author

wgtmac commented Apr 12, 2024

@github-actions crossbow submit -g conan

Copy link

Revision: 2a2d398

Submitted crossbow builds: ursacomputing/crossbow @ actions-d779ca6ba6

Task Status
conan-maximum GitHub Actions
conan-minimum GitHub Actions

@wgtmac
Copy link
Member Author

wgtmac commented Apr 12, 2024

@github-actions crossbow submit -g conan

Copy link

Revision: 5259cd0

Submitted crossbow builds: ursacomputing/crossbow @ actions-caea67edfa

Task Status
conan-maximum GitHub Actions
conan-minimum GitHub Actions

@wgtmac
Copy link
Member Author

wgtmac commented Apr 12, 2024

Finally they are all green! PTAL @kou @raulcd

Copy link
Member

@kou kou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Thanks!!!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should improve our CMake files to remove this but we can do it as a follow-up task.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have created #41196 and will work on this.

@github-actions github-actions bot added awaiting review Awaiting review awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Apr 12, 2024
@wgtmac wgtmac merged commit 18c74b0 into apache:main Apr 15, 2024
60 of 61 checks passed
@wgtmac wgtmac removed awaiting review Awaiting review awaiting merge Awaiting merge labels Apr 15, 2024
@github-actions github-actions bot added the awaiting committer review Awaiting committer review label Apr 15, 2024
vibhatha pushed a commit to vibhatha/arrow that referenced this pull request Apr 15, 2024
…ache#39729)

### What changes are included in this PR?

Merged upstream conan using `ci/conan/merge_upstream.sh ../conan-center-index/`

### Are these changes tested?

Will be tested on CI.

### Are there any user-facing changes?

No
* Closes: apache#35804

Authored-by: Gang Wu <[email protected]>
Signed-off-by: Gang Wu <[email protected]>
Copy link

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 18c74b0.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.

tolleybot pushed a commit to tmct/arrow that referenced this pull request May 2, 2024
…ache#39729)

### What changes are included in this PR?

Merged upstream conan using `ci/conan/merge_upstream.sh ../conan-center-index/`

### Are these changes tested?

Will be tested on CI.

### Are there any user-facing changes?

No
* Closes: apache#35804

Authored-by: Gang Wu <[email protected]>
Signed-off-by: Gang Wu <[email protected]>
vibhatha pushed a commit to vibhatha/arrow that referenced this pull request May 25, 2024
…ache#39729)

### What changes are included in this PR?

Merged upstream conan using `ci/conan/merge_upstream.sh ../conan-center-index/`

### Are these changes tested?

Will be tested on CI.

### Are there any user-facing changes?

No
* Closes: apache#35804

Authored-by: Gang Wu <[email protected]>
Signed-off-by: Gang Wu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CI][Packaging][Conan] Synchronize to upstream conan-center-index
3 participants