Git Repository is cached to different path each run

[Zephilinox]

Hey :) we're using CPM 0.35 and we were seeing cache misses in ccache. I believe this is being caused by the path that CPM is downloading dependencies to changing (CPM_SOURCE_CACHE is set)

For example, one CI job will report the following

-- CPM: adding package a@0 (39c696e05ffbcec63a89b16e6d0a8f5a81803d0b to /.cache/cpm/a/53bf86368c3519b360cabd404562908f61c8d31d)
-- Found Git: /usr/bin/git (found version "2.25.1") 
-- CPM: adding package [email protected] (v2.12.1 at /.cache/cpm/catch2/48298e25cc7abae02d19d6fc9f52adddcc4dfc91)
-- CPM: adding package b@0 (57ae6de5383d5ed33e728e7dba1f36d4217fc130 to /.cache/cpm/b/441365e832abf8537d5ebee4355b61c7c0b62bd4)

where as the next shows

-- CPM: adding package a@0 (39c696e05ffbcec63a89b16e6d0a8f5a81803d0b to /.cache/cpm/a/4078ef9cb0becef1d5e9ad625bcfe6fff39fb176)
-- Found Git: /usr/bin/git (found version "2.25.1") 
-- CPM: adding package [email protected] (v2.12.1 at /builds/leap-v5-platform/camera-integration-toolkit/.cache/cpm/catch2/48298e25cc7abae02d19d6fc9f52adddcc4dfc91)
-- CPM: adding package b@0 (57ae6de5383d5ed33e728e7dba1f36d4217fc130 to /.cache/cpm/b/47b5fb6eb87f516625081082e15b1546bfc5ce74)

notably a and b are internal self-hosted gitlab repositories.
Catch2's path is stable across jobs.

Catch2 is fetched via

CPMAddPackage("gh:catchorg/[email protected]")

where as both internal repos are fetched similarly, with a looking like:

CPMAddPackage(
  NAME a
  GIT_REPOSITORY "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.company.com/common/a.git"
  GIT_TAG "39c696e05ffbcec63a89b16e6d0a8f5a81803d0b"
)

where the GIT_TAG is a commit hash on our default branch, not an actual tag

is there something we're missing to enable consistent cached paths across runs?

[itavero]

The problem is probably that the CI_JOB_TOKEN changes in between runs. Not sure if setting CPM_USE_NAMED_CACHE_DIRECTORIES would help in this case.

Depending on how #333 will be implemented and if it will be merged into CPM.cmake, that might be a good solution to your problem.

[Zephilinox]

ooh thanks :)

unfortunately it looks like that only adds an additional subfolder, and doesn't change the hash used for the path

    if(CPM_USE_NAMED_CACHE_DIRECTORIES)
      string(SHA1 origin_hash "${origin_parameters};NEW_CACHE_STRUCTURE_TAG")
      set(download_directory ${CPM_SOURCE_CACHE}/${lower_case_name}/${origin_hash}/${CPM_ARGS_NAME})
    else()
      string(SHA1 origin_hash "${origin_parameters}")
      set(download_directory ${CPM_SOURCE_CACHE}/${lower_case_name}/${origin_hash})
    endif()

might be able to work around it or hack it in though :b

would a PR be welcome to provide an option to override the hash in some way/change the hash to not use the URL?

[itavero]

I'm not a maintainer of CPM.cmake, but I can imagine that instead of overriding the hash, perhaps it's useful to have an option to exclude the authentication part of the URL from the input for generating the hash.

@TheLartians, any thoughts on this?

[OlivierLDff]

I believe you should use something like:

git config --global url."https://olivierldff:[email protected]".insteadOf "https://github.com"

And only keep the url without the token in your source cmakelists file.
Syntax is what I use for GitHub, I'm sure it can be adapted fo gitlab.

[TheLartians]

IMO @OlivierLDff's solution is the way to go here, as it's more general purpose and keeps token handling outside of CMake. Note that this works effortlessly for other package managers such as pip as well. We are currently using this for packages in a self-hosted GitLab instance without issue.

[OlivierLDff]

Maybe it should be highlighted as a cool trick in readme, I discovered this technique from some of your comments but it is not mentioned a lot on the internet.

[TheLartians]

Oh that's a good idea! I'm currently quite busy, so feel free to update the readme in a PR if you want.

[TheLartians]

@Zephilinox does using the git config described above solve the issue for you?