[REVIEW] [ENH] Ability to use ccache to speedup local builds #729

teju85 · 2022-06-30T14:56:32Z

I noticed that we have support for sccache in build.sh for building RAFT, but no ccache (especially if one wants to do local builds). This PR adds support for the latter. Also includes changes to install ccache in raft dev conda env.

Let me know in case you have any concerns with adding this.

achirkin · 2022-07-01T05:46:45Z

NB: I meant to follow up on PR #651 , but kept postponing it :)

achirkin · 2022-07-01T05:51:15Z

Would it make sense to enable ccache by default if no arguments are passed to build.sh?

teju85 · 2022-07-01T07:29:44Z

@achirkin I agree with Robert's comment in your PR above. I think this should be an option that should be enabled by the users when they want it. So, that's why this PR makes changes to the build.sh only.

achirkin · 2022-07-01T07:36:00Z

Sure, I totally agree. I've just put it here to link the discussion and to not forget to close that PR. (That still leaves open the question if it's ok to enable ccache when no arguments are passed to build.sh)

achirkin · 2022-07-01T07:40:15Z

conda/environments/raft_dev_cuda11.5.yml

@@ -23,6 +23,7 @@ dependencies:
 - doxygen>=1.8.20
 - libfaiss>=1.7.0
 - faiss-proc=*=cuda
+- ccache


Shall we do the same changes in cuml? I personally always use cuml_dev environment for raft development, and I suspect that's not me only.

Agree with you. If this PR gets accepted by ops team, we could do the same in cuML too.

robertmaynard · 2022-07-01T12:16:41Z

build.sh

@@ -18,7 +18,7 @@ ARGS=$*
 # script, and that this script resides in the repo dir!
 REPODIR=$(cd $(dirname $0); pwd)

-VALIDARGS="clean libraft pyraft pylibraft docs tests bench clean -v -g --install --compile-libs --compile-nn --compile-dist --allgpuarch --sccache --no-nvtx --show_depr_warn -h --buildfaiss --minimal-deps"
+VALIDARGS="clean libraft pyraft pylibraft docs tests bench clean -v -g --install --compile-libs --compile-nn --compile-dist --allgpuarch --sccache --ccache --no-nvtx --show_depr_warn -h --buildfaiss --minimal-deps"


In general I have recommended we switch over to a command line pattern like --cache-tool=<program> the reason for this is the following:

It allows a user to specify either sccache or ccache.

Since they provide a name or path we don't require the command to be on the PATH

It removes the need to check if both options are enabled

It unifies the hasArg --ccache and hasArg --sccache conditionals into a single branch

It allows people to also use cache tools like distcc

I like this approach! Thanks Robert. Let me update the PR.

@robertmaynard can I get a re-review please?

…e options

robertmaynard

Looks amazing! Thank you

build.sh

Co-authored-by: AJ Schmidt <[email protected]>

ajschmidt8

The rest of the --ccache-tool instances in build.sh still need to be updated as mentioned in my comment below.

[REVIEW] [ENH] Ability to use ccache to speedup local builds #729 (comment)

…he-tool

teju85 · 2022-07-06T16:11:25Z

ha... I missed your suggestion. Thanks AJ for catching this. The latest commit above should address this now.

teju85 · 2022-07-07T09:00:43Z

@gpucibot merge

ahendriksen · 2022-07-18T16:05:15Z

In the nightly development docker image, the ccache integration does not yet work 100% of time time. I have been using

./build.sh  tests bench --cache-tool=ccache

today. At the end of the day, compilation still seemed not to hit the cache. These are the statistics that ccache reports: (note the unsupported source language statistic)

(rapids) root@492c2feb1be4:/workspace/raft/cpp/build# ccache -s
cache directory                     /root/.ccache
primary config                      /root/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Mon Jul 18 15:54:10 2022
cache hit (direct)                    10
cache hit (preprocessed)               2
cache miss                            27
cache hit rate                     30.77 %
unsupported source language          533   <---------------------------
cleanups performed                     0
files in cache                        83
cache size                           5.9 MB
max cache size                       5.0 GB

Does anybody know how to resolve this issue? The only thing I have been able to find is this ccache issue ccache/ccache#772.

robertmaynard · 2022-07-18T18:40:27Z

@ahendriksen I expect that the version of ccache you are using doesn't fully support the CUDA language.

In Ubuntu 20.04 the ccache version is 3.7.7 and numerous issues with the CUDA language was backported to the 3 series in 3.7.9. I know locally ( and rapids-compose ) both use ccache 4.5+ for full CUDA support.

ahendriksen · 2022-07-19T10:06:13Z

@robertmaynard Thank you! That solved the issue completely. Cache is now hit 100% of the time and build time is down to 9 seconds.

added ability to use ccache to speedup local builds

340b421

teju85 requested review from a team as code owners June 30, 2022 14:56

teju85 added enhancement New feature or request non-breaking Non-breaking change labels Jun 30, 2022

teju85 changed the title ~~[REVIEW] [FEA] Ability to use ccache to speedup local builds~~ [REVIEW] [ENH] Ability to use ccache to speedup local builds Jun 30, 2022

achirkin reviewed Jul 1, 2022

View reviewed changes

robertmaynard reviewed Jul 1, 2022

View reviewed changes

used a --ccache-tool option instead of separate --sccache and --ccach…

0672024

…e options

teju85 added the improvement Improvement / enhancement to an existing function label Jul 4, 2022

robertmaynard approved these changes Jul 6, 2022

View reviewed changes

ajschmidt8 reviewed Jul 6, 2022

View reviewed changes

build.sh Outdated Show resolved Hide resolved

Update build.sh

c8ac076

Co-authored-by: AJ Schmidt <[email protected]>

raydouglass approved these changes Jul 6, 2022

View reviewed changes

ajschmidt8 requested changes Jul 6, 2022

View reviewed changes

updated rest of the occurrences of ccache-tool option usages with cac…

7e4cc5e

…he-tool

ajschmidt8 approved these changes Jul 6, 2022

View reviewed changes

rapids-bot bot merged commit 2b27bad into rapidsai:branch-22.08 Jul 7, 2022

teju85 deleted the fea-ext-enable-ccache branch July 7, 2022 09:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[REVIEW] [ENH] Ability to use ccache to speedup local builds #729

[REVIEW] [ENH] Ability to use ccache to speedup local builds #729

teju85 commented Jun 30, 2022

achirkin commented Jul 1, 2022

achirkin commented Jul 1, 2022

teju85 commented Jul 1, 2022

achirkin commented Jul 1, 2022 •

edited

Loading

achirkin Jul 1, 2022

teju85 Jul 4, 2022

robertmaynard Jul 1, 2022

teju85 Jul 4, 2022

teju85 Jul 6, 2022

robertmaynard left a comment

ajschmidt8 left a comment

teju85 commented Jul 6, 2022

teju85 commented Jul 7, 2022

ahendriksen commented Jul 18, 2022

robertmaynard commented Jul 18, 2022

ahendriksen commented Jul 19, 2022

[REVIEW] [ENH] Ability to use ccache to speedup local builds #729

[REVIEW] [ENH] Ability to use ccache to speedup local builds #729

Conversation

teju85 commented Jun 30, 2022

achirkin commented Jul 1, 2022

achirkin commented Jul 1, 2022

teju85 commented Jul 1, 2022

achirkin commented Jul 1, 2022 • edited Loading

achirkin Jul 1, 2022

Choose a reason for hiding this comment

teju85 Jul 4, 2022

Choose a reason for hiding this comment

robertmaynard Jul 1, 2022

Choose a reason for hiding this comment

teju85 Jul 4, 2022

Choose a reason for hiding this comment

teju85 Jul 6, 2022

Choose a reason for hiding this comment

robertmaynard left a comment

Choose a reason for hiding this comment

ajschmidt8 left a comment

Choose a reason for hiding this comment

teju85 commented Jul 6, 2022

teju85 commented Jul 7, 2022

ahendriksen commented Jul 18, 2022

robertmaynard commented Jul 18, 2022

ahendriksen commented Jul 19, 2022

achirkin commented Jul 1, 2022 •

edited

Loading