Broken C++ Coverage with hermetic clang toolchain - No override for COVERAGE_GCOV_PATH and LLVM_COV possible with --test_env

[omar-droubi]

Description of the bug:

Hello Everyone,

Starting with Bazel 7.2.1 it's no longer possible to generate c++ code coverage using a hermetic Clang toolchain. I am using pw_toolchain_bazel to generate the toolchains definitions. Since our system doesn't have GCOV installed, and would like to use LLVM for the coverage, we override COVERAGE_GCOV_PATH and LLVM_COV with --test_env:

bazelrc content:

test --test_env=COVERAGE_GCOV_PATH=external/_main~_repo_rules~llvm_toolchain/bin/llvm-profdata
test --test_env=LLVM_COV=external/_main~_repo_rules~llvm_toolchain/bin/llvm-cov

By default Bazel extracts these two env variables from the tool_path for GCOV and LLVM_COV, but not from the action_configs for coverage actions.

cc_helper.bzl@1090

def _get_coverage_environment(ctx, cc_config, cc_toolchain):
    if not ctx.configuration.coverage_enabled:
        return {}

    env = {
        "COVERAGE_GCOV_PATH": cc_toolchain.tool_path(tool = "GCOV"),
        "LLVM_COV": cc_toolchain.tool_path(tool = "LLVM_COV"),
        "LLVM_PROFDATA": cc_toolchain.tool_path(tool = "LLVM_PROFDATA"),
        "GENERATE_LLVM_LCOV": "1" if cc_config.generate_llvm_lcov() else "0",
    }

Given that the current recommend way is to use action_configs and not tool_paths to define Action Tools, we do not set the tool path for GCOV and LLVM_COV, and their value is set to "".

This was fine since we could override their values using --test_env. However after the change 87b0a1f we can no longer override them.

This is happening because of the change in the order of env variables population in src/main/java/com/google/devtools/build/lib/exec/TestPolicy.java.

// Overwrite with the environment common to all tests, see --test_env.
// Omar: Here COVERAGE_GCOV_PATH and LLVM_COV take the correct values from --test_env
testAction.getConfiguration().getTestActionEnvironment().resolve(env, clientEnv);

// Rule-specified test env.
// Omar: getExtraTestEnv() has already COVERAGE_GCOV_PATH and LLVM_COV values defined 
// by cc_helper and they are set to "". These values overwrite what was supplied using --test_env
testAction.getExtraTestEnv().resolve(env, clientEnv);

Proposed Solutions:

1. _get_coverage_environment in cc_helper.bzl should use get_tool_for_action instead of tool_path.
2. Remove setting to default if tools are not found in _get_coverage_environment.
3. Explicit filtering for COVERAGE_GCOV_PATH, LLVM_COV, LLVM_PROFDATA,GENERATE_LLVM_LCOV in resolve function.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

WSL Ubuntu 20.04

What is the output of bazel info release?

release 7.2.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

87b0a1f

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

@bazel-io flag

[fmeum]

@c-mita

[iancha1992]

@bazel-io fork 7.4.0

[omar-droubi]

Hi @c-mita / @fmeum : Is this topic still in the pipeline, or are you waiting for the full migration to rules_cc to work on it there ?

[c-mita]

Given that the current recommend way is to use action_configs and not tool_paths to define Action Tools, we do not set the tool path for GCOV and LLVM_COV, and their value is set to "".

I'm don't know if action_config does anything with the coverage tools - why not have the toolchain specify the path for the tools?

[omar-droubi]

In Pigweed it was not allowed to mix ActionConfig and plain tool_path, and also now in rules_cc, the toolchain config only accepts a cc_tool_map which is just a list of ActionConfigs. There is no way I can find to inject a plain old tool_path.

[keith]

i submitted #24670

[iancha1992]

A fix for this issue has been included in Bazel 7.5.0 RC2. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.5.0rc2. Thanks!

[axeluhlig]

We use --test_env=CC_CODE_COVERAGE_SCRIPT=external/cxx_tooling/cc_code_coverage_script.sh and still get the GCov does not exist at the given path: '' error when using Bazel 7.5.0rc2. It does not pick up the value we provide but instead uses external/bazel_tools/tools/test/collect_cc_coverage.sh.

I just tested it again, it Bazel 7.1 our value for CC_CODE_COVERAGE_SCRIPT gets propagated through.

Are we just using an unsupported feature here or is this bug only partially fixed?

[keith]

I think that one is complaining about either BAZEL_CC_COVERAGE_TOOL or GCOV, maybe setting one of those will sidestep?

[jmdsouza]

I think that one is complaining about either BAZEL_CC_COVERAGE_TOOL or GCOV, maybe setting one of those will sidestep?

I'm Axel's colleague. We have resolved the previous error mentioned of GCov not existing by running coverage --build_runfile_links and this helps us execute coverage runs by using the default Bazel coverage scripts (collect_cc_coverage.sh).

However, previously we were able to provide a custom script to execute instead of collect_cc_coverage.sh. Just want to provide more details on our use case. We currently use --test_env=CC_CODE_COVERAGE_SCRIPT= to override the default Bazel coverage script in order to add -sparse to help maintain the file size and help with disk space issues in CI

This line in particular from collect_cc_coverage.sh is modified for this purpose.

  "${COVERAGE_GCOV_PATH}" merge -output "${output_file}.data" \
      "${COVERAGE_DIR}"/*.profraw 

Is this something that is supported? Thanks!

[keith]

I'm not sure i would say it was ever "supported", but you aren't the only people using it so i think we should fix it

[jmdsouza]

I'm not sure i would say it was ever "supported", but you aren't the only people using it so i think we should fix it

I just wanted to provide an update in case any others might need a workaround. Instead of setting CC_CODE_COVERAGE_SCRIPT through --test-env, we set the environment variables through cc_test. We did encounter a similar issue with LCOV_MERGER. RunEnvironmentInfo should also work to set these environment variables.