General project updates; rebase onto upstream v6.11; add notes about CONFIG_WERROR

.github/workflows/llvm-trunk-linux-mainline.yml

@@ -8,7 +8,7 @@ on:
     paths:
     - '.github/workflows/llvm-trunk-linux-mainline.yml'
     - 'ci/linux-mainline/**'
-    - 'patches/v2.0/**'
+    - 'patches/v3.0/**'
     - 'scripts/**'
   workflow_dispatch:
   schedule:

README.md

@@ -5,17 +5,19 @@
 [![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/xlab-uiuc/linux-mcdc/llvm-trunk-linux-v5.15.153.yml?label=LLVM%20trunk%2BLinux%20v5.15.153)](https://github.com/xlab-uiuc/linux-mcdc/actions/workflows/llvm-trunk-linux-v5.15.153.yml)
 
 This repository demonstrates [KUnit](https://docs.kernel.org/dev-tools/kunit/index.html)'s
-modified condition/decision coverage (MC/DC) of 5.15.y Linux kernel using
-[Clang source-based code coverage](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html)
-and [`llvm-cov`](https://llvm.org/docs/CommandGuide/llvm-cov.html). A patch set
-for mainline kernel is also being prepared.
+modified condition/decision coverage (MC/DC) of 5.15.y and mainline Linux kernel
+using [Clang source-based code coverage](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html)
+and [`llvm-cov`](https://llvm.org/docs/CommandGuide/llvm-cov.html).
 
 <!--
 Primary
 development of the kernel patch set is being performed in the [xlab-uiuc/llvm-cov](https://github.com/xlab-uiuc/linux-cov)
 project.
 -->
 
+**Patches can be found under [`patches/`](patches/), or join LKML discussions:
+[link](https://lore.kernel.org/lkml/[email protected]/)**
+
 Follow the instructions [here](docs/measure-kernel-mcdc.md) to get started.
 
 Example text coverage report: [link](https://github.com/xlab-uiuc/linux-mcdc/actions/runs/10013137034/job/27681036852#step:8:7) (login with any GitHub account required)
@@ -24,34 +26,17 @@ Example HTML coverage report:
 
 <img src="screenshot.png" width="70%">
 
-Tentative repository structure:
-
-```text
-linux-mcdc
-|
-├── ci/{linux-v5.15.153}
-│   ├── 1_install_deps.sh
-│   ├── 2_pull_source.sh
-│   ├── 3_get_llvm.sh
-│   ├── 4_build_kernel.sh
-│   └── 5_boot_kernel_and_collect_coverage.sh
-|
-├── docs
-│   ├── elisa-slides.pdf
-│   └── measure-kernel-mcdc.md
-|
-├── patches
-│   ├── README.md
-│   └── {v0.4,v0.5,v0.6}
-|
-├── README.md
-|
-├── screenshot.png
-|
-└── scripts
-    ├── build-llvm.sh
-    └── q
-```
+We gave three talks in [LPC 2024](https://lpc.events/event/18/page/224-lpc-2024-overview):
+
+- [Making Linux Fly: Towards a Certified Linux Kernel](https://lpc.events/event/18/contributions/1718/) (Refereed Track)
+  [[recording](https://www.youtube.com/live/1KWkfHxTqYY?feature=shared&t=3957)]
+  [[slides](https://lpc.events/event/18/contributions/1718/attachments/1584/3477/LPC'24%20Fly%20(no%20animation).pdf)]
+- [Measuring and Understanding Linux Kernel Tests](https://lpc.events/event/18/contributions/1793/) (Kernel Testing & Dependability MC)
+  [[recording](https://www.youtube.com/live/kcr8NXEbzcg?feature=shared&t=9380)]
+  [[slides](https://lpc.events/event/18/contributions/1793/attachments/1624/3447/LPC'24%20Linux%20Testing.pdf)]
+- [Source-based code coverage of Linux kernel](https://lpc.events/event/18/contributions/1895/) (Safe Systems with Linux MC)
+  [[recording](https://www.youtube.com/live/kcr8NXEbzcg?feature=shared&t=23820)]
+  [[slides](https://lpc.events/event/18/contributions/1895/attachments/1643/3462/LPC'24%20Source%20based%20(short).pdf)]
 
 We gave an [ELISA](https://elisa.tech/) seminar titled "Making Linux Fly: Towards Certified Linux
 Kernel".

patches/latest

@@ -1 +1 @@
-v2.0/
+v3.0/

patches/v2.0/0000-cover-letter.patch

@@ -1,7 +1,7 @@
 From 7c60851daae39815fa696fc2265e838b1913f7ac Mon Sep 17 00:00:00 2001
 From: Wentao Zhang <[email protected]>
 Date: Wed, 4 Sep 2024 18:13:07 -0500
-Subject: [RFC PATCH 0/4] Enable measuring the kernel's Source-based Code Coverage and MC/DC with Clang
+Subject: [PATCH v2 0/4] Enable measuring the kernel's Source-based Code Coverage and MC/DC with Clang
 
 This series adds support for building x86-64 kernels with Clang's Source-
 based Code Coverage[1] in order to facilitate Modified Condition/Decision

patches/v2.0/0001-llvm-cov-add-Clang-s-Source-based-Code-Coverage-supp.patch

@@ -1,7 +1,7 @@
 From 13c86fc7877aec4892832554845968d7415f2960 Mon Sep 17 00:00:00 2001
 From: Wentao Zhang <[email protected]>
 Date: Fri, 30 Aug 2024 14:48:03 -0500
-Subject: [RFC PATCH 1/4] llvm-cov: add Clang's Source-based Code Coverage
+Subject: [PATCH v2 1/4] llvm-cov: add Clang's Source-based Code Coverage
  support
 
 Add infrastructure to support Clang's source-based code coverage [1]. This

patches/v2.0/0002-llvm-cov-add-Clang-s-MC-DC-support.patch

@@ -1,7 +1,7 @@
 From e7d38a2115b9c1041dcda404f102888c9894da10 Mon Sep 17 00:00:00 2001
 From: Wentao Zhang <[email protected]>
 Date: Fri, 30 Aug 2024 14:49:24 -0500
-Subject: [RFC PATCH 2/4] llvm-cov: add Clang's MC/DC support
+Subject: [PATCH v2 2/4] llvm-cov: add Clang's MC/DC support
 
 Add infrastructure to enable Clang's Modified Condition/Decision Coverage
 (MC/DC) [1].

patches/v2.0/0003-x86-disable-llvm-cov-instrumentation.patch

@@ -1,7 +1,7 @@
 From f4cc26b9d49b7c34671c04a8b62485a311530298 Mon Sep 17 00:00:00 2001
 From: Wentao Zhang <[email protected]>
 Date: Fri, 30 Aug 2024 14:48:53 -0500
-Subject: [RFC PATCH 3/4] x86: disable llvm-cov instrumentation
+Subject: [PATCH v2 3/4] x86: disable llvm-cov instrumentation
 
 Disable instrumentation for arch/x86/crypto/curve25519-x86_64.c. Otherwise
 compilation would fail with "error: inline assembly requires more registers

patches/v2.0/0004-x86-enable-llvm-cov-support.patch

@@ -1,7 +1,7 @@
 From 7c60851daae39815fa696fc2265e838b1913f7ac Mon Sep 17 00:00:00 2001
 From: Wentao Zhang <[email protected]>
 Date: Fri, 30 Aug 2024 14:49:06 -0500
-Subject: [RFC PATCH 4/4] x86: enable llvm-cov support
+Subject: [PATCH v2 4/4] x86: enable llvm-cov support
 
 Set ARCH_HAS_* options to "y" in kconfig and include section description in
 linker script.

patches/v3.0/0000-cover-letter.patch

@@ -0,0 +1,194 @@
+From fba32f3502243c3f1bcb967667f6618897b8bf80 Mon Sep 17 00:00:00 2001
+From: Wentao Zhang <[email protected]>
+Date: Tue, 8 Oct 2024 16:51:11 -0500
+Subject: [PATCH v3 0/5] Enable measuring the kernel's Source-based Code Coverage and MC/DC with Clang
+
+This series adds support for building x86-64 kernels with Clang's Source-
+based Code Coverage[1] in order to facilitate Modified Condition/Decision
+Coverage (MC/DC)[2] that provably correlates to source code for all levels
+of compiler optimization.
+
+The newly added kernel/llvm-cov/ directory complements the existing gcov
+implementation. Gcov works at the object code level which may better
+reflect actual execution. However, Gcov lacks the necessary information to
+correlate coverage measurement with source code location when compiler
+optimization level is non-zero (which is the default when building the
+kernel). In addition, gcov reports are occasionally ambiguous when
+attempting to compare with source code level developer intent.
+
+In the following gcov example from drivers/firmware/dmi_scan.c, an
+expression with four conditions is reported to have six branch outcomes,
+which is not ideally informative in many safety related use cases, such as
+automotive, medical, and aerospace.
+
+        5: 1068:	if (s == e || *e != '/' || !month || month > 12) {
+branch  0 taken 5 (fallthrough)
+branch  1 taken 0
+branch  2 taken 5 (fallthrough)
+branch  3 taken 0
+branch  4 taken 0 (fallthrough)
+branch  5 taken 5
+
+On the other hand, Clang's Source-based Code Coverage instruments at the
+compiler frontend which maintains an accurate mapping from coverage
+measurement to source code location. Coverage reports reflect exactly how
+the code is written regardless of optimization and can present advanced
+metrics like branch coverage and MC/DC in a clearer way. Coverage report
+for the same snippet by llvm-cov would look as follows:
+
+ 1068|      5|	if (s == e || *e != '/' || !month || month > 12) {
+  ------------------
+  |  Branch (1068:6): [True: 0, False: 5]
+  |  Branch (1068:16): [True: 0, False: 5]
+  |  Branch (1068:29): [True: 0, False: 5]
+  |  Branch (1068:39): [True: 0, False: 5]
+  ------------------
+
+Clang has added MC/DC support as of its 18.1.0 release. MC/DC is a fine-
+grained coverage metric required by many automotive and aviation industrial
+standards for certifying mission-critical software [3].
+
+In the following example from arch/x86/events/probe.c, llvm-cov gives the
+MC/DC measurement for the compound logic decision at line 43.
+
+   43|     12|			if (msr[bit].test && !msr[bit].test(bit, data))
+  ------------------
+  |---> MC/DC Decision Region (43:8) to (43:50)
+  |
+  |  Number of Conditions: 2
+  |     Condition C1 --> (43:8)
+  |     Condition C2 --> (43:25)
+  |
+  |  Executed MC/DC Test Vectors:
+  |
+  |     C1, C2    Result
+  |  1 { T,  F  = F      }
+  |  2 { T,  T  = T      }
+  |
+  |  C1-Pair: not covered
+  |  C2-Pair: covered: (1,2)
+  |  MC/DC Coverage for Decision: 50.00%
+  |
+  ------------------
+   44|      5|				continue;
+
+As the results suggest, during the span of measurement, only condition C2
+(!msr[bit].test(bit, data)) is covered. That means C2 was evaluated to both
+true and false, and in those test vectors C2 affected the decision outcome
+independently. Therefore MC/DC for this decision is 1 out of 2 (50.00%).
+
+To do a full kernel measurement, instrument the kernel with
+LLVM_COV_KERNEL_MCDC enabled, and optionally set a
+LLVM_COV_KERNEL_MCDC_MAX_CONDITIONS value (the default is six). Run the
+testsuites, and collect the raw profile data under
+/sys/kernel/debug/llvm-cov/profraw. Such raw profile data can be merged and
+indexed, and used for generating coverage reports in various formats.
+
+  $ cp /sys/kernel/debug/llvm-cov/profraw vmlinux.profraw
+  $ llvm-profdata merge vmlinux.profraw -o vmlinux.profdata
+  $ llvm-cov show --show-mcdc --show-mcdc-summary                         \
+             --format=text --use-color=false -output-dir=coverage_reports \
+             -instr-profile vmlinux.profdata vmlinux
+
+The first two patches enable the llvm-cov infrastructure, where the first
+enables source based code coverage and the second adds MC/DC support. The
+next patch disables instrumentation for curve25519-x86_64.c for the same
+reason as gcov. The final patch enables coverage for x86-64.
+
+The choice to use a new Makefile variable LLVM_COV_PROFILE, instead of
+reusing GCOV_PROFILE, was deliberate. More work needs to be done to
+determine if it is appropriate to reuse the same flag. In addition, given
+the fundamentally different approaches to instrumentation and the resulting
+variation in coverage reports, there is a strong likelihood that coverage
+type will need to be managed separately.
+
+This work reuses code from a previous effort by Sami Tolvanen et al. [4].
+Our aim is for source-based *code coverage* required for high assurance
+(MC/DC) while [4] focused more on performance optimization.
+
+This initial submission is restricted to x86-64. Support for other
+architectures would need a bit more Makefile & linker script modification.
+Informally we've confirmed that arm64 works and more are being tested.
+
+Note that Source-based Code Coverage is Clang-specific and isn't compatible
+with Clang's gcov support in kernel/gcov/. Currently, kernel/gcov/ is not
+able to measure MC/DC without modifying CFLAGS_GCOV and it would face the
+same issues in terms of source correlation as gcov in general does.
+
+Some demo and results can be found in [5]. We will talk about this patch
+series in the Refereed Track at LPC 2024 [6].
+
+Known Limitations:
+
+Kernel code with logical expressions exceeding
+LVM_COV_KERNEL_MCDC_MAX_CONDITIONS will produce a compiler warning.
+Expressions with up to 47 conditions are found in the Linux kernel source
+tree (as of v6.11), but 46 seems to be the max value before the build fails
+due to kernel size. As of LLVM 19 the max number of conditions possible is
+32767.
+
+As of LLVM 19, certain expressions are still not covered, and will produce
+build warnings when they are encountered:
+
+"[...] if a boolean expression is embedded in the nest of another boolean
+ expression but separated by a non-logical operator, this is also not
+ supported. For example, in x = (a && b && c && func(d && f)), the d && f
+ case starts a new boolean expression that is separated from the other
+ conditions by the operator func(). When this is encountered, a warning
+ will be generated and the boolean expression will not be
+ instrumented." [7]
+
+
+[1] https://clang.llvm.org/docs/SourceBasedCodeCoverage.html
+[2] https://en.wikipedia.org/wiki/Modified_condition%2Fdecision_coverage
+[3] https://digital-library.theiet.org/content/journals/10.1049/sej.1994.0025
+[4] https://lore.kernel.org/lkml/[email protected]/
+[5] https://github.com/xlab-uiuc/linux-mcdc
+[6] https://lpc.events/event/18/contributions/1718/
+[7] https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#mc-dc-instrumentation
+
+---
+v2 -> v3:
+
+* Rebased onto v6.12-rc2 from v6.11-rc6.
+* llvm-cov:
+  * Improve documentation regarding LLVM raw profile version.
+  * Remove license boilerplate text.
+* Kconfig:
+  * Make LLVM_COV_KERNEL and GCOV_KERNEL exclusive.
+  * Improve documentation for ARCH_HAS_LLVM_COV,
+    ARCH_HAS_LLVM_COV_PROFILE_ALL and LLVM_COV_KERNEL_MCDC_MAX_CONDITIONS.
+  * Remove "default n".
+
+---
+
+Wentao Zhang (5):
+  llvm-cov: add Clang's Source-based Code Coverage support
+  llvm-cov: add Clang's MC/DC support
+  x86: disable llvm-cov instrumentation
+  x86: enable llvm-cov support
+  MAINTAINERS: add "LLVM-COV BASED KERNEL PROFILING" entry
+
+ MAINTAINERS                       |   6 +
+ Makefile                          |   9 ++
+ arch/Kconfig                      |   1 +
+ arch/x86/Kconfig                  |   2 +
+ arch/x86/crypto/Makefile          |   3 +-
+ arch/x86/kernel/vmlinux.lds.S     |   2 +
+ include/asm-generic/vmlinux.lds.h |  36 +++++
+ kernel/Makefile                   |   1 +
+ kernel/llvm-cov/Kconfig           | 118 +++++++++++++++
+ kernel/llvm-cov/Makefile          |   4 +
+ kernel/llvm-cov/fs.c              | 243 ++++++++++++++++++++++++++++++
+ kernel/llvm-cov/llvm-cov.h        | 159 +++++++++++++++++++
+ scripts/Makefile.lib              |  23 +++
+ scripts/mod/modpost.c             |   2 +
+ 14 files changed, 608 insertions(+), 1 deletion(-)
+ create mode 100644 kernel/llvm-cov/Kconfig
+ create mode 100644 kernel/llvm-cov/Makefile
+ create mode 100644 kernel/llvm-cov/fs.c
+ create mode 100644 kernel/llvm-cov/llvm-cov.h
+
+-- 
+2.45.2
+