🐥 baby steps towards stable Rust

Started working on #4 (support for stable Rust). First issue we need to
solve is to get access to the harness (since we don't really want to
implement it ourselves).

There is https://crates.io/crates/libtest crate, which is recent version
of Rust internal test harness, extracted as a crate. However, it only
compiles on nightly, so it won't help us here.

There is also https://crates.io/crates/rustc-test, but it is 2 years
old. I haven't checked its features, but might not support some of the
desired functionality (like, JSON output in tests? colored output?).

So, the third option (which I'm using here) is to use `test` crate from
the Rust itself and also set `RUSTC_BOOTSTRAP=1` for our crate so we
can access it on stable channel. Not great, but works for now.

Second issue is to get access to the tests. On nightly, we use
`#[test_case]` to hijack Rust tests registration so we can get access to
them in nightly.

Cannot do that on stable. What would help here is something along the
lines of https://internals.rust-lang.org/t/idea-global-static-variables-extendable-at-compile-time/9879
or https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions.
Don't have that, so we use https://crates.io/crates/ctor crate to build
our own registry of tests, similar to https://crates.io/crates/inventory.

The caveat here is potentially hitting dtolnay/inventory#7
issue which would manifest itself as test being silently ignored. Not
great, but let's see how bad it will be.

Third piece of the puzzle is to intercept execution of tests. This is
done by asking users to use `harness = false` in their `Cargo.toml`, in
which case we take full control of test execution.

Finally, the last challenge is that with `harness = false`, we don't
have a good way to intercept "standard" tests (`#[test]`): https://users.rust-lang.org/t/capturing-test-when-harness-false-in-cargo-toml/28115

So, the plan here is to provide `#[datatest::test]` attribute that will
behave similar to built-in `#[test]` attribute, but will use our own
registry for tests. No need to support `#[bench]` as it is not supported
on stable channel anyway.

The caveat in this case is that if you use built-in `#[test]`, your test
will be silently ignored. Not great, not sure what to do about it.

Proper solution, of course, would be driving RFC for custom test
frameworks: rust-lang/rust#50297 😅

Cargo.toml

@@ -10,6 +10,10 @@ description = """
 Data-driven tests in Rust
 """
 
+[[test]]
+name = "datatest_stable"
+harness = false
+
 [build-dependencies]
 version_check = "0.9.1"
 

build.rs

@@ -8,4 +8,4 @@ fn main() {
         println!("cargo:rustc-cfg=feature=\"stable\"");
     }
     println!("cargo:rustc-env=RUSTC_BOOTSTRAP=1");
-}
+}

datatest-derive/src/lib.rs

@@ -2,19 +2,14 @@
 #![deny(unused_must_use)]
 extern crate proc_macro;
 
-#[macro_use]
-extern crate syn;
-#[macro_use]
-extern crate quote;
-extern crate proc_macro2;
-
 use proc_macro2::{Span, TokenStream};
+use quote::quote;
 use std::collections::HashMap;
 use syn::parse::{Parse, ParseStream, Result as ParseResult};
 use syn::punctuated::Punctuated;
 use syn::spanned::Spanned;
 use syn::token::Comma;
-use syn::{ArgCaptured, FnArg, Ident, ItemFn, Pat};
+use syn::{braced, parse_macro_input, ArgCaptured, FnArg, Ident, ItemFn, Pat};
 
 type Error = syn::parse::Error;
 
@@ -253,14 +248,15 @@ fn files_internal(
     let orig_func_name = &func_item.ident;
 
     let (kind, bencher_param) = if info.bench {
-        (quote!(BenchFn), quote!(bencher: &mut ::datatest::__internal::Bencher,))
+        (
+            quote!(BenchFn),
+            quote!(bencher: &mut ::datatest::__internal::Bencher,),
+        )
     } else {
         (quote!(TestFn), quote!())
     };
 
-    let registration = test_registration(channel);
-
-    // Adding `#[allow(unused_attributes)]` to `#orig_func` to allow `#[ignore]` attribute
+    let registration = test_registration(channel, &desc_ident);
     let output = quote! {
         #registration
         #[automatically_derived]
@@ -347,14 +343,13 @@ impl Parse for DataTestArgs {
     }
 }
 
-
 /// Wrapper that turns on behavior that works on stable Rust.
 #[proc_macro_attribute]
 pub fn data_stable(
     args: proc_macro::TokenStream,
     func: proc_macro::TokenStream,
 ) -> proc_macro::TokenStream {
-    files_internal(args, func, Channel::Stable)
+    data_internal(args, func, Channel::Stable)
 }
 
 /// Wrapper that turns on behavior that works only on nightly Rust.
@@ -374,7 +369,7 @@ fn data_internal(
     let mut func_item = parse_macro_input!(func as ItemFn);
     let cases: DataTestArgs = parse_macro_input!(args as DataTestArgs);
     let cases = match cases {
-        DataTestArgs::Literal(path) => quote!(::datatest::yaml(#path)),
+        DataTestArgs::Literal(path) => quote!(datatest::yaml(#path)),
         DataTestArgs::Expression(expr) => quote!(#expr),
     };
 
@@ -430,8 +425,7 @@ fn data_internal(
         )
     };
 
-    let registration = test_registration(channel);
-
+    let registration = test_registration(channel, &desc_ident);
     let output = quote! {
         #registration
         #[automatically_derived]
@@ -472,7 +466,28 @@ fn data_internal(
     output.into()
 }
 
-
-fn test_registration(channel: Channel) -> TokenStream {
-    quote!(#[test_case])
-}
+fn test_registration(channel: Channel, desc_ident: &syn::Ident) -> TokenStream {
+    match channel {
+        // On nightly, we rely on `custom_test_frameworks` feature
+        Channel::Nightly => quote!(#[test_case]),
+        // On stable, we use `ctor` crate to build a registry of all our tests
+        Channel::Stable => {
+            let registration_fn =
+                syn::Ident::new(&format!("{}__REGISTRATION", desc_ident), desc_ident.span());
+            let tokens = quote! {
+                #[allow(non_snake_case)]
+                #[datatest::__internal::ctor]
+                fn #registration_fn() {
+                    use ::datatest::__internal::RegistrationNode;
+                    static mut REGISTRATION: RegistrationNode = RegistrationNode {
+                        descriptor: &#desc_ident,
+                        next: None,
+                    };
+                    // This runs only once during initialization, so should be safe
+                    ::datatest::__internal::register(unsafe { &mut REGISTRATION });
+                }
+            };
+            tokens
+        }
+    }
+}

src/lib.rs

@@ -115,12 +115,17 @@ mod data;
 mod files;
 mod runner;
 
+/// Internal re-exports for the procedural macro to use.
 #[doc(hidden)]
 pub mod __internal {
     pub use crate::data::{DataBenchFn, DataTestDesc, DataTestFn};
     pub use crate::files::{DeriveArg, FilesTestDesc, FilesTestFn, TakeArg};
     pub use crate::runner::assert_test_result;
     pub use crate::test::Bencher;
+    pub use ctor::ctor;
+
+    // To maintain registry on stable channel
+    pub use crate::runner::{register, RegistrationNode};
 }
 
 pub use crate::runner::runner;

src/runner.rs

@@ -2,6 +2,7 @@ use crate::data::{DataTestDesc, DataTestFn};
 use crate::files::{FilesTestDesc, FilesTestFn};
 use crate::test::{ShouldPanic, TestDesc, TestDescAndFn, TestFn, TestName};
 use std::path::{Path, PathBuf};
+use std::sync::atomic::{AtomicPtr, Ordering};
 
 fn derive_test_name(root: &Path, path: &Path, test_name: &str) -> String {
     let relative = path.strip_prefix(root).unwrap_or_else(|_| {
@@ -217,6 +218,27 @@ fn adjust_for_test_name(opts: &mut crate::test::TestOpts, name: &str) {
     }
 }
 
+pub struct RegistrationNode {
+    pub descriptor: &'static dyn TestDescriptor,
+    pub next: Option<&'static RegistrationNode>,
+}
+
+static REGISTRY: AtomicPtr<RegistrationNode> = AtomicPtr::new(std::ptr::null_mut());
+
+pub fn register(new: &mut RegistrationNode) {
+    let reg = &REGISTRY;
+    let mut current = reg.load(Ordering::SeqCst);
+    loop {
+        let previous = reg.compare_and_swap(current, new, Ordering::SeqCst);
+        if previous == current {
+            new.next = unsafe { previous.as_ref() };
+            return;
+        } else {
+            current = previous;
+        }
+    }
+}
+
 /// Custom test runner. Expands test definitions given in the format our test framework understands
 /// ([DataTestDesc]) into definitions understood by Rust test framework ([TestDescAndFn] structs).
 /// For regular tests, mapping is one-to-one, for our data driven tests, we generate as many
@@ -259,23 +281,14 @@ pub fn runner(tests: &[&dyn TestDescriptor]) {
 
     let mut rendered: Vec<TestDescAndFn> = Vec::new();
     for input in tests.iter() {
-        match input.as_datatest_desc() {
-            DatatestTestDesc::Test(test) => {
-                // Make a copy as we cannot take ownership
-                rendered.push(TestDescAndFn {
-                    desc: test.desc.clone(),
-                    testfn: clone_testfn(&test.testfn),
-                })
-            }
-            DatatestTestDesc::FilesTest(files) => {
-                render_files_test(files, &mut rendered);
-                adjust_for_test_name(&mut opts, &files.name);
-            }
-            DatatestTestDesc::DataTest(data) => {
-                render_data_test(data, &mut rendered);
-                adjust_for_test_name(&mut opts, &data.name);
-            }
-        }
+        render_test_descriptor(*input, &mut opts, &mut rendered);
+    }
+
+    // Gather tests registered via our registry (stable channel)
+    let mut current = unsafe { REGISTRY.load(Ordering::SeqCst).as_ref() };
+    while let Some(node) = current {
+        render_test_descriptor(node.descriptor, &mut opts, &mut rendered);
+        current = node.next;
     }
 
     // Run tests via standard runner!
@@ -286,6 +299,30 @@ pub fn runner(tests: &[&dyn TestDescriptor]) {
     }
 }
 
+fn render_test_descriptor(
+    input: &dyn TestDescriptor,
+    opts: &mut crate::test::TestOpts,
+    rendered: &mut Vec<TestDescAndFn>,
+) {
+    match input.as_datatest_desc() {
+        DatatestTestDesc::Test(test) => {
+            // Make a copy as we cannot take ownership
+            rendered.push(TestDescAndFn {
+                desc: test.desc.clone(),
+                testfn: clone_testfn(&test.testfn),
+            })
+        }
+        DatatestTestDesc::FilesTest(files) => {
+            render_files_test(files, rendered);
+            adjust_for_test_name(opts, &files.name);
+        }
+        DatatestTestDesc::DataTest(data) => {
+            render_data_test(data, rendered);
+            adjust_for_test_name(opts, &data.name);
+        }
+    }
+}
+
 pub trait Termination {
     fn is_success(&self) -> bool;
 }
@@ -296,7 +333,7 @@ impl Termination for () {
     }
 }
 
-impl <T, E> Termination for Result<T, E> {
+impl<T, E> Termination for Result<T, E> {
     fn is_success(&self) -> bool {
         self.is_ok()
     }