Allow running build script separately

[ifsheldon]

Describe the problem you are trying to solve

It seems now the build scripts like build.rs are not necessarily run before checking all codes in the project. That leads to a potential dependency issue. The problem I encountered is that in my code, a macro tries to load a file statically, which is generated in build.rs, then since build.rs is not run at the very beginning, rustc complains, when dealing with this macro, that it cannot find the need file.

The macro I used in main is wgpu::include_spirv!("shaders/shader.spv"), which is to "load a SPIR-V module statically" according to its doc. And in the build script, basically I just compile shader files(like shader.vert) into spv files. The workaround I used is to clean all compile cache and run cargo run directly. In this case, the build script seems to be run first, then everything is fine. But, if I run cargo build without the presence of the needed file, rustc will complain.

I didn't find out any methods that can run build scripts separately or at very least make sure build scripts are run before everything else starts in the cargo book nor on the web nor in the community(see this). Correct me if you know any methods to do so and thanks in advance.

Describe the solution you'd like

Give us a cargo command to run build scripts separately or a cargo flag that ensures build scripts will run first before anything starts.

Notes

[Eh2406]

Build script for a library are supposed to run before that library is built. If you are seeing a different behaver, please make a minimal reproducible example that we can run.

[ifsheldon]

Oh my fault, sorry......... My build script keeps track of the changes of targets but not the built artifacts, so when the built artifacts are accidentally deleted, since the targets haven't been changed, the build script won't run again, then the macro cannot find the artifacts so rustc complains.

But besides that, it seems now we cannot run the build script separately. Can you provide such feature? so that, like in my case, I can run the build script separately to generate the needed artifacts again.

[Eh2406]

I am a little confused, what is the advantage of a manual way to run build scripts over the existing ways to get cargo to run it when needed rerun-if-changed directives?

[ifsheldon]

Well..... In the link I referred, the one who wanted to run build script separately wanted to do so for CI purpose. In my case, the automatic way is good but I also want to run it separately sometimes when build targets are not changed but some dependencies or artifacts have change for some reasons. I'm not proposing discarding the automatic way but giving us a manual way.

[ifsheldon]

Any updates?

I can find another case in which we definitely should have build.rs run manually.

Say I have a program that needs compiled shader bytecodes.

// main.rs
fn main() {
    let s1 = &wgpu::include_spirv!("shaders/1.frag.spv");
    let s2 = &wgpu::include_spirv!("shaders/2.frag.spv");
    // first comment out below, then add a new shader file then uncomment this
    // let s3 = &wgpu::include_spirv!("shaders/3.frag.spv"); 
    println!("Hello, world!");
}

At first I need two compiled shaders, then I decided to use one more by uncommenting let s3 = ...

And my build.rs finds out recursively shader files and compiles them.

// build.rs
use anyhow::*;
use glob::glob;
use std::fs::{read_to_string, write};
use std::path::PathBuf;

struct ShaderData {
    src: String,
    src_path: PathBuf,
    spv_path: PathBuf,
    kind: shaderc::ShaderKind,
}

impl ShaderData {
    pub fn load(src_path: PathBuf) -> Result<Self> {
        let extension = src_path
            .extension()
            .context("File has no extension")?
            .to_str()
            .context("Extension cannot be converted to &str")?;
        let kind = match extension {
            "vert" => shaderc::ShaderKind::Vertex,
            "frag" => shaderc::ShaderKind::Fragment,
            "comp" => shaderc::ShaderKind::Compute,
            _ => bail!("Unsupported shader {}", src_path.display()), // returns an error
        };
        let src = read_to_string(src_path.clone())?;
        let spv_path = src_path.with_extension(format!("{}.spv", extension));
        Ok(Self {
            src,
            src_path,
            spv_path,
            kind,
        })
    }
}

fn main() -> Result<()> {
    let mut shader_paths = [
        glob("./src/**/*.vert")?,
        glob("./src/**/*.frag")?,
        glob("./src/**/*.comp")?,
    ];
    let shaders = shader_paths
        .iter_mut()
        .flatten()
        .map(|glob_result| ShaderData::load(glob_result?))
        .collect::<Vec<Result<_>>>()
        .into_iter()
        .collect::<Result<Vec<_>>>()?;

    let mut compiler = shaderc::Compiler::new().context("Unable to create shader compiler")?;

    for ref shader in shaders {
        // This tells cargo to rerun this script if something in /src/ changes.
        println!(
            "cargo:rerun-if-changed={}",
            shader.src_path.as_os_str().to_str().unwrap()
        );
        let compiled = compiler.compile_into_spirv(
            &shader.src,
            shader.kind.clone(),
            &shader.src_path.to_str().unwrap(),
            "main",
            None,
        )?;
        write(&shader.spv_path, compiled.as_binary_u8());
    }
    Ok(())
}

And the cargo.toml is

[package]
name = "temp_rust"
version = "0.1.0"
authors = ["ifsheldon <[email protected]>"]
edition = "2018"


[dependencies]
wgpu = "0.8"

[build-dependencies]
anyhow = "1.0"
fs_extra = "1.1"
glob = "0.3"
shaderc = "0.7"

Now I run the main with 1.frag and 2.frag in my src folder, everything goes fine. Then I add one more shader file, which is 3.frag, and uncomment the mentioned line above. Then, rustc will complain about something like cannot find 3.frag.spv because none of the two registered shader files have been modified so build.rs won't be rerun to register and compile new files.
So, now I have two workarounds:

1. Rewrite the build.rs and make it register the folder so then any new contents will trigger rerunning build.rs
2. Delete ./target and rebuild everything
   Or, you can just provide us with a manual run on build.rs, which is much simpler and faster for us developer than the above two workarounds when dealing with complex and many build dependencies.

[Eh2406]

1. is not hard to do. shader.src_path => shader.src_path.parent(). to quote the doc linked above "If the path points to a directory, it will scan the entire directory for any modifications."

Cargo makes changes very slowly even when we agree on the design. Don't expect a quick fix.
In the mean time you can use touch build.rs && build. You can even make an alias for it, so that it is available to tows using your project.

[ifsheldon]

touch build.rs && build and aliasing together seem to be a good quick fix for now and thanks!

The quick fix to 1. also works for me, but I think this should not be considered ideal, because maybe developers only cares about changes in (maybe certain types of) files but not the folder, like we may add some temp files to a folder but we don't intend to trigger rerunning build.rs at all.

Or, sometimes, like my shader example, I don't want to trigger rerunning build.rs even when I did changed the registered shader files, maybe I just added some comments or maybe it just a temp change that is not staged yet.

What I meant is change-based rerunning build.rs is a good default but it is not enough for more complicated scenarios.

[ehuss]

I'm going to close this, as it is mostly intended behavior, and this is somewhat a duplicate of #7178. I would encourage you to work with making rerun-if-changed handle the cases you want. Another alternative is to set up a separate command to rebuild the shaders using a workflow similar to how xtask works, and the build script can share the same code with that. Otherwise, I suggest maybe following #7178 for any updates there.