Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bazel Integration #478

Closed
dmadisetti opened this issue May 12, 2021 · 12 comments · May be fixed by #1255
Closed

Bazel Integration #478

dmadisetti opened this issue May 12, 2021 · 12 comments · May be fixed by #1255

Comments

@dmadisetti
Copy link

rules_rust is the canonical bazel integration for rust. It looks like it would be pretty easy to patch something together with cargo_build_script, but keeping paths relative during build looks like it might be a pain.

Maybe it's easier than it looks? It'd just be nice to have a canonical example to reference, or a starlark rule I can blindlessly call.

@adetaylor
Copy link
Collaborator

@martinboehme do you have any advice you can offer here?

@martinboehme
Copy link
Collaborator

If you want to do it right, it's a bit more involved, because you need to use cc_common to get the right command line / environment variables to pass on to the C++ compiler.

I've got a proof of concept that does this in Starlark, and I've been considering submitting it to rules_rust or autocxx, but it still needs some cleaning up and some tests before it's ready for that.

rules_rust is the most central place, but I think it only contains rules for relatively stable / canonical components. autocxx is still pretty experimental, so maybe the autocxx repo would be a better place for the rule to live, at least for now. @adetaylor , what do you think?

@adetaylor
Copy link
Collaborator

I'm certainly open to having bazel rules in the autocxx repo, so long as they are tested in github actions - hopefully much of that can be copied from https://github.com/dtolnay/cxx/blob/master/.github/workflows/ci.yml?

@adetaylor
Copy link
Collaborator

I'm going to close this for now, but I'm certainly happy to accept a pull request for bazel rules in future (on the condition that it's automatically tested on github CI - this is something that has been achieved for cxx, so it's obviously possible)

@jon-chuang
Copy link

jon-chuang commented Nov 25, 2021

I would be quite interested in implementing the autocxx equivalent of cxx's cxxbridge-cmd tool, which dumps the autogenerated C++ headers that can then be consumed by Bazel BUILD.

The use case is leveraging existing Bazel BUILD system in a large C++ codebase.

The pathway I am thinking of for packaging the build in cargo is to first download and install autocxxxbridge-cmd, run autocxxbridge-cmd to dump the headers and cc files, then invoke bazel from the cargo build script via OS env cmdline to build the shared object and archive files. I am not sure how to integrate these object files after the fact into the -sys crate.

As a matter of fact I am not even sure if this pathway is needed, since one is very likely autogenerating rust-side code and possibly no C++ at all for this use case. So do let me know if so, and one can skip the step of needing to build a autocxxbridge-cmd tool.

@jon-chuang
Copy link

jon-chuang commented Nov 25, 2021

Actually, thinking a bit further, this sort of defeats the purpose. The purpose of using bazel is to first resolve, download and build external C++ dependencies. Since neither autocxx nor cxx can function without those dependencies first available, I'm not sure how Bazel can be integrated with cxx/autocxx for them to do their work.

If no C++ codegen is required, this may not be a problem. Bazel can include the C++ libs as a dependency to the Rust crate, for which there are bazel build rules, and which does Rust-side code gen via proc macros.
Edit: this is probably false too, since the proc macros also require resolving the various header files.

If C++ codegen is required, then one might want an additional Bazel rule that can perform the codegen while leveraging Bazel to provide the external libs as dependencies.

@adetaylor
Copy link
Collaborator

Hi, the equivalent of cxxbridge-cmd exists, it's https://crates.io/crates/autocxx-gen/, and it's intended for exactly the same purposes of embedding into build systems like Bazel.

You use it exactly the same way as cxxbridge-cmd except that it also needs to know the C++ include path and any #defines. This is massively annoying to get out of our build system, gn (which is very similar to Bazel in its need for static dependencies everywhere). We had to create a fake C++ toolchain which, instead of having a compiler, ran a python script that recorded the include path and any applicable definitions, such that they could later be retrieved by the script which called autocxx_gen. Hopefully it won't be quite so annoying for Bazel.

Good luck achieving the same for Bazel and I'm happy to answer any questions!

@jon-chuang
Copy link

jon-chuang commented Nov 25, 2021

Thanks @adetaylor ! Is that python script or the tool that leverages the data from cc_common that @martinboehme alluded to available somewhere?

I'd like to see if I can make it work for me and if so, perhaps later on it could be cleaned up for general use, living in this repo.

I think for a test we'd want to at least have an external bazel dependency like boost or absl, to make sure that these dependencies and their paths after resolution can be passed on to the autocxx codegen components.

@jon-chuang
Copy link

jon-chuang commented Nov 26, 2021

Actually, it was thankfully simple to get the include dirs from a bazel rule:
print(ctx.attr.deps[0][CcInfo].compilation_context.includes.to_list())

However, this does not include all header files referenced. Meaning to say, that
[CcInfo].compilation_context.includes does not contain all of the headers in [CcInfo].compilation_context.headers. Which is pretty weird.

@bsilver8192
Copy link
Contributor

Leaving a note here that I'm working on this. I got a simple example working today, going to keep working on it. I'll share a link here when I've got something more fully working.

@bsilver8192
Copy link
Contributor

I've got something that kinda-sorta works. However, I ran into an interesting problem: Bazel toolchains come with flags for a C++ compiler, not necessarily the same as the libclang that autocxx passes flags to. I'm currently using a libclang from most of my C++ toolchains, and the other C++ toolchain is a gcc of similar vintage that accepts most of the same flags, but I'm not sure how to solve this in general. Some ideas:

  1. Require using a matching libclang and Clang-based C++ toolchain.
  2. Allowlist and map recognized flags. This is hard to maintain.
  3. Blocklist and map known-bad flags. This is also hard to maintain.
  4. Require the caller to map flags. This just punts the problem to the user, which isn't very helpful. Maybe some kind of API that allows choosing and extending any of the other options?

Warning flags are also interesting. My toolchain enables -Wunused-private-field, which somehow broke things when it triggered. Maybe the -Werror is taking effect, but only the warnings get printed for some reason? I think giving the user a way to disable any problematic warnings they have is fine, it's hard to do anything else. Or maybe just filter out -W* flags, although that means they can't be disable either.

Anybody have thoughts on how to handle this?

@adetaylor
Copy link
Collaborator

Sympathy, but not a lot in the way of concrete suggestions... I'm fortunate to be working with clang-based toolchains and are (theoretically) able to use a matching libclang.

@nak3 nak3 mentioned this issue Feb 21, 2023
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants