haskell_cabal_library, cabalopts and --extra-{include,lib}-dirs

[stevana]

I'm trying to build a cabal library that depends on some C headers and shared objects. I managed to get it building with the following rules in my build file:

cc_library(
    name = "z3.h",
    hdrs = ["@z3.dev//:include"],
)

cc_import(
    name = "libz3.so",
    shared_library = "@z3.lib//:lib",
)

haskell_cabal_library(
    name = "z3",
    deps = packages["z3"].deps + [":z3.h", ":libz3.so"],
    srcs = glob([
        "**",
    ]),
    cabalopts = ["--extra-include-dirs=/home/u/src/p/bazel-p/external/z3.dev/include/",
                 "--extra-lib-dirs=/home/u/src/p/bazel-p/external/z3.lib/lib/"],
    version = "408.2",
    visibility = [
        "//visibility:public",
    ],
)

But I cannot figure out how to get rid of the /home/u/src/p/bazel-p/ part (which is specific to my machine) though. I've tried "--extra-include-dirs=$(location :z3.h), $(location @z3.dev//:include) (after adding it to deps) and many variants without any success (bazel complains it cannot find the z3.h file). I've also tried to make the path relative, e.g. ../../external/z3.dev/include, but that makes cabal unhappy as it seems to expect absolute paths.

All examples I've found in your repo seem to depend on .h files that are part of the repo itself, e.g. https://github.com/tweag/rules_haskell/tree/master/examples/vector/include . Is this necessary or is it possible to fix what I've started?

[aherrmann]

cc_library has a strip_include_prefix attribute for that purpose. In recent versions of Bazel it no longer needs the external part in the path, so in your case it would look something like this:

cc_library(
    name = "z3.h",
    hdrs = ["@z3.dev//:include"],
    strip_include_prefix = "/include",
)

If possible, it's often easier to define such a target right inside @z3.dev. For example, in rules_haskell we do this for a Nix provided zlib here. An example that defines the target outside of the external repository exists here.

Side note, cc_import has some short comings compared to cc_library: It doesn't have a strip_include_prefix attribute, and it doesn't handle libraries with versioned sonames correctly. See tweag/rules_nixpkgs#60 and this bazel-discuss thread. You can replace cc_import by cc_library to avoid these issues. In your example that might look like this:

cc_library(
    name = "z3",
    srcs = [@z3.lib//:lib],
    hdrs = ["@z3.dev//:include"],
    strip_include_prefix = "/include",
)

In rules_haskell we import a Nix provided zlib in such a way here.

[stevana]

That worked like a charm, thanks for your quick reply!

[stevana]

@aherrmann While this works for me on linux, it doesn't seem to work on macos:

 * Missing (or bad) C library: z3
 This problem can usually be solved by
installing the system package that provides this library (you may need the
"-dev" version). If the library is already installed but in a non-standard
location then you can use the flags --extra-include-dirs= and --extra-lib-dirs=
to specify where it is.

Tried a bunch of variations, including closely following the zlib example you linked to, but no luck.

Are there any special macos gotchas to keep in mind? (One thing I noticed that seems a bit odd is that the documentation for cc_library's srcs lists .so files but not .dylib?)

[aherrmann]

Indeed, that looks like a missing dynamic library file. Can you share how libz3 is imported/defined?

E.g. with a rules_nixpkgs imported package it's important that the result cc_library target includes the right .dylib file. E.g. that there is no glob(["*.so"]) that doesn't cover .dylib or that the Nix derivation doesn't look too different on MacOS compared to Linux. You can use bazel query --output build to debug which files are included in such targets. E.g. rules_haskell's libz on Linux looks as follows:

$ bazel query @zlib.dev//:zlib --output build
cc_library(
  name = "zlib",
  ...
  srcs = ["@nixpkgs_zlib//:lib"],
  ...
)
$ bazel query --output build @nixpkgs_zlib//:lib
filegroup(
  name = "lib",
  srcs = ["@nixpkgs_zlib//:lib/libz.so", "@nixpkgs_zlib//:lib/libz.so.1", "@nixpkgs_zlib//:lib/libz.so.1.2.11"],
)

A potential gotcha is that when Bazel builds a cc_library dynamic library from source it will use a .so extension on MacOS as well. So, if you're building libz3 from source in Bazel then this may result in a .so file.

Side note, Cabal and GHC generally expect .dylib files on MacOS. However, rules_haskell adds a compatibility layer so that Bazel generated .so files will also be recognized.

[stevana]

Here's a fresh repo, created using your start script with the --use-nix flag passed in. A more recent nixpkgs_git_repository commit is used to avoid the problem in #1475, and a more recent haskell_register_ghc_nixpkgs and the latest rules_haskell commit is also used.

On Linux this build and runs fine using bazel_4 from nixpkgs (same commit as pinned in nixpkgs_git_repository), on MacOS (11.2.2 Big Sur) the error * Missing (or bad) C library: z happens.

Here's some bazel query output:

$ bazel query @zlib.dev//:zlib --output build
# /private/var/tmp/_bazel_u/27f48b7878424df51f952eb5fecdee64/external/zlib.dev/BUILD:8:11
cc_library(
  name = "zlib",
  visibility = ["//visibility:public"],
  tags = ["__CC_RULES_MIGRATION_DO_NOT_USE_WILL_BREAK__"],
  generator_name = "zlib",
  generator_function = "cc_library",
  generator_location = "/private/var/tmp/_bazel_u/27f48b7878424df51f952eb5fecdee64/external/zlib.dev/BUILD:8:11",
  srcs = ["@nixpkgs_zlib//:lib"],
  linkstatic = True,
  hdrs = ["@zlib.dev//:include"],
  strip_include_prefix = "include",
)
# Rule zlib instantiated at (most recent call last):
#   /private/var/tmp/_bazel_u/27f48b7878424df51f952eb5fecdee64/external/zlib.dev/BUILD:8:11        in <toplevel>
#   /private/var/tmp/_bazel_u/27f48b7878424df51f952eb5fecdee64/external/rules_cc/cc/defs.bzl:63:22 in cc_library

Loading: 0 packages loaded

$ bazel query --output build @nixpkgs_zlib//:lib
# /private/var/tmp/_bazel_u/27f48b7878424df51f952eb5fecdee64/external/nixpkgs_zlib/BUILD:8:10
filegroup(
  name = "lib",
  srcs = ["@nixpkgs_zlib//:lib/libz.1.2.11.dylib",
"@nixpkgs_zlib//:lib/libz.1.dylib", "@nixpkgs_zlib//:lib/libz.dylib"],
)
# Rule lib instantiated at (most recent call last):
#   /private/var/tmp/_bazel_u/27f48b7878424df51f952eb5fecdee64/external/nixpkgs_zlib/BUILD:8:10 in <toplevel>

[aherrmann]

@stevana Thank you for sharing the details. I tried to reproduce the issue locally. I don't have a Big Sur machine available right now, I tested this on Mojave instead. In that case I was having issues with GHC 8.10.3 not finding otool, using GHC 8.8.3 instead resolved that issue. I'm not sure if this is relevant to the issue you're encountering.

I ran the build in a pure Nix shell on latest nixpkgs-unstable (21.05pre274251.f5f6dc053b1) with -p bazel_4 -p nix and it passed without the issue you encountered. The queries that you posted match what I'm seeing in my attempts.

I noticed that the WORKSPACE file in your repro is missing nixpkgs_cc_configure. (It's included in the start script on master, but we haven't cut a new release, yet). Could you retry with the following lines added to WORKSPACE?

load(
    "@io_tweag_rules_nixpkgs//nixpkgs:nixpkgs.bzl",
    "nixpkgs_cc_configure",
    ...
)
...
nixpkgs_cc_configure(repository = "@nixpkgs")

[stevana]

Pushed two more commits: one that adds nixpkgs_cc_configure and the other which adds shell.nix. nix-shell followed by bazel build //... on Linux works, while on MacOS:

Setup.hs: Missing dependency on a foreign library:
* Missing (or bad) C library: z
...

[stevana]

Another data point: works on Big Sur 11.1 for another colleague (after downgrading ghc to 8.8.3)

[aherrmann]

@stevana Thank you for the updates.

on MacOS (11.2.2 Big Sur) the error * Missing (or bad) C library: z happens.

Another data point: works on Big Sur 11.1 for another colleague (after downgrading ghc to 8.8.3)

Does it work on MacOS 11.2.2 Big Sur with GHC 8.8.3? I'm wondering whether this is related to the GHC version, or to the OS version.

The start script is also working on our MacOS CI with the fixes that you mentioned, see #1509. The CI nodes are using Catalina, Mac OS X 10.15.7 19H512.

[stevana]

Does it work on MacOS 11.2.2 Big Sur with GHC 8.8.3?

Unfortunately not.

I'm wondering whether this is related to the GHC version, or to the OS version.

I'm confused, I thought since this is from within a nix-shell, calling bazel (from nixpkgs) which is configured to use ghc from nixpkgs via rules_nixpkgs that this should all be hermetic and nicely isolated, but seems something is leaking? Or could it be that the MacOS 11.2.2 Big Sur machine is misconfigured somehow?

[aherrmann]

I'm confused, I thought since this is from within a nix-shell, calling bazel (from nixpkgs) which is configured to use ghc from nixpkgs via rules_nixpkgs that this should all be hermetic and nicely isolated, but seems something is leaking?

There are some components that are imported from the OS on MacOS even inside Nix. E.g. certain system libraries. To my knowledge this is enforced on MacOS and Nix cannot provide these itself in a fully hermetic way. rules_haskell itself also directly uses some of the OS's tools, e.g. otool. So, it's possible that this issue is related to the OS version.

[stevana]

Interesting, would it make sense to try to make rules_haskell use otool from nixpkgs?

Is there anything else we can try/do at the moment? Or should I just reopen the issue, wait to see if anyone else runs into the same problem, and retry later with a more recent version of nixpkgs and macos to see if perhaps the problem has gone away?

[stevana]

Rerunning xcode-select --install seem have fixed the problem!

[aherrmann]

@stevana That's great news! I'm glad the issue is fixed. Thanks for keeping us up to date on this issue!