Nix + macOS codesigning: -f flag needed

[anmonteiro]

    The error I get without `-f` is:

ocaml5.0.0+alpha1-conan-cli> libc++abi: terminating with uncaught exception of type std::runtime_error: file is already signed. pass -f to sign regardless.

Originally posted by @anmonteiro in #6260 (comment)

[emillon]

Thanks! Can you see this error in the test suite? If not, I'd be interested in a repro case if you can.

[anmonteiro]

You can reproduce it with any already signed binary:

$ nix shell nixpkgs#darwin.sigtool
$ command -v codesign
/nix/store/31x5zrydp6vygqx218hipi81c80n7lc1-sigtool-0.1.2/bin/codesign
$ cp /usr/bin/file ./random-file
$ codesign -s - ./random-file
libc++abi: terminating with uncaught exception of type std::runtime_error: file is already signed. pass -f to sign regardless.
[1]    25473 abort      codesign -s - ./random-file
$ codesign -s - -f ./random-file

[emillon]

I meant in the context of a dune build.

[anmonteiro]

Created a repro here (https://github.com/anmonteiro/dune-repro). Just using dune-site triggers the issue, as I assume dune does substitutions there for sourceroot.

Happy to walk you through how to modify / set up a new test version. Should be as easy as modifying the dune_3 src in flake.nix: https://github.com/anmonteiro/dune-repro/blob/6c3e94c8d976c35238acf6cbac133c01728cdfeb/flake.nix#L14-L18

You can probably add /Users/username/path/to/dune instead, and try a local development version. You might just need to pass --impure for nix but it'll warn you.

[emillon]

I don't have access to a M1 machine at the moment (should be the case later) so I won't be able to fix that for 3.6 but that should be doable for 3.7.

Are you fine with patching in the meantime @anmonteiro ?

(For context, we can't just add -f to the flags in dune because this adds different output in vanilla macos and in macos+nix.)

[rgrinberg]

(For context, we can't just add -f to the flags in dune because this adds different output in vanilla macos and in macos+nix.)

Why is that an issue?

[emillon]

The test suite is going to fail on one and succeed on the other, so we need some kind of postprocessing if we go that route.
The alternative is making sure we never call codesign on an already signed binary, which I thought was the case.

[rgrinberg]

The test suite already fails on nixos for many other unrelated reasons.

[rgrinberg]

Btw, which test case do you have in mind?

[emillon]

I don't have a mac machine to test at the moment, but (in the non-nix case) if you add -f to the list of flags, there are some places where codesign output appears.

[rgrinberg]

Okay, then I think basically both options seem similar. Either the nix distribution patches the binary or the output of the tests. @anmonteiro what do you prefer?

[emillon]

Part of the issue seems that -f seems to have different semantics on the two codesign tools. When we call codesign in dune, the binary already has a signature, so I'm not sure why -f is not necessary and there's no "replacing a signature" log output.

[anmonteiro]

I'm happy to continue patching on my end, but might be a little selfish, as only folks using my overlays will get the fix.

[emillon]

Great! I do want to solve this for everybody but I don't have the hardware to make it happen for 3.6, but that should be fine for 3.7. Thanks!

[emillon]

We're going to have a look at this with @voodoos soon.

[greedy]

This already causes a failure in the dune test suite on macos+nix in promote-only-when-needed:

$ dune build @promote-only-when-needed
File "test/blackbox-tests/test-cases/promote/promote-only-when-needed.t", line 1, characters 0-0:
------ test/blackbox-tests/test-cases/promote/promote-only-when-needed.t
++++++ test/blackbox-tests/test-cases/promote/promote-only-when-needed.t.corrected
File "test/blackbox-tests/test-cases/promote/promote-only-when-needed.t", line 52, characters 0-1:
 |  > (executable
 |  >  (name hello)
 |  >  (libraries dune-build-info)
 |  >  (promote))
 |  > EOF
 |
 |  $ cat >hello.ml <<EOF
 |  > print_endline "Hello, World!";
 |  > (match Build_info.V1.version () with
 |  >   | Some _ -> print_endline "Has version info"
 |  >   | None -> print_endline "Has no version info")
 |  > EOF
 |
 |  $ dune build hello.exe --verbose 2>&1 | grep "Promoting"
 |  Promoting "_build/default/hello.exe" to "hello.exe"
 |  $ ./hello.exe
-|  Hello, World!
-|  Has version info
+|  ./hello.exe: No such file or directory
+|  [127]
 |
 |Bug: Dune currently re-promotes versioned executables on every restart.
 |
 |# CR-someday amokhov: Fix this.
 |
 |  $ dune build hello.exe --verbose 2>&1 | grep "Promoting"
 |  Promoting "_build/default/hello.exe" to "hello.exe"

The grep "Promoting" is hiding a codesign error that fails the build:

File "dune", line 2, characters 7-12:
2 |  (name hello)
           ^^^^^
.#hello.exe.dune-temp: is already signed