Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use output dir for empty packages to be hermetic #3098

Merged
merged 2 commits into from
Mar 31, 2022

Conversation

joeljeske
Copy link
Contributor

@joeljeske joeljeske commented Mar 30, 2022

What type of PR is this?

Bug fix

What does this PR do? Why is it needed?

Some packages are not deterministic, rather they have different content hashes when rebuilt. This occurs when rules_go creates an _empty.go file to satisfy the compiler. In those cases, the empty file is created in the OS temp dir, which changes across compiler invocations. The path of this file changes the content hash of the resulting package, thus breaking caches and downstream builds.

This PR changes the location of this tmp file to the output path of the package, such that it is the same output path across compiler invocations.

Reproduction steps:

bazel build //tests/core/go_library:empty 
md5 bazel-bin/tests/core/go_library/empty.a

rm -f  bazel-bin/tests/core/go_library/empty.a

bazel build //tests/core/go_library:empty 
md5 bazel-bin/tests/core/go_library/empty.a

Other notes for review

I noted a broad use of tmpdir or workdir within rules_go, and this issue may be prevalent in other conditions. If determinism is a priority, perhaps an audit of tmp dir uses may be warranted.

It would be useful to have a test that validations reproducibility of this kind, but I'm not aware of a good way to structure that type of test. Reproducibility test has been added

// objects.
// _empty.go needs to be in a deterministic location (not tmpdir) in order
// to ensure deterministic output
emptyPath := filepath.Join(filepath.Dir(outPath), "_empty.go")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joeljeske
Copy link
Contributor Author

There is a failure with windows resolving the empty filepath. Presumably it might be fixed by resolving the absolute path and using that instead of the relative path, but then I'm concerned it would suffer from the same non determinism.

I'm open to suggestions. I also don't have a windows box for testing this

@achew22
Copy link
Member

achew22 commented Mar 30, 2022

@joeljeske, it seems like you're onto something here. WRT writing a test that covers this, I think we can use the preexisting reproducibility_test.go. It would be nice to have a test that failed before merging this in, would. you be willing to take a whack at it?

@joeljeske joeljeske force-pushed the fix-determinisim-empty branch from 50fb24b to f981365 Compare March 31, 2022 16:12
@joeljeske
Copy link
Contributor Author

Fantastic, @achew22, thanks for the tip. I had not seen those tests. I added a simple empty lib test that fails at its commit.

@achew22 achew22 merged commit 4013954 into bazel-contrib:master Mar 31, 2022
@achew22
Copy link
Member

achew22 commented Mar 31, 2022

Great contribution, thank you @joeljeske for helping out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants