Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-35596: [C++][CI] Improve compilation caching with PCG #35597

Merged
merged 1 commit into from
May 16, 2023

Conversation

pitrou
Copy link
Member

@pitrou pitrou commented May 15, 2023

Rationale for this change

The PCG headers use the __DATE__ and __TIME__ macros, which makes builds non-deterministic:
imneme/pcg-cpp#59

Deterministic builds are useful for a number of reasons. One is security audits of binary artifacts, another (that directly affects us) is making compilation caching as efficient as possible. Preprocessor-based compilation caching breaks when time-dependent macros are used.

What changes are included in this PR?

Remove the struct static_arbitrary_seed construct from PCG.
Also, enable a gcc/clang warning that detects the use of non-deterministic preprocessor macros.

Are these changes tested?

Yes, by the additional warning (which is turned into an error in "checkin" warnings mode).

Are there any user-facing changes?

No.

Also warn/error out if non-deterministic compilation macros such as `__TIME__` are used (with gcc and clang at least).
@github-actions
Copy link

@github-actions github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels May 15, 2023
Copy link
Member

@assignUser assignUser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, it seems this is not a feature used in arrow (or it's development)?

@pitrou
Copy link
Member Author

pitrou commented May 16, 2023

The static_arbitrary_seed feature seems rather useless as it creates a rather bad-quality seed.

@pitrou pitrou merged commit f6e4479 into apache:main May 16, 2023
@pitrou pitrou deleted the gh-35596-pcg-ccache branch May 16, 2023 08:13
@ursabot
Copy link

ursabot commented May 16, 2023

Benchmark runs are scheduled for baseline = 8be70c1 and contender = f6e4479. f6e4479 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.2% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.51% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.45% ⬆️0.06%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] f6e44794 ec2-t3-xlarge-us-east-2
[Finished] f6e44794 test-mac-arm
[Finished] f6e44794 ursa-i9-9960x
[Finished] f6e44794 ursa-thinkcentre-m75q
[Finished] 8be70c13 ec2-t3-xlarge-us-east-2
[Finished] 8be70c13 test-mac-arm
[Finished] 8be70c13 ursa-i9-9960x
[Finished] 8be70c13 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@ursabot
Copy link

ursabot commented May 16, 2023

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

rtpsw pushed a commit to rtpsw/arrow that referenced this pull request May 16, 2023
…e#35597)

### Rationale for this change

The PCG headers use the `__DATE__` and `__TIME__` macros, which makes builds non-deterministic:
imneme/pcg-cpp#59

Deterministic builds are useful for a number of reasons. One is security audits of binary artifacts, another (that directly affects us) is making compilation caching as efficient as possible. Preprocessor-based compilation caching breaks when time-dependent macros are used.

### What changes are included in this PR?

Remove the `struct static_arbitrary_seed` construct from PCG.
Also, enable a gcc/clang warning that detects the use of non-deterministic preprocessor macros.

### Are these changes tested?

Yes, by the additional warning (which is turned into an error in "checkin" warnings mode).

### Are there any user-facing changes?

No.

* Closes: apache#35596

Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[C++] PCG breaks compilation caching
4 participants