Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[R][Release] Add checksum verification to pre-compiled binaries #37941

Closed
assignUser opened this issue Sep 28, 2023 · 1 comment · Fixed by #38115
Closed

[R][Release] Add checksum verification to pre-compiled binaries #37941

assignUser opened this issue Sep 28, 2023 · 1 comment · Fixed by #38115
Assignees
@assignUser
Labels
Component: R Component: Release Priority: Blocker Marks a blocker for the release Type: enhancement
Milestone
14.0.0

Comments

@assignUser
Copy link
Member

Describe the enhancement requested

We currently don't verify checksums for the pre-compiled binaries we use. We should add that (skip if nightly repo) and update the checksums after the asf release prior to the cran submission.

Component(s)

R, Release
@assignUser assignUser mentioned this issue Sep 28, 2023
Merged
@assignUser assignUser added the Priority: Blocker Marks a blocker for the release label Sep 28, 2023
@assignUser
Copy link
Member Author

I have marked this as a blocker as we want this in place for the CRAN release (the autobrew build also checks checksums so we want parity with that).

paleolimbot pushed a commit that referenced this issue Oct 5, 2023
@assignUser @jonkeane @kou
GH-37923: [R] Move macOS build system to nixlibs.R (#37684)
dd8734d 
This PR modifies the build system of the R package to no longer rely on auto/homebrew. Instead this PR adds the infrastructure and code paths to use the same type of  pre-compiled libarrow binaries as we use for Linux. The main difference is the use of the binaries even on CRAN (as we previously also used binaries in form of brew bottles).

The addition of the new artifacts to tasks.yml should ensure that they get uploaded to the nightly repo as well as to the artifactory during the release (@ kou please confirm). 

A summary of the changes in this PR:
- update `r/configure` and `r/tools/nixlibs.R` to enable the source build on macOS and usage of precompiled binaries using the existing mechanism to test compile a program to detect the exisitng openssl version
- added tests for the changes in nixlibs.R
- update the binary allow-list
- Add the build jobs for libarrow binaries for arm64 and x86_64 macos with openssl 1.1 and 3.0 to the `r-binary-packages` job
- Use the binaries to build the nightly packages  
- bump snappy version to 1.1.10 (and patch it on 10.13) due to build issues with the current version. This also touches on a number of issues in regards to a sanitizer issue we have had for a long time: #32562 #31766
- Disable the centos binary test step: #37922

Follow up issues:
- #37921
- #37941 
- #37945
* Closes: #37923

Lead-authored-by: Jacob Wujciak-Jens <[email protected]>
Co-authored-by: Jonathan Keane <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Dewey Dunnington <[email protected]>
@assignUser assignUser self-assigned this Oct 5, 2023
@github-actions github-actions bot mentioned this issue Oct 7, 2023
Merged
@ianmcook ianmcook added this to the 14.0.0 milestone Oct 10, 2023
thisisnic pushed a commit that referenced this issue Oct 11, 2023
@assignUser
GH-37941: [R][CI][Release] Add checksum verification for pre-compiled…
b20e0ae 
… binaries (#38115)

### Rationale for this change

This change is to restore parity with the previous solution on macOS (brew does cs validation) and improve security for windows and linux. This also align with CRAN policy.

### What changes are included in this PR?

This PR adds a script that can be run after the arrow release (once all files have been pushed to the artifactory) before the CRAN submission to download the checksum files for the pre-compiled binaries which are already added through the usual release. *libs.R have been extended to use these checksum files to validate the downloaded binaries.

### Are these changes tested?

The r-binary-packages nightlies generate checksums and use them when building binary packages, this way the code path is tested. They do not modify the actual src package though.

### Are there any user-facing changes?
no (outside of log messages)
* Closes: #37941

Authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Nic Crane <[email protected]>
@assignUser assignUser mentioned this issue Oct 13, 2023
Merged
JerAguilon pushed a commit to JerAguilon/arrow that referenced this issue Oct 23, 2023
@assignUser @jonkeane @kou
apacheGH-37923: [R] Move macOS build system to nixlibs.R (apache#37684)
433aea0 
This PR modifies the build system of the R package to no longer rely on auto/homebrew. Instead this PR adds the infrastructure and code paths to use the same type of  pre-compiled libarrow binaries as we use for Linux. The main difference is the use of the binaries even on CRAN (as we previously also used binaries in form of brew bottles).

The addition of the new artifacts to tasks.yml should ensure that they get uploaded to the nightly repo as well as to the artifactory during the release (@ kou please confirm). 

A summary of the changes in this PR:
- update `r/configure` and `r/tools/nixlibs.R` to enable the source build on macOS and usage of precompiled binaries using the existing mechanism to test compile a program to detect the exisitng openssl version
- added tests for the changes in nixlibs.R
- update the binary allow-list
- Add the build jobs for libarrow binaries for arm64 and x86_64 macos with openssl 1.1 and 3.0 to the `r-binary-packages` job
- Use the binaries to build the nightly packages  
- bump snappy version to 1.1.10 (and patch it on 10.13) due to build issues with the current version. This also touches on a number of issues in regards to a sanitizer issue we have had for a long time: apache#32562 apache#31766
- Disable the centos binary test step: apache#37922

Follow up issues:
- apache#37921
- apache#37941 
- apache#37945
* Closes: apache#37923

Lead-authored-by: Jacob Wujciak-Jens <[email protected]>
Co-authored-by: Jonathan Keane <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Dewey Dunnington <[email protected]>
JerAguilon pushed a commit to JerAguilon/arrow that referenced this issue Oct 23, 2023
@assignUser
apacheGH-37941: [R][CI][Release] Add checksum verification for pre-co…
ba7b240 
…mpiled binaries (apache#38115)

### Rationale for this change

This change is to restore parity with the previous solution on macOS (brew does cs validation) and improve security for windows and linux. This also align with CRAN policy.

### What changes are included in this PR?

This PR adds a script that can be run after the arrow release (once all files have been pushed to the artifactory) before the CRAN submission to download the checksum files for the pre-compiled binaries which are already added through the usual release. *libs.R have been extended to use these checksum files to validate the downloaded binaries.

### Are these changes tested?

The r-binary-packages nightlies generate checksums and use them when building binary packages, this way the code path is tested. They do not modify the actual src package though.

### Are there any user-facing changes?
no (outside of log messages)
* Closes: apache#37941

Authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Nic Crane <[email protected]>
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
@assignUser @jonkeane
@kou @loicalleyne
apacheGH-37923: [R] Move macOS build system to nixlibs.R (apache#37684)
4503fb0 
This PR modifies the build system of the R package to no longer rely on auto/homebrew. Instead this PR adds the infrastructure and code paths to use the same type of  pre-compiled libarrow binaries as we use for Linux. The main difference is the use of the binaries even on CRAN (as we previously also used binaries in form of brew bottles).

The addition of the new artifacts to tasks.yml should ensure that they get uploaded to the nightly repo as well as to the artifactory during the release (@ kou please confirm). 

A summary of the changes in this PR:
- update `r/configure` and `r/tools/nixlibs.R` to enable the source build on macOS and usage of precompiled binaries using the existing mechanism to test compile a program to detect the exisitng openssl version
- added tests for the changes in nixlibs.R
- update the binary allow-list
- Add the build jobs for libarrow binaries for arm64 and x86_64 macos with openssl 1.1 and 3.0 to the `r-binary-packages` job
- Use the binaries to build the nightly packages  
- bump snappy version to 1.1.10 (and patch it on 10.13) due to build issues with the current version. This also touches on a number of issues in regards to a sanitizer issue we have had for a long time: apache#32562 apache#31766
- Disable the centos binary test step: apache#37922

Follow up issues:
- apache#37921
- apache#37941 
- apache#37945
* Closes: apache#37923

Lead-authored-by: Jacob Wujciak-Jens <[email protected]>
Co-authored-by: Jonathan Keane <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Dewey Dunnington <[email protected]>
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
@assignUser @loicalleyne
apacheGH-37941: [R][CI][Release] Add checksum verification for pre-co…
b708512 
…mpiled binaries (apache#38115)

### Rationale for this change

This change is to restore parity with the previous solution on macOS (brew does cs validation) and improve security for windows and linux. This also align with CRAN policy.

### What changes are included in this PR?

This PR adds a script that can be run after the arrow release (once all files have been pushed to the artifactory) before the CRAN submission to download the checksum files for the pre-compiled binaries which are already added through the usual release. *libs.R have been extended to use these checksum files to validate the downloaded binaries.

### Are these changes tested?

The r-binary-packages nightlies generate checksums and use them when building binary packages, this way the code path is tested. They do not modify the actual src package though.

### Are there any user-facing changes?
no (outside of log messages)
* Closes: apache#37941

Authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Nic Crane <[email protected]>
@assignUser assignUser mentioned this issue Feb 7, 2024
Merged
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@assignUser @jonkeane
@kou @dgreiss
apacheGH-37923: [R] Move macOS build system to nixlibs.R (apache#37684)
7125ca1 
This PR modifies the build system of the R package to no longer rely on auto/homebrew. Instead this PR adds the infrastructure and code paths to use the same type of  pre-compiled libarrow binaries as we use for Linux. The main difference is the use of the binaries even on CRAN (as we previously also used binaries in form of brew bottles).

The addition of the new artifacts to tasks.yml should ensure that they get uploaded to the nightly repo as well as to the artifactory during the release (@ kou please confirm). 

A summary of the changes in this PR:
- update `r/configure` and `r/tools/nixlibs.R` to enable the source build on macOS and usage of precompiled binaries using the existing mechanism to test compile a program to detect the exisitng openssl version
- added tests for the changes in nixlibs.R
- update the binary allow-list
- Add the build jobs for libarrow binaries for arm64 and x86_64 macos with openssl 1.1 and 3.0 to the `r-binary-packages` job
- Use the binaries to build the nightly packages  
- bump snappy version to 1.1.10 (and patch it on 10.13) due to build issues with the current version. This also touches on a number of issues in regards to a sanitizer issue we have had for a long time: apache#32562 apache#31766
- Disable the centos binary test step: apache#37922

Follow up issues:
- apache#37921
- apache#37941 
- apache#37945
* Closes: apache#37923

Lead-authored-by: Jacob Wujciak-Jens <[email protected]>
Co-authored-by: Jonathan Keane <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Dewey Dunnington <[email protected]>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@assignUser @dgreiss
apacheGH-37941: [R][CI][Release] Add checksum verification for pre-co…
73c8558 
…mpiled binaries (apache#38115)

### Rationale for this change

This change is to restore parity with the previous solution on macOS (brew does cs validation) and improve security for windows and linux. This also align with CRAN policy.

### What changes are included in this PR?

This PR adds a script that can be run after the arrow release (once all files have been pushed to the artifactory) before the CRAN submission to download the checksum files for the pre-compiled binaries which are already added through the usual release. *libs.R have been extended to use these checksum files to validate the downloaded binaries.

### Are these changes tested?

The r-binary-packages nightlies generate checksums and use them when building binary packages, this way the code path is tested. They do not modify the actual src package though.

### Are there any user-facing changes?
no (outside of log messages)
* Closes: apache#37941

Authored-by: Jacob Wujciak-Jens <[email protected]>
Signed-off-by: Nic Crane <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@assignUser assignUser

Labels
Component: R Component: Release Priority: Blocker Marks a blocker for the release Type: enhancement
Projects
None yet
Milestone
14.0.0
Development

Successfully merging a pull request may close this issue.

GH-37941: [R][CI][Release] Add checksum verification for pre-compiled binaries assignUser/arrow
2 participants
@ianmcook @assignUser