Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build missing sunos packages / The case of the wrong sha's #586

Closed
3 of 11 tasks
MylesBorins opened this issue Jan 5, 2017 · 16 comments
Closed
3 of 11 tasks

Build missing sunos packages / The case of the wrong sha's #586

MylesBorins opened this issue Jan 5, 2017 · 16 comments

Comments

@MylesBorins
Copy link
Contributor

MylesBorins commented Jan 5, 2017 •
edited
Loading

These need to be built and promoted
@MylesBorins
Copy link
Contributor Author

MylesBorins commented Jan 5, 2017 •
edited
Loading

doodle... I just compiled and promoted the v4 and v6 releases. I figured that the build process would be deterministic and didn't check the sha's when I promoted the assets. The result is that a bunch of the promoted releases now have different sha's then the original release.

@jbergstroem do we have a way to roll back the release server? @rvagg / @nodejs/ctc is this an issue? Should we just keep the new shas? I'm a bit concerned about there now being different releases with the same sha's in the wild.

Sorry about making this mess

edit: we can see the original sha's in the nodejs.org blog releases

6.9.3 before

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

c55e35ccf71f868d6b7075f20c14b9d0c2c8a3ca98db823b0c5a83ba09e5a984  node-v6.9.3-aix-ppc64.tar.gz
ae79277f15b8b2f173b97e44e2d4c65a8de4254c2c7da0dcd754b4e39658a779  node-v6.9.3-darwin-x64.tar.gz
239c196ab56ee875ad300159cfc3f5bd0a87ce457961046ce9518868a983d618  node-v6.9.3-darwin-x64.tar.xz
43764ddd3829cd3ad22b1e6870fd7d058e2f9a2cf3fd3ea21a25772a18fe0a88  node-v6.9.3-headers.tar.gz
d0cd948b6d585f64e6ef9ba61c1ee6b3c703670f0bad04e613ad6914b011151f  node-v6.9.3-headers.tar.xz
f9eaf8dbd926770795ec5a5670a824bab25ec5b19c9803584c342777daef272c  node-v6.9.3-linux-arm64.tar.gz
fc461a64ef0d2f6267436e95f966df8673276a6344c9389d41cfa06da07ab878  node-v6.9.3-linux-arm64.tar.xz
5247665cac023be266cdff06abff0f784f0b5d737edff7dcabb12ceb115cdb36  node-v6.9.3-linux-armv6l.tar.gz
27941dcafa8d9cb0529f1b88831fc40837118a6471410cfca77fa42c5d57415e  node-v6.9.3-linux-armv6l.tar.xz
01793465bd7ddd6cdb798799c5e4ee107fd6dc77e013bfb602d9f677395d9465  node-v6.9.3-linux-armv7l.tar.gz
30fa10c799db76732998913a2195f45041f5c2417800740c43cee9b7dcaf7b33  node-v6.9.3-linux-armv7l.tar.xz
15fceb4cac03ea4cfa54e202d39ac260aec21575057029a0b5b21463ee88c683  node-v6.9.3-linux-ppc64le.tar.gz
0dc6cf753cc25f14f2f310d22d40dbc8c273dce38a9776b430bac319cb27ac6b  node-v6.9.3-linux-ppc64le.tar.xz
3ccf0fa6543714e1a745648fea9ab86e0566599e3618d578e9009835bfcc79a3  node-v6.9.3-linux-ppc64.tar.gz
ce25234a057f1c0a744bfcfafd2f6d0fc78bc554451c422ec3220c7f1e755d7d  node-v6.9.3-linux-ppc64.tar.xz
8d387365cd3a7c56c5f603561458cb303351803cea3c409b7ff14a2f88bebf40  node-v6.9.3-linux-s390x.tar.gz
c3ea05fd0a7e2216d699da8fda2223538f3b5ecc88fbdb5cdcefe10e4439bf19  node-v6.9.3-linux-s390x.tar.xz
89fbe01f6ccba0295a121ca32e3f0da772319406a8dba5f63e4797a4df6cb5ad  node-v6.9.3-linux-x64.tar.gz
7e60f6f54a836ab8346d0dc60f8c35522a839872084e76acba892920502392d2  node-v6.9.3-linux-x64.tar.xz
5e5e95f47a71eb3316ff4aa520f5174f622196eb591d11a0948314dc211d0e0e  node-v6.9.3-linux-x86.tar.gz
514ff425fd85179c8c065eb7c44c37416d5a80b2e6b87d3a1dcb496616cbc42e  node-v6.9.3-linux-x86.tar.xz
b2898e8261a28df40d640672ee4fb61b4e46b4b87d601c863d2003ff97b5230b  node-v6.9.3.pkg
a29cab343c4695c6609a80503b9a1fcab12952c1632f821f6d7a5851dadc6549  node-v6.9.3-sunos-x86.tar.gz
47e38e6ade9c300f003b28873163e193f76b5137dc9ef6fbf31b6ff7fd72fcee  node-v6.9.3-sunos-x86.tar.xz
5abdc3b77e011d664e13d74cab130680a8652b5cd23a63d2a17496d91399d5b5  node-v6.9.3.tar.gz
98ea92695e9df27c8a2719406e0be51967f06c5ed4e0f6ee5f1e8460814d2723  node-v6.9.3.tar.xz
a58c42b95d5359de9c72c01e1c7abf772a294cafb2ac7428011f8b4c99efc868  node-v6.9.3-win-x64.7z
bda76db75bed655b5b8f01022f33ca7e61b8be175b871dcc218bdf84f6403a1f  node-v6.9.3-win-x64.zip
76c13d2814c5bc2dd51cb6e5e49f6da8986a01d6f606e5aae2b024313a01b62b  node-v6.9.3-win-x86.7z
b971e08b0dbe4285f743a2b612065ef187273731f59e10a012143c80f0c7861c  node-v6.9.3-win-x86.zip
791185c3a771350cd1ea62dac0b6ad0958eb7cf1f1f5e67e5c2ae68ca6db1e32  node-v6.9.3-x64.msi
5ead92cfd27d501ca60889cee1fd3c5bfe3b7ddf2e07a7f927cfeae52cfcf94c  node-v6.9.3-x86.msi
47fef39ededc0dcb348689676de82973ea42715367b84f841167c01cb6398725  win-x64/node.exe
4b1586514603e3d4f78374e05b0617681ae7729a8ecef230b57617714cfafab6  win-x64/node.lib
66968b714afe766c4ff7cbee9bcc77b330649a1e6861c9b6b61d35db81d41e5a  win-x64/node_pdb.7z
4363c4f40461fc08b53076984c6910147ebcbbf93ba43f243eb0d51311fa7d0f  win-x64/node_pdb.zip
9ffd3eec57eebb87cb5b5377dd2213d6c00e40bc8ae4cd5a74582a3cfc037556  win-x86/node.exe
1b044b08e8e2a0f78781c85cc5facdabd04c0a8e73fd991ae3e5a3053784c14f  win-x86/node.lib
921953d2ffb57fa6aa53c001e4bcfec2494f7a68400cd3985c8eaa23786247eb  win-x86/node_pdb.7z
ff1e5f746eb731f81f80b75e41c27b104888361e0187d05195a242eaf9358d52  win-x86/node_pdb.zip
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCAAGBQJYbDoeAAoJEJM7AfQLXKlG2QoH/jN2p9Np5CGXlmaJn7hZJEf+
3Z4MIE5uLeoADWsb+QKYQiS4OBQZDoKyivXzctv4EK0q8VlLt4QrSDUdziF3Uh4A
hsyqoM4SjRHBsEJVFTcwuBbZUKOtGSvjkJJ97pFki70kd+R9xXCqHzyPmTVU3hLH
awlPLIQENI1UxbjMXiOz1TOquP0J6PVUONd3Qe3qJ3oLG8e4aPDha75SFE84vvqa
VnsFKMi45ngzt0nBrxW+ASpahPdEGgeJ1IIXYXG9sdCZBR+J0MYYL/r7HGmx6HlS
jwYtJHsz/zd76JKpRnekk1m8VgEA9o7wRyUSI7bepRw2Q67wv3rFGG7IQHlccLU=
=XtXG
-----END PGP SIGNATURE-----

6.9.3 after

ced91ebd70173714117e3d4787036b7047c5000ab28896ac128fbd3607dd99f1  node-v6.9.3-aix-ppc64.tar.gz
ae79277f15b8b2f173b97e44e2d4c65a8de4254c2c7da0dcd754b4e39658a779  node-v6.9.3-darwin-x64.tar.gz
239c196ab56ee875ad300159cfc3f5bd0a87ce457961046ce9518868a983d618  node-v6.9.3-darwin-x64.tar.xz
43764ddd3829cd3ad22b1e6870fd7d058e2f9a2cf3fd3ea21a25772a18fe0a88  node-v6.9.3-headers.tar.gz
d0cd948b6d585f64e6ef9ba61c1ee6b3c703670f0bad04e613ad6914b011151f  node-v6.9.3-headers.tar.xz
2b0aec9caf1afb5b4cb417dafb2701a2a104e669a0dcb2005497c7f636211ed8  node-v6.9.3-linux-arm64.tar.gz
a45a79d03b48704e140a640a45130aa7b84be9e57f070f9aade8448ce07d18dc  node-v6.9.3-linux-arm64.tar.xz
5247665cac023be266cdff06abff0f784f0b5d737edff7dcabb12ceb115cdb36  node-v6.9.3-linux-armv6l.tar.gz
27941dcafa8d9cb0529f1b88831fc40837118a6471410cfca77fa42c5d57415e  node-v6.9.3-linux-armv6l.tar.xz
ba7cec96cf4893ac5eaf2aaf8768cf8e5d7b69ecf25a48e18e8832183fc39e9a  node-v6.9.3-linux-armv7l.tar.gz
d86187fbf2ee4875438cc8fc53506db998398f165227ec162160a6fedef957cf  node-v6.9.3-linux-armv7l.tar.xz
df59c6bd0b2de004fcec8101f2f671107f63a3e29d436eb02c922348a423fcc2  node-v6.9.3-linux-ppc64le.tar.gz
a4d708f07a44a534c85ddc3a35d0557ce9412af3ad2a16373a871cea104942fd  node-v6.9.3-linux-ppc64le.tar.xz
979f1bec0deac73fdaf45c07f121f9ab7fad0f28c1543710851c878c4a14ac12  node-v6.9.3-linux-ppc64.tar.gz
2227f81699effb0ef9582582e5ad40172a1378aa1b1a0ece0a7a67083f94491d  node-v6.9.3-linux-ppc64.tar.xz
3acc43559272d5e2d17cc2eab797cc8646e16daa1655dbbceecccc18bc6cbf5f  node-v6.9.3-linux-s390x.tar.gz
09f428e4598be8d289a4c49e4dad1491335a15527fb6277cd542034b8772fa4d  node-v6.9.3-linux-s390x.tar.xz
5957fd9b65c346f0d0afb1adc8bde98fa04bf613ee51ef9570d287bda73314a2  node-v6.9.3-linux-x64.tar.gz
f072719f5810a0fd8ba1d882eb19e546d54bf675b393b8478ff89b304669876b  node-v6.9.3-linux-x64.tar.xz
fd25af7ac2728d2321417dfaa408c2f29a8fc1a230f9dbda1a43a2345e6be338  node-v6.9.3-linux-x86.tar.gz
5fd85f7e99bff5e3fa23293bb379ff8f0063c6dbf9bc6caa7bcb160edd38d76b  node-v6.9.3-linux-x86.tar.xz
b2898e8261a28df40d640672ee4fb61b4e46b4b87d601c863d2003ff97b5230b  node-v6.9.3.pkg
229b336d9be8ce86a0a3d96ef11333df2e3fd2b57cfc4e54bb2355e0b6215a97  node-v6.9.3-sunos-x64.tar.gz
edfb132cd42a51524a7a5b9970db65ba188667f74cefbadb51a3a44f1263c46f  node-v6.9.3-sunos-x64.tar.xz
36cbeeed50d4539cc650a7373c79f553d5ef44038e6ffe7d296553d1a12b0443  node-v6.9.3-sunos-x86.tar.gz
a97a83f05234cf7a27bbaa6c680795a163496e07f7f801b8773c6f660fe1f51d  node-v6.9.3-sunos-x86.tar.xz
5abdc3b77e011d664e13d74cab130680a8652b5cd23a63d2a17496d91399d5b5  node-v6.9.3.tar.gz
98ea92695e9df27c8a2719406e0be51967f06c5ed4e0f6ee5f1e8460814d2723  node-v6.9.3.tar.xz
abc083584830e01042655c18c9fd178177835da72ace54489163a65613091068  node-v6.9.3-win-x64.7z
a6670c5790052c626e72bc5a0ef983236a54a71de784076784e8565688fcf7fc  node-v6.9.3-win-x64.zip
3613a8d0aa30450b8803d06badd42eaecffb37978ff691e557657a93c87cccf0  node-v6.9.3-win-x86.7z
338c19fbb7d7655fea1ccb65b85299269420ffa1f1b10cc0362d10ce6b7e818e  node-v6.9.3-win-x86.zip
7c4829e708ae9491b8af76c45ac5ab0eeeb04f084527d049b673a054a084bf11  node-v6.9.3-x64.msi
b94eb09987990fc4b5dfa6df16106d23ad8089b26a9b87c0576de98aacbcfb69  node-v6.9.3-x86.msi
2551c19ee4e6c03ce55dd916ee4dfcbcfec3cd71300164abf4be8a06b77c2ea4  win-x64/node.exe
c8ec2fae6968ff9ebf056fa95a42f7edc25146a46c1f70429d76d6acb4474a0f  win-x64/node.lib
18d84d29c7472bdc3974490e7dfcc7d6a4fd50a9d7a126351a13fa644d453d8f  win-x64/node_pdb.7z
83503d437d7869575bb69c02203b29871a4a51301bf8118aa391ed732a56b114  win-x64/node_pdb.zip
4d803cde8e701c67c434b05232f2686fa4dd1317a19dacc1e2f76d6219b85feb  win-x86/node.exe
54b43a3f880a175d75faead2a20a4a56103ba29ab6d325092d835cc480dc5b63  win-x86/node.lib
a672dcf5206896ae1925273173c5ef7d4b7aec39dcb98929170b0e68a03aeafa  win-x86/node_pdb.7z
2db6c76614c56b4b43ad4ffc47b9bdab5e6fc548358eeff877e193e54c751ee9  win-x86/node_pdb.zip

@MylesBorins
Copy link
Contributor Author

discussed with @rvagg, he is going to attempt to recover the promoted releases from backup.

@mscdex mscdex mentioned this issue Jan 5, 2017
Closed
@rvagg
Copy link
Member

rvagg commented Jan 5, 2017

OK, I could only find valid backups for v4.6.2 v4.7.0 and v6.9.2 in /backup/static/dist/nodejs/release on the backup server. @jbergstroem is this because the backup script hasn't run since the newest two were out, today/yesterday? Is there an incremental for dist or only this? Is it just because I caught it before the backup ran again that I happened to catch the old versions of these files (mtimes on them put them earlier than today and SHASUMS indicates they are the original—compared to blog—I haven't shasumed them myself). Is there anywhere else that we might find original copies of v4.7.1 and v6.9.3 other than in ci-release.nodejs.org jobs as artifacts?

What I've done with v4.6.2 v4.7.0 and v6.9.2 is the following:

  • Copied the entire dist/release directories for these versions into /home/staging/nodejs/release/ on the staging/web server
  • Copied the sunos-x64 binaries from the new builds into the same staging directories
  • Removed the SHASUMS files
  • Ran for i in $(ls); do touch ${i}.done; done to make the .done flag files that tell the promote script that they are ready to be promoted.
  • Changed ownership back to staging.staging

So, @MylesBorins, they are in staging now for you to re-promote. If you run tools/release.sh and promote v4.6.2 v4.7.0 and v6.9.2, all of the files that are in staging will be moved over to dist and you'll get to make and sign a new SHASUMS file which will have the original shasums plus the new sunos-x64 shasums and be signed by you. You should probably then update the blog posts with the new SHASUMS content which has the sunos-x64 files.

For the other builds, we have two ways to add just the sunos-x64 binaries:

  1. Run the builds on ci-release and cancel everything but the sunos-x64 builders. Then you should get just the files you want in staging and when you promote those it'll just be additive.
  2. Run the builds as normal on ci-release but then on the server manually remove everything but sunos-x64 in staging. I don't believe anyone but me has ever fiddled around in staging before but technically everyone on the release crew can do it as you have ssh access as [email protected] and the files are located in /home/staging/nodejs/release/vX.Y.Z/. It's probably too intimidating though so I'm happy to step in and do this bit for you when the time comes.

And fwiw, the original releasers don't need to trigger the builds on ci-release, they only need to promote so they can sign. Any one of us who has ci-release access can trigger them at any time and they'll just wait in staging for the appropriate person, so no need for complex people-coordination to make this happen.

@ghost ghost mentioned this issue Jan 5, 2017
Closed
@rvagg
Copy link
Member

rvagg commented Jan 5, 2017

If we can't recover the original v4.7.1 and v6.9.3 then my vote would be to simply update the blog posts with the new SHASUMS rather than mess around with pulling them out individually from Jenkins. The problem's we're having are simply about discrepancies which we can fix with brute force on the blog rather than attempting to roll back.

@jbergstroem
Copy link
Member

@rvagg said:
is this because the backup script hasn't run since the newest two were out, today/yesterday? Is there an incremental for dist or only this?

Unfortunately (in this case), we don't increment these: https://github.com/nodejs/build/blob/master/backup/backup_scripts/dist.sh

@rvagg said:
If we can't recover the original v4.7.1 and v6.9.3 then my vote would be to simply update the blog posts with the new SHASUMS rather than mess around with pulling them out individually from Jenkins. The problem's we're having are simply about discrepancies which we can fix with brute force on the blog rather than attempting to roll back.

This is a big no-no for me. The blog posts are not the only place where these checksums are stored. Every package manager that's been updated have probably stored their own sum. The only way forward are version bumps. Each artifact at release needs to be considered pristine.

I think this also should sink into the workflow we do with releases. I'm not quite sure why these changed between the runs but one way of handling this could be making files read only on the dist server?

@jbergstroem
Copy link
Member

Case in point:

[08:25:23]  <damongant>	we've got a little security/release issue over in #node.js if a dev could take a look
[08:25:48]  <damongant>	checksums for 6.9.2 changed
[08:26:03]  <damongant>	and broke some builds
[08:26:16]  <evanlucas>	damongant changed for what binaries?
[08:26:31]  <damongant>	evanlucas, source package, contents are the same tho
[08:26:46]  <damongant>	timestamp on the tar.xz is different
[08:27:22]  <damongant>	https://aur.archlinux.org/packages/nodejs-lts-boron/ see the first comment, the sha256 I packaged against is still in the 6.9.2 announcement though

Doing another minor is probably the simplest way for most parties?

@gibfahn
Copy link
Member

gibfahn commented Jan 5, 2017

Doing another minor is probably the simplest way for most parties?

@jbergstroem do you mean a v4.7.2 and v6.9.4 (patch)?

@jbergstroem
Copy link
Member

@gibfahn yeah.

@gibfahn
Copy link
Member

gibfahn commented Jan 5, 2017

Makes sense to me (but I don't have to do the release 😉 ). If we're going to do it the sooner the better though right?

@MylesBorins
Copy link
Contributor Author

So I just attempted to promote the release and found that the permissions are set incorrectly and I cannot promote the builds. They are currently set to 755 rather than 775, and as such the build cannot be promoted

@MylesBorins MylesBorins changed the title Build missing sunos packages Build missing sunos packages / The case of the wrong sha's Jan 5, 2017
@MylesBorins
Copy link
Contributor Author

Fixed the ones we had backups of. v4.7.1 and v6.9.3 still have the wrong shas

@MylesBorins
Copy link
Contributor Author

I've proposed new releases v4.7.2 and v6.9.4 that are simply semver patch bumps to remove ambiguits with the release

nodejs/node#10639
nodejs/node#10640

Please chime in those issues on thoughts about the special releases

@MylesBorins MylesBorins mentioned this issue Jan 5, 2017
Closed
@rvagg
Copy link
Member

rvagg commented Jan 5, 2017

This is a big no-no for me. The blog posts are not the only place where these checksums are stored. Every package manager that's been updated have probably stored their own sum. The only way forward are version bumps. Each artifact at release needs to be considered pristine.

point taken, good call on doing brand new releases to clear things up @MylesBorins

@MylesBorins
Copy link
Contributor Author

v4.7.2 and v6.9.4 have been released to fix this issue. I am going to go through and update the blog posts with the missing shas and information. Should we make an amendment to the v4.7.1 and v6.9.3 posts to explain what happened? Should we include the sha's that are now on the server?

@chorrell chorrell mentioned this issue Jan 6, 2017
Merged
@febLey febLey mentioned this issue Jan 6, 2017
Closed
4 tasks
@gibfahn
Copy link
Member

gibfahn commented Jan 6, 2017 •
edited
Loading

+1 to updating the v4.7.1 and v6.9.3 blog posts to point people to the later version and including both SHAs they might have.

@chorrell chorrell mentioned this issue Jan 6, 2017
Merged
@bnoordhuis bnoordhuis mentioned this issue Feb 22, 2017
Closed
@MylesBorins
Copy link
Contributor Author

Closing, reopen if we should revisit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jbergstroem @rvagg @MylesBorins @gibfahn