Rust nightlies published to the same archive date two nights in a row, broke everything

[brson]

These two nightlies published to the same archive (04-17) twice in a row:

• http://buildbot.rust-lang.org/builders/nightly-dist-rustc-linux/builds/494
• http://buildbot.rust-lang.org/builders/nightly-dist-rustc-linux/builds/495

Then the packaging build failed.

This seems to have broken rustup, which is getting checksum failures, and rustup.sh (which is downloading the wrong files).

cc @alexcrichton

I've started rebuilding the rust-packaging build, but seems like we should have some more validation that we don't reuse archive dates.

[brson]

rust-packaging is failing because arm-unknown-linux-gnueabi doesn't exist in channel-rustc-nightly, and indeed you can see that in the recent rustc build there was no rustc for that architecture.

[brson]

Whereas there were the previous night.

[brson]

I think there are at least two fixes that need to happen here: 1) rust-buildbot shouldn't overwrite files in the archives. Unfortunately it's not obvious how to make s3cmd do this. 2) rust-buildbot should allow fetch-inputs/package-rust to fail for nogate platforms.

[brson]

Hopefully this fixes itself with the nightly that starts in an hour.

[alexcrichton]

Hm I think what may have happened here was:

• The bots were offline at night for 4/15 (for some unknown reason), but the builds were scheduled.
• Noticing this, I restarted buildbot the next day, 4/16
• The builders, taking forever to finish, started a build on 4/16 UTC and finished 4/17 UTC (as midnight UTC is 5pm PST). That is, I restarted the bots in the morning of 4/16 PST (probably ~10am) and ~8 hrs later (the length of the build) it was 6pm 4/16 PST but 4/17 UTC
• The builders then uploaded the 4/16 nightly to the date of 4/17
• The next nightly, on 4/17, was then rightfully uploaded to 4/17 as well.

So... I guess we could do a few things. One is to actually prevent s3cmd from doing this (boy would that be nice). Another would be to always cancel nightly builds whenever buildbot restarts, assuming that the nightly build only works if triggered and completed normally. Finally we could somehow give as input to the build step the nightly date it's building for (e.g. written out by the "trigger") which hopefully only ever runs once a day. I guess we could even extend that to have the trigger detect when it was last run and if it was < 24 hours it bails.

Now as for the ARM build, I have no idea what's going on there.

[alexcrichton]

We could try to use s3cmd's sync feature with --skip-overwrite, but doesn't look to be the same as "upload but error if we overwrite". We'd probably have to script it ourselves to list things, see if any inputs are in the output, and then bail if yes.

[edunham]

Help wanted: Add build step to test whether anything exists at the URL it's about to upload to. https://github.com/rust-lang/rust-buildbot/blob/master/master/master.cfg

[alexcrichton]

No longer a problem that buildbot is gone!