x/build: trybots should rebase when testing

[bradfitz]

Currently with the trybot code, if Go's master is A and we have pending CLs B and C on A, currently we test B-on-A and C-on-A, but then when B is merged, we're still testing C on A, instead of C-on-B.

We should do:

https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#rebase-change

... which will add a patch set to the change, rebased. It could get spammy, though. Shawn says "don't complain when you have patch set 85 on a trivial one line change". So maybe we want to be careful when we do it. Maybe only for auto-submit.

[minux]

We can always do git cherry-pick after git fetch the CL in trybot. Any failures in merging is a build failure (the CL won't submit successfully anyway) Even if rebase the CL on gerrit, each time its author updates the CL, the rebased patch set will be overwritten.

[bradfitz]

We could, except that the trybot doesn't use any git right now. It pulls tar.gz files from Gerrit of a commit. We don't even have the "git" command on the builders.

[minux]

Ahh, that makes the problem harder. Could we add a patch primitive to the buildlet and send a patch for it to apply on latest git master tarball? We might be able to reuse golang.org/x/codereview/patch for that. But we need to add fuzzy support to that package.

[bradfitz]

I wouldn't do anything to the buildlet for this. We could apply the patch using a new service in a Docker container under the coordinator, using real git and/or real patch, and then extract the resulting tarball to push to the buildlets, like normal.

[minux]

Yeah, that's better.

[josharian]

This'll need to respect the branch as well to keep trybot useful for release branch testing.

[bradfitz]

@rsc says in email "Please keep using the exact commit. Thanks."

Russ, can you elaborate on what you'd like to see long-term? Would it be useful to also try the commit cherry-picked on top of tip?

[minux]

Please see https://golang.org/cl/6802/ as an extreme example of why this
issue should be fixed.

All trybots are fine with that CL, but because the CL is not rebased to the
latest master, after submission, it fails all builds.

[minux]

Sorry for the false alert. The problem I mentioned in the last comment is
caused by me running trybot on the wrong patch set. Will file a separate
issue.

Update: the actual reason is not me running on the wrong patch set.
It's because the auto-merge feature of Gerrit failed to merge the
change correctly. How could that be possible? (Note patch set 2 is created
by Gerrit, not by the author of the CL.)

So again, trybot should use rebase to test the patch in order to catch this
problem.

[minux]

Trybot really should rebase before testing, two more cases:

1. CL 7032 removes one argument to gc.Naddr, trybot passes, but
   the build breaks after it's submitted because master contains cmd/7g,
   but that CL is based on a commit before the introduction of 7g, so
   it doesn't change 7g.
2. trybot reports failure for CL 7080 on nacl/amd64p32, however,
   that's because the CL is based on a commit before the nacl build fix.
   The trybot messages asks to see the build dashboard if the failure
   is a new one, and according to the dashboard, it's indeed a new
   failure, but actually it's not.

[rsc]

Attempting to rebase completely breaks any testing of multi-CL changes, since the CL will be considered in isolation instead of the work that has come before. And at least some of us do pretty much nothing but multi-CL changes.

In the case of a commit hash where the parent is on origin/master (so it's either the bottom of or not part of a multi-CL change) then rebasing would be fine. But it's very important not to rebase otherwise, unless somehow you rebase the entire sequence.

[minux]

git fetch origin git rebase origin/master will rebase the whole sequence. In fact, that's how you manually rebase your multi-CL series to latest master.

[bradfitz]

Okay, this is now much easier.

We're now using our own Git rev -> .tar.gz server, so we can do things like this more easily, since we don't depend on Gerrit to generate the .tar.gz.

[dmitshur]

This would've helped catch a problem in CL 161497, where the tests passed with an older commit as a parent, but started to fail after being rebased on latest master.

If there are concerns with always rebasing in all trybot configurations, we can add just one extra configuration that tests on Linux while performing the rebase (similar to how we test subrepos on previous Go releases via additional trybot configurations).

/cc @bradfitz @stamblerre

[bradfitz]

If there are concerns with always rebasing in all trybot configurations, we can add just one extra configuration that tests on Linux while performing the rebase (similar to how we test subrepos on previous Go releases via additional trybot configurations).

That's a good place to start. But any +1 TryBot-Result vote would kinda need to expire once anything was submitted. Better than nothing, though. Or we could just focus our effort on a submit queue instead, letting bots do merging.

[randall77]

I worry that if the trybot fails, all the line numbers in the error log will be wrong.
Rebasing moves your changes around and there's no way to correspond the errors back to the line numbers of code in your local client (except by syncing your client to the same point as the trybot did).

It won't happen super-often, but when it does it will be mightily confusing.

[bradfitz]

@randall77, agreed. We'd probably need to have the bot upload the rebased tree to Gerrit. Its failure message already includes the commit hash, so then you could check it out locally and find the line.

[josharian]

What if the trybots try the rebased tree first, and on failure, try the existing tree (on failed builders only)?

We'd probably need to have the bot upload the rebased tree to Gerrit.

This would be frustrating when working with a branch of changes, some perhaps unmailed. And my git fu is strong; for many this could lead to lots of support requests.

[bradfitz]

By upload I didn't mean screwing with the owner's CL. Just creating a ref so people can pull it.

[gopherbot]

Change https://golang.org/cl/170457 mentions this issue: cmd/coordinator: on trybot failure, mention that trybots don't rebase