Fail to checkout PR when a push -f is done to the PR while is being already built #148
Comments
|
I'm testing this on v1.532.3 with v1.12 of the plugin and do not see it on a non-matrix job. |
|
I've tried this on a matrix build as well, and it seems to work fine. When you execute the "git push -f", is the build step actually running? Or is the build waiting in a quiet period? |
|
On Wed, May 14, 2014 at 08:11:53AM -0700, Valdis Rigdon wrote:
It's running, I didn't check with the quiet period. I'll look to update |
|
Note that my testing is being done on Windows, but I can't imagine that would affect this. |
|
I'm testing doing the Anyway, by pushing Now I'll test again to push |
|
And I can still reproduce the problem when pushing |
|
Is the pull request you are testing mergeable? |
|
On Wed, May 14, 2014 at 09:05:05AM -0700, Valdis Rigdon wrote:
Yes, there is also a 2 minutes quiet period for the job, but I have the This is a poors man time diagram: _ <- push/create PR : is quiet period |
|
On Wed, May 14, 2014 at 09:13:43AM -0700, Valdis Rigdon wrote:
Yes it is. |
|
Are you able to attach Jenkins logs? Add a logger for "org.jenkinsci.plugins.ghprb" set to DEBUG. In addition, seeing the Git plugin logs would help. |
|
On Wed, May 14, 2014 at 09:17:52AM -0700, Valdis Rigdon wrote:
Mmm, how can I do that? I added a logger for |
|
ALL works. Sent from my iPad On May 15, 2014, at 6:03 AM, "Leandro Lucarella" <[email protected]mailto:[email protected]> wrote: On Wed, May 14, 2014 at 09:17:52AM -0700, Valdis Rigdon wrote:
Mmm, how can I do that? I added a logger for Reply to this email directly or view it on GitHubhttps://github.com//issues/148#issuecomment-43191268. |
|
Here is the log for a Logger: The hash used there is OK, 853e20559daea977aa6b63d53d378e875258e147 is the new HEAD for that PR. Console output for the failed job (the log for one of the matrix configs): The hash 655e53cff41bffdb724ded7e4a6be3fc2e541284 is obviously the wrong one, and the hash for the previous HEAD for the PR, before the |
|
One other difference in my testing is that I'm using polling. I will try that. |
|
Under SCM --> Git, what additional behaviors do you have set? I'm using push notifications now, and a matrix job, and it still seems to do the right thing. |
|
The log for the matrix config that's failing doesn't show a call to |
|
One more question -- are the matrix builds being run on remote build agents? |
Right now, "clean after checkout", but I tried disabling it before and it didn't help. |
Mmm, this looks weird. I have no idea of who is in charge of doing that or how can I change the behaviour. |
No, it's all local, they run in a vagrant VM, but that's in the build script, and the build is broken before that. |
|
And I actually, this test repository is not even using vagrant, the build script is just: |
|
What version of Git is on the box? |
|
On Fri, May 16, 2014 at 05:20:16AM -0700, Valdis Rigdon wrote:
git version 1.7.9.5 |
|
Have you tried upgrading to the 1.9.x release? I'm trying to eliminate various differences between your setup and what I'm testing on here. |
|
On Fri, May 16, 2014 at 06:38:16AM -0700, Valdis Rigdon wrote:
I can upgrade, sure. |
|
Mmm... I upgraded Git to 1.9.3 and Jenkins to the new LTS version 1.554.1, and I can't reproduce the problem with that configuration. |
|
I'll re-enable the PR builder in other jobs and report back if I find any jobs failing. |
|
Hi, all, And if we rerun the job - all works fine. |
|
reported in Jenkins Jira as well - https://issues.jenkins-ci.org/browse/JENKINS-26748 |
|
We are also experiencing this problem with matrix jobs. The parent has the correct reference but then when it starts the child jobs it looks like it passes along an old sha1 from a previous merge and not the updated one. Output from parent job It is checking out the merge HEAD of the pull request at However, when the child builds start you see (snipped output) It is trying to checkout |
|
@martyngigg ghprb is innocent, it's an upstream GitHub bug, as far as my analysis goes. If you still have the workspaces from your failed builds, try "git log" on your build nodes - the tip of the branch they're on should be an auto-generated commit merging the branch and master. This commitid is in refs/remotes/origin/pr/*/merge. There will be a discrepancy between what your successful nodes believe this commit should be, and what the failures think it should be. If you check the timestamp on the merge commits, they will differ - even though all nodes in the matrix job started simultaneously & git fetch'd simultaneously. I filed a support issue with GitHub late last week, but have had nothing back on the topic. |
|
@directhex Thanks for the update. I had spotted the differing commits but not the timestamps. Hopefully they will respond soon. |
|
Has anyone found a workaround for this issue? It's been a major problem for us whenever rebasing (which happens fairly often). |
|
On Mon, Mar 16, 2015 at 10:12:59AM -0700, JP Simard wrote:
I'm not having this problem often anymore, but when it happens usually |
|
"Rebuild Last" always succeeds, but manually having to trigger builds kind of defeats the purpose of continuous integration. |
|
I still haven't heard anything from Github, including when I put out feelers via internal contacts. The problem is at their end. |
|
Yeah, I've the same issue and it doesn't depend of git version. |
|
@janinko can we add a config switch to disable the whole merge mechanism? As in we execute the job on the plain branch and leave it to the developers to rebase the branch if needed. Thanks a lot! |
|
Check the readme, but there is an environment variable that gives you the original commit, not the test merge one. |
|
@DavidTanner I assume you are talking about this? Will this help with respect to this issue and NOT try to perform the merge at all? |
|
The merge happens because you created a pull request on a GitHub. This plugin just tells the git client what commit/branch/etc to use when it does the checkout portion. |
|
Thanks David. I take that as a no then. Too bad - these bloody issue is almost rendering this plugin unusable. :-( |
|
It's a problem only with matrix jobs. Here's what happens:
The race condition here is that because the PR merge commit is ephemeral (i.e. it gets recreated every time there's a commit to either the PR, or the repo the PR targets). If there's any commit between when the matrix master running and the matrix slaves running (e.g. if one of your matrix axis slaves is overloaded so the scheduling takes some time), the problem occurs because the commitid that the matrix master saw no longer exists by the time the matrix slaves fetch the repo. The "easy" fix is to make sure matrix slaves build the merge ref, not the sha1. Or, as we did, don't use matrix jobs at all. |
|
Great input @directhex thx That easy fix has to be implemented within the plugin? Or is this something we can somehow configure already as if today? |
|
@directhex you have it almost correct. If it is mergeable, then ${sha1} = pr/###/merge, else ${sha1} = abcdef1, where abcdef1 is sha of last commit to the pull request when we queried the api. If ${sha1} is pr/###/merge, then the actual sha hash is resolved by the jenkins git plugin when checking out the workspace. |
Nice debugging, folks. In my situation I have to use matrix builds. Is there any way of working around this currently by tweaking settings or will it need a bugfix in a future |
|
Just to clarify, it sounds like the underlying issue with matrix builds is that GitHub force pushes to the Further expanding the code snippet from This makes it look like you could avoid this problem by using |
|
I'm experiencing a related issue on a jenkins and self hosted git repository too after rebasing the development tree, it's old tip tree-ish id is evaluated for checkout even I prune / delete workspace completely, including git force clone. My speculation: The plugin seems to evaluate the last successfull build for getting the id to calculate a diff? what is strange to me and what let me find a fix/workaround for me: this is the one from the failed one after rebasing and push forcing develop: 6bc8861 is no more the remote I checked back the git repository on the jenkins server: The error So: I fixed up the given name / id for the remote and changed it to origin and: anything working again! Perhaps this might help someone too, still I'm not sure / I expect the root-cause is somewhere else. unfortunatly: all branches are rebuild, even they were build with the "wrong" configuration |
We did exactly that to resolve this issue, which we were hitting probably a third(!) of the time. Patch against git (not ghprb) plugin 2.4.0: diff --git a/src/main/java/hudson/plugins/git/GitSCM.java b/src/main/java/hudson/plugins/git/GitSCM.java
index aca465d..bbddc53 100644
--- a/src/main/java/hudson/plugins/git/GitSCM.java
+++ b/src/main/java/hudson/plugins/git/GitSCM.java
@@ -915,6 +915,7 @@ public class GitSCM extends GitSCMBackwardCompatibility {
// every MatrixRun should build the same marked commit ID
+ /*
if (build instanceof MatrixRun) {
MatrixBuild parentBuild = ((MatrixRun) build).getParentBuild();
if (parentBuild != null) {
@@ -926,6 +927,7 @@ public class GitSCM extends GitSCMBackwardCompatibility {
}
}
}
+ */
// parameter forcing the commit ID to build
if (candidates.isEmpty() ) {The patch just bluntly disables the git plugin's special case logic for matrix builds, so each child will use I posted this also on JENKINS-26290, where hopefully a better fix can be found. I suspect an optimal solution would require changes to both plugins. |
|
Using ${ghprbActualCommit} instead of ${sha1} seems to be a decent work around for my team. We were having this issue using the MultiJob jenkins. |
|
Hi everyone. We switched to using ${ghprbActualCommit} but the problem still stands. The initial parent-child jobs works just fine, but on the second run the parent has the correct details but the child job fails from trying to checkout the wrong commit. In our case oftentimes there are new commits in the pull request while the first PR job is still running, hence the failure. Are there any further solutions/advice/experience for what to do in this situation? |
|
There's a separate, similar-looking bug in the Git plugin that causes a "reference is not a tree" failure, unrelated to matrix builds or push -f. There's a potential fix in JENKINS-2629. The characteristic error is that the build is supposed to test "origin/pr/123/merge", and then proceeds to fail to checkout a different PR's branch, like "origin/pr/345/merge", resulting in |
This issue comes from https://issues.jenkins-ci.org/browse/JENKINS-22537
Please consult that issue for details. Here is the summary of the latest research.
The setup is a job using a configuration matrix (I'm not sure if this is strictly necessary to reproduce the problem though) with only one variable and 2 configurations (which is only the Ubuntu version to do the build). Then the ghprb plugin is activated and configured to trigger a build by the GH hook.
Steps to reproduce:
sleep 60git commit --amendand just change the commit message)git push -fto your PR branchThe error (this is the console output for one of the configurations in the matrix, the other have the same error) is:
This is the console output for the whole job (not one configuration in particular):
The text was updated successfully, but these errors were encountered: