Skip testMergeWithMatrixBuild if gpgsign enabled #1564
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The testMergeWithMatrixBuild test fails randomly on several machines when commit.gpgsign and tag.gpgsign are not enabled if the TestPreBuildMerge implementation is used. It passes consistently when PreBuildMerge is used. Rather than spend the time trying to diagnose the intermittent failures, this configuration allows the test to be skipped if either of those configuration settings are enabled. Other tests in this class are able to use TestPreBuildMerge without issue. Tested by enabling commit.gpgsign in ~/.gitconfig and by enabling commit.gpgsign in a $XDG_CONFIG_HOME/.gitconfig file. Test was skipped when gpgsign was enabled in either of those files.
olamy
reviewed
Mar 18, 2024
| /* The testMergeWithMatrixBuild test fails randomly on several | ||
| * machines when commit.gpgsign and tag.gpgsign are not | ||
| * enabled if the TestPreBuildMerge implementation is used. It | ||
| * passes consistently when PreBuildMerge is used. Rather than |
There was a problem hiding this comment.
interesting. what sort of error did you get? Did it happen on https://ci.jenkins.io/?
but this doesn't mean I really want to spend time on that :)
There was a problem hiding this comment.
I ran more tests to check failure rates on various machines in my home lab. Failure rate seemed to decrease as machine performance decreased.
- RHEL 8 with Ryzen 5 - 16/16 failed
- Ubuntu 22.04 with Xeon(R) X5660 - 14/16 failed
- Ubuntu 22.04 with Xeon(R) X3450 - 13/16 failed
- Ubuntu 22.04 with i5-6400 - 12/16 failed
- Debian testing with i5-2400 - 7/16 failed
|
On Sun, Mar 17, 2024 at 11:26 PM Olivier Lamy ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In src/test/java/hudson/plugins/git/GitSCMSlowTest.java
<#1564 (comment)>
:
> @@ -340,6 +360,18 @@ public GitClient decorate(GitSCM scm, GitClient git) throws IOException, Interru
@test
public void testMergeWithMatrixBuild() throws Exception {
assumeTrue("Test class max time " + MAX_SECONDS_FOR_THESE_TESTS + " exceeded", isTimeAvailable());
+ /* The testMergeWithMatrixBuild test fails randomly on several
+ * machines when commit.gpgsign and tag.gpgsign are not
+ * enabled if the TestPreBuildMerge implementation is used. It
+ * passes consistently when PreBuildMerge is used. Rather than
interesting. what sort of error did you get? Did it happen on
https://ci.jenkins.io/?
It does not fail on ci.jenkins.io but it fails at least 50% of the time on
computers in my home lab, including Red Hat Enterprise Linux 8, Ubuntu
22.04, and Debian Testing.
The failure message is:
Commit message: "Initial Commit"
Using 'Changelog to branch' strategy.
Triggering xyz » b
Triggering xyz » a
xyz » b completed with result FAILURE
xyz » a completed with result SUCCESS
Finished: FAILURE
…------
Expected: is <SUCCESS>
but: was <FAILURE>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at
org.jvnet.hudson.test.JenkinsRule.assertBuildStatus(JenkinsRule.java:1502)
at
hudson.plugins.git.AbstractGitTestCase.build(AbstractGitTestCase.java:274)
at
hudson.plugins.git.GitSCMSlowTest.testMergeWithMatrixBuild(GitSCMSlowTest.java:359)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at
org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:656)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.lang.Thread.run(Thread.java:1583)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
GitSCMSlowTest.testMergeWithMatrixBuild:359->AbstractGitTestCase.build:274
unexpected build status; build log was:
I'm also not ready to spend lots of time studying it more deeply. This
change works on ci.jenkins.io and in my home lab and it retains the test
for many environments, including ci.jenkins.io.
Message ID: ***@***.***>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Skip testMergeWithMatrixBuild if gpgsign enabled
The testMergeWithMatrixBuild test fails randomly on several machines when commit.gpgsign and tag.gpgsign are not enabled if the TestPreBuildMerge implementation is used. It passes consistently when PreBuildMerge is used.
Rather than spend the time trying to diagnose the intermittent failures, this configuration allows the test to be skipped if either of those configuration settings are enabled.
Other tests in this class are able to use TestPreBuildMerge without issue.
Would like a review from @olamy, if possible.
Testing done
Tested by enabling commit.gpgsign in ~/.gitconfig and by enabling commit.gpgsign in a $XDG_CONFIG_HOME/.gitconfig file. Test was skipped when gpgsign was enabled in either of those files.
Checklist
Types of changes