Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove containers grace period #686

Merged
merged 5 commits into from
Jan 20, 2020

Conversation

Vlatombe
Copy link
Member

@Vlatombe Vlatombe commented Jan 16, 2020

Nothing interesting should be running in agents containers when it is requested to be terminated, so don't wait these 30 seconds to kill remaining processes inside it.

Should speed up a lot tests hopefully.

Subsumes #689 and #690, please review this commit.

@kerogers-cloudbees
Copy link
Contributor

There's a test failure in ContainerExecDecoratorTest.testCommandExecution() Is that existing or was that test relying on the grace period to execute?

@Vlatombe
Copy link
Member Author

It's also on master. I never hit it when running tests locally but it seems to happen systematically when running on kind 0.7.0 (#685), not sure why.

@Vlatombe
Copy link
Member Author

Vlatombe commented Jan 17, 2020

Saves about 14 minutes (40 minutes total vs. 54 min)

@Vlatombe
Copy link
Member Author

Test failure already seen in #553

@Vlatombe
Copy link
Member Author

Looks like buffer from other threads is printed instead.

[4] pid is 20
[0] pid is 21
Executing command: "sh" "-c" "cd /tmp; echo [1] pid is \$\$ > test; cat /tmp/test" 
exit
Executing command: "sh" "-c" "cd /tmp; echo [2] pid is \$\$ > test; cat /tmp/test" 
exit
Executing command: "sh" "-c" "cd /tmp; echo [3] pid is \$\$ > test; cat /tmp/test" 
exit
Executing command: "sh" "-c" "cd /tmp; echo [9] pid is \$\$ > test; cat /tmp/test" 
exit
[7] pid is 51
[5] pid is 53
[5] pid is 53
[5] pid is 53
Thread 0 finished
Thread 4 finished
[2] pid is 81
Thread 8 finished
Thread 6 finished
Thread 5 finished
Thread 2 finished
Thread 7 finished
[9] pid is 82
Thread 3 finished
[9] pid is 82

See duplicated [5] and [9]

@Vlatombe
Copy link
Member Author

@jglick looks like some synchronization issue when running in cpu constrainted environment.

@Vlatombe
Copy link
Member Author

@jglick Don't review this right now, cleaning up the patch first.

Copy link
Member

@jglick jglick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(trying to ignore changes from #690)

@@ -198,6 +198,7 @@ public Pod build() {
builder.withNodeSelector(nodeSelector);
}

builder.withTerminationGracePeriodSeconds(0L);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really safe?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering as well in case some process really need to do cleanup. Probably safer to set it for all tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

safer to set it for all tests

Do you mean set it only for tests?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is what I meant

@@ -120,7 +120,7 @@ public void configureCloud() throws Exception {
.withCommand("cat").withTty(true).withWorkingDir("/home/jenkins/agent1").build();
String podName = "test-command-execution-" + RandomStringUtils.random(5, "bcdfghjklmnpqrstvwxz0123456789");
pod = client.pods().create(new PodBuilder().withNewMetadata().withName(podName)
.withLabels(getLabels(this, name)).endMetadata().withNewSpec().withContainers(c, d).withNodeSelector(Collections.singletonMap("kubernetes.io/os", "linux")).endSpec().build());
.withLabels(getLabels(this, name)).endMetadata().withNewSpec().withContainers(c, d).withNodeSelector(Collections.singletonMap("kubernetes.io/os", "linux")).withTerminationGracePeriodSeconds(0L).endSpec().build());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine here in tests.

@Vlatombe Vlatombe merged commit cfcc2e5 into jenkinsci:master Jan 20, 2020
@Vlatombe Vlatombe deleted the remove-grace-period branch January 20, 2020 10:11
@Vlatombe Vlatombe added the test Tests label Jan 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test Tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants