Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set Setpgid on child git processes #19865

Merged
merged 11 commits into from
Jun 3, 2022

Conversation

zeripath
Copy link
Contributor

@zeripath zeripath commented Jun 1, 2022

When Gitea is running as PID 1 git will occassionally orphan child processes leading
to (defunct) processes. This PR simply sets Setpgid to true on these child processes
meaning that these defunct processes will also be correctly reaped.

Fix #19077

Signed-off-by: Andrew Thornton [email protected]

When Gitea is running as PID 1 git will occassionally orphan child processes leading
to (defunct) processes. This PR simply sets Setpgid to true on these child processes
meaning that these defunct processes will also be correctly killed.

Fix go-gitea#19077

Signed-off-by: Andrew Thornton <[email protected]>
@zeripath zeripath added this to the 1.17.0 milestone Jun 1, 2022
@zeripath
Copy link
Contributor Author

zeripath commented Jun 1, 2022

PLEASE CHECK THIS COMPILES ON WINDOWS AND RUNS OK ON WINDOWS.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jun 1, 2022
Copy link
Member

@delvh delvh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately don't have a windows machine to test.

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jun 1, 2022
@zeripath
Copy link
Contributor Author

zeripath commented Jun 1, 2022

Hmm... it looks like it doesn't compile on Windows. -- have adjusted this to make it work on doze

zeripath added 3 commits June 1, 2022 17:50
Signed-off-by: Andrew Thornton <[email protected]>
Signed-off-by: Andrew Thornton <[email protected]>
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Jun 1, 2022
modules/git/command_windows.go Outdated Show resolved Hide resolved
modules/git/command_unix.go Outdated Show resolved Hide resolved
@wxiaoguang
Copy link
Contributor

Looks good to me while I think there could be a comment in code to remember the reason why it is needed.

And I am not sure whether there could be a case that: Gitea -> SubProcessA -> SubProcessB , then although SubProcessA is killed (and there is no defunct due to Setpgid), SubProcessB is still running for a long time (still somewhat resource leaking?).

@codecov-commenter

This comment was marked as outdated.

@singuliere
Copy link
Contributor

Looks good to me while I think there could be a comment in code to remember the reason why it is needed.

And I am not sure whether there could be a case that: Gitea -> SubProcessA -> SubProcessB , then although SubProcessA is killed (and there is no defunct due to Setpgid), SubProcessB is still running for a long time (still somewhat resource leaking?).

You misunderstand the purpose of Setpgid, let me explain:

although SubProcessA is killed (and there is no defunct due to Setpgid)...

SubProcessA is not defunct/zombie because Gitea waits for it, not because of Setpgid

SubProcessB is still running for a long time...

the purpose of Setpgid is to make sure SubProcessB is also killed when SubProcessA is killed. For a more detailed explanation and a reproducer, please see https://hostea.org/blog/zombies/.

@wxiaoguang
Copy link
Contributor

wxiaoguang commented Jun 2, 2022

func main() {
	c := exec.Command("/bin/bash", "-c", `echo "sleep 100000" | bash`)
	c.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
	fmt.Printf("run err = %v", c.Run())
}
go run test-setpgid.go
pstree
 |      \-+- 86853 user test-setpgid
 |       \-+= 86855 user /bin/bash -c echo "sleep 100000" | bash
 |         \-+- 86857 user bash
 |           \--- 86858 user sleep 100000

kill 86855
run err = signal: terminated

pstree
 |-+- 86857 user bash
 | \--- 86858 user sleep 100000

@singuliere
Copy link
Contributor

kill -9 -- -86855

Would have worked as intended.

@wxiaoguang
Copy link
Contributor

func main() {
	ctx, _ := context.WithTimeout(context.Background(), time.Second*2)
	c := exec.CommandContext(ctx, "/bin/bash", "-c", `echo "while true; do echo running; sleep 1; done" | bash`)
	c.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
	c.Stdout = os.Stdout
	fmt.Printf("run err = %v", c.Run())
}
$ go run test-setpgid.go
running
running
run err = signal: killed
running
running
running
...

@singuliere
Copy link
Contributor

singuliere commented Jun 2, 2022

package main

import (
	"fmt"
	"os"
	"os/exec"
	"syscall"
	"time"
)

func main() {
	c := exec.Command("/bin/bash", "-c", `echo "while true; do echo running; sleep 1; done" | bash`)
	c.Stdout = os.Stdout
	c.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
	c.Start()
	time.Sleep(5 * time.Second)
	syscall.Kill(-c.Process.Pid, syscall.SIGKILL)
	fmt.Println("run err = ", c.Wait())
}
$ go run /tmp/pgidworks.go 
running
running
running
running
running
run err =  signal: killed
$

@wxiaoguang
Copy link
Contributor

Then, does Gitea use your syscall.Kill in modules/git/command.go or modules/process/manager_exec.go? Or Gitea just simply uses contexts with deadlines?

@zeripath
Copy link
Contributor Author

zeripath commented Jun 3, 2022

This PR is not designed to kill children - but to simply allow for them to be reaped.

We need to check that this does stop the (defunct) processes from occuring.

@zeripath
Copy link
Contributor Author

zeripath commented Jun 3, 2022

I've just checked and it does.

@wxiaoguang

This comment was marked as off-topic.

@singuliere

This comment was marked as off-topic.

@zeripath zeripath merged commit 1d04e86 into go-gitea:main Jun 3, 2022
@zeripath zeripath deleted the set-pgid-on-child-processes branch June 3, 2022 14:36
zeripath added a commit to zeripath/gitea that referenced this pull request Jun 3, 2022
Backport go-gitea#19865

When Gitea is running as PID 1 git will occassionally orphan child processes leading
to (defunct) processes. This PR simply sets Setpgid to true on these child processes
meaning that these defunct processes will also be correctly reaped.

Fix go-gitea#19077

Signed-off-by: Andrew Thornton <[email protected]>
@zeripath zeripath added backport/done All backports for this PR have been created backport/v1.16 labels Jun 3, 2022
@zeripath

This comment was marked as off-topic.

@zeripath
Copy link
Contributor Author

zeripath commented Jun 3, 2022

Please stop both of you - I've deleted the last few comments as I don't think there is anything to be gained from them.

@go-gitea go-gitea locked as resolved and limited conversation to collaborators Jun 3, 2022
@go-gitea go-gitea deleted a comment from wxiaoguang Jun 3, 2022
@go-gitea go-gitea deleted a comment from singuliere Jun 3, 2022
@go-gitea go-gitea deleted a comment from singuliere Jun 3, 2022
@go-gitea go-gitea deleted a comment from wxiaoguang Jun 3, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport/done All backports for this PR have been created lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Defunct git processes when running Gitea as PID 1 on docker
8 participants