Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman-remote start --attach : can only attach to created or running containers - currently in state stopped #9597

Closed
edsantiago opened this issue Mar 3, 2021 · 25 comments
Closed

podman-remote start --attach : can only attach to created or running containers - currently in state stopped #9597

edsantiago opened this issue Mar 3, 2021 · 25 comments
Labels
flakes Flakes from Continuous Integration kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. remote Problem is in podman-remote stale-issue

Comments

@edsantiago
Copy link
Member

Looks like a race condition. Symptom is:

$ podman-remote [options] start --attach test
Error: unable to start container b1ce2aadbf58178fe75cb41da71dd0bf8e4b53cea21029da2a5c4d08a7f32da7: can only attach to created or running containers - currently in state stopped: container state improper

Podman run restart containers [It] Podman start after successful run
@edsantiago edsantiago added flakes Flakes from Continuous Integration kind/bug Categorizes issue or PR as related to a bug. remote Problem is in podman-remote labels Mar 3, 2021
@github-actions
Copy link

github-actions bot commented Apr 3, 2021

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Apr 5, 2021

@mheon this code looks like it will always race. Since we need to attach to the container before starting it?

@mheon
Copy link
Member

mheon commented Apr 5, 2021

The attach code on the API side should be able to handle this - it will take containers in a bad state and initialize them so we can attach. I think this is a bug in the API code for Attach.

@github-actions
Copy link

github-actions bot commented May 6, 2021

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented May 6, 2021

@mheon Do you have any time to tackle this?

@mheon
Copy link
Member

mheon commented May 6, 2021

Probably not, I'll be gone most of next week

@github-actions
Copy link

github-actions bot commented Jun 6, 2021

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Jun 7, 2021

@mheon Do you have time for this now?

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Jul 11, 2021

@mheon PTAL

@edsantiago
Copy link
Member Author

I haven't seen this since the initial issue. Feel free to close.

@mheon mheon closed this as completed Jul 12, 2021
@edsantiago
Copy link
Member Author

Reopening. Seen just now in f35 root, remote, in the seemingly-jinxed PR #11795

not ok 60 podman logs restarted - k8s-file
... 
# # podman-remote --url unix:/tmp/podman_tmp_L82A run --log-driver=k8s-file --name logtest quay.io/libpod/testimage:20210610 sh -c start=0; if test -s log; then start=`tail -n 1 log`; fi; seq `expr $start + 1` `expr $start + 10` | tee -a log
... (shows 1-10 correctly)
# # podman-remote --url unix:/tmp/podman_tmp_L82A start -a logtest
# Error: unable to start container 484b076605d18bb7f32187b2838a26b5fe5b8c604534803a20d00af701e73344: can only attach to created or running containers - currently in state stopped: container state improper
# [ rc=125 (** EXPECTED 0 **) ]

@edsantiago edsantiago reopened this Nov 16, 2021
@edsantiago edsantiago mentioned this issue Nov 16, 2021
Merged
@edsantiago
Copy link
Member Author

Bump. This is becoming a multiple-times-a-day flake. example

@edsantiago edsantiago mentioned this issue Nov 22, 2021
Merged
@edsantiago
Copy link
Member Author

[sys] 60 podman logs restarted - k8s-file

@edsantiago edsantiago mentioned this issue Dec 1, 2021
Merged
@edsantiago
Copy link
Member Author

This one is one of our biggest flakes right now. Who can take a look?

[sys] 60 podman logs restarted - k8s-file

[sys] 61 podman logs restarted journald

@vrothberg
Copy link
Member

I will take a look ✔️

@vrothberg
Copy link
Member

Ouch, this one even flakes locally.

@vrothberg vrothberg mentioned this issue Dec 6, 2021
Merged
vrothberg added a commit to vrothberg/libpod that referenced this issue Dec 6, 2021
@vrothberg
fix remote run/start flake
207065c 
Fix the flake reported in containers#9597 with a workaround to at least stop
wasting energy until the root cause has been found and fixed.

It seems that a remote run returns before the container has transitioned
into the `exited` state which ultimately breaks a subsequent remote
start with attach.

Signed-off-by: Valentin Rothberg <[email protected]>
@vrothberg
Copy link
Member

vrothberg commented Dec 6, 2021 •
edited
Loading

Easy reproducer on my workstation:

./bin/podman-remote run --replace --name test alpine sh -c 'echo 123'; ./bin/podman-remote start -a test

@vrothberg
Copy link
Member

There's something (race) going on in remote start/attach but I cannot put my finger on it.

@mheon could you take a look? The flake has been dealt with but the bug is still there.

edsantiago pushed a commit to edsantiago/libpod that referenced this issue Dec 8, 2021
@vrothberg @edsantiago
fix remote run/start flake
921e9ec 
Fix the flake reported in containers#9597 with a workaround to at least stop
wasting energy until the root cause has been found and fixed.

It seems that a remote run returns before the container has transitioned
into the `exited` state which ultimately breaks a subsequent remote
start with attach.

Signed-off-by: Valentin Rothberg <[email protected]>
@github-actions
Copy link

github-actions bot commented Jan 7, 2022

A friendly reminder that this issue had no activity for 30 days.

@github-actions
Copy link

github-actions bot commented Feb 7, 2022

A friendly reminder that this issue had no activity for 30 days.

@edsantiago
Copy link
Member Author

edsantiago commented Feb 7, 2022 •
edited
Loading

Last instance I see is from December 8, two days after @vrothberg's #12519, so the wait seems to have taken care of the flake (thank you Valentin!).

Since that was a sweep-under-the-rug, not a fix, I'm not willing to close this PR [EDIT: duh, issue, not PR.]

@vrothberg
Copy link
Member

I concur. It's really just a workaround but not a satisfactory solution.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Mar 25, 2022

Since no one is working on this and Ed said we can close it...

@rhatdan rhatdan closed this as completed Mar 25, 2022
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
flakes Flakes from Continuous Integration kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. remote Problem is in podman-remote stale-issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@edsantiago @rhatdan @mheon @vrothberg