[🐛 Bug]: Stopping node container now stops attached video container

[MJB222398]

What happened?

I was using Selenium 4.23. I have a Docker selenium grid with 3 nodes (Chrome, Edge, Firefox) and then 3 corresponding video containers, one for each of these. When tests are done I stop the 3 node containers so that the video recording can finish and then pull the video files from the video containers. Works every time.

I had to upgrade to Selenium 4.27 to avoid other bugs and now the above no longer works. It seems that stopping the node container now stops the corresponding video container. Why is this? Is it a bug or is it expected? If so is this breaking change documented? What am I supposed to do instead? Is there another way to stop the recording without stopping the node container?

I tried versions in between to find when the bug was released. The first version I see it in is 20240829. The bug is not present in 20240820.

Command used to start Selenium Grid with Docker (or Kubernetes)

N/A

Relevant log output

2024-12-17 16:01:31,658 INFO supervisord started with pid 8
2024-12-17 16:01:32,661 INFO spawned: 'video-ready' with pid 9
2024-12-17 16:01:32,662 INFO spawned: 'video-upload' with pid 10
2024-12-17 16:01:32,663 INFO spawned: 'video-recording' with pid 11
2024-12-17 16:01:32,673 [video.uploader] - UPLOAD_RETAIN_LOCAL_FILE is set to false, force to use RCLONE command: move
2024-12-17 16:01:32,675 INFO success: video-ready entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-12-17 16:01:32,675 INFO success: video-upload entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-12-17 16:01:32,675 INFO success: video-recording entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-12-17 16:01:32,680 [video.uploader] - Waiting for /tmp/uploadpipe to be present
2024-12-17 16:01:32,732 [endpoint.checks] - Endpoint http://chrome:5555/status is not available - status code: 000
2024-12-17 16:01:32,735 [video.recorder] - Video folder exists: /videos
2024-12-17 16:01:32,736 [video.recorder] - Waiting for the display chrome:99.0 is open
2024-12-17 16:01:32,930 [video.recorder] - Display chrome:99.0 is open with dimensions 1920x1080
[x11grab @ 0x564ba054ed00] Cannot get the image data event_error: response_type:0 error_code:1 sequence:9 resource_id:1003 minor_code:4 major_code:130.
[x11grab @ 0x564ba054ed00] Continuing without shared memory.
[x11grab @ 0x564ba054ed00] Stream #0: not enough frames to estimate rate; consider increasing probesize

Operating System

Ubuntu 20.04

Docker Selenium version (image tag)

4.27.0-20241204

Selenium Grid chart version (chart version)

No response

[github-actions]

@MJB222398, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

Hi, can you share docker file, I would like to see env vars for each container also.

[MJB222398]

Hi, I don't have a docker file. I build up the grid one by one. I can send you the environment variables though sure. Is this not expected behaviour then?

[MJB222398]

@VietND96 OK so for a video container:

NVIDIA_VISIBLE_DEVICES=all
HOSTNAME=0d2a0a2cd1cf
S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0
LANGUAGE=en_US.UTF-8
SE_SUPERVISORD_CHILD_LOG_DIR=/tmp
SE_VIDEO_FILE_NAME=video.mp4
SE_SERVER_PROTOCOL=http
SE_SCREEN_WIDTH=1920
SE_VIDEO_POLL_INTERVAL=1
SEL_UID=1200
PWD=/
NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
SE_VIDEO_UPLOAD_ENABLED=false
TZ=UTC
VIDEO_FOLDER=/videos
DISPLAY_NUM=99
SE_PRESET=-preset ultrafast
DISPLAY_CONTAINER_NAME=edge
HOME=/home/seluser
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
VIRTUAL_ENV=/lsiopy
S6_VERBOSITY=1
S6_STAGE2_HOOK=/docker-mods
SE_FRAME_RATE=15
SE_SUPERVISORD_PID_FILE=/tmp/supervisord.pid
SE_CODEC=libx264
TERM=xterm
DEBCONF_NONINTERACTIVE_SEEN=true
SHLVL=1
SE_VIDEO_INTERNAL_UPLOAD=true
SEL_USER=seluser
SE_SUPERVISORD_LOG_LEVEL=info
PIP_ROOT_USER_ACTION=ignore
SE_SCREEN_HEIGHT=1080
LD_LIBRARY_PATH=/usr/local/lib
LIBVA_DRIVERS_PATH=/usr/local/lib/x86_64-linux-gnu/dri
LSIO_FIRST_PARTY=true
FILE_NAME=recording.mp4
SE_SUPERVISORD_LOG_FILE=/tmp/supervisord.log
SE_VIDEO_FILE_NAME_TRIM_REGEX=[:alnum:]-_
PATH=/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBIAN_FRONTEND=noninteractive
SEL_GID=1201
SE_UPLOAD_DESTINATION_PREFIX=
_=/usr/bin/printenv

And then for the node container:

SE_ENABLE_BROWSER_LEFTOVERS_CLEANUP=false
SE_START_NO_VNC=true
SE_NO_VNC_PORT=7900
SEL_DOWNLOAD_DIR=/home/seluser/Downloads
HOSTNAME=de1ef131f002
SE_BROWSER_LEFTOVERS_PROCESSES_SECS=7200
LANGUAGE=en_US.UTF-8
SE_NODE_MAX_SESSIONS=1
SE_SUPERVISORD_CHILD_LOG_DIR=/tmp
SE_JAVA_SSL_TRUST_STORE=/opt/selenium/secrets/server.jks
SE_HTTPS_CERTIFICATE=/opt/selenium/secrets/tls.crt
SE_SERVER_PROTOCOL=http
SE_SCREEN_WIDTH=1920
SE_SCREEN_DPI=96
CONFIG_FILE=/opt/selenium/config.toml
SE_ENABLE_TLS=false
SEL_UID=1200
PWD=/
HUB_HOST=selenium-hub
SE_START_XVFB=true
TZ=UTC
SE_OPTS=--log-level CONFIG
DISPLAY_NUM=99
SE_REJECT_UNSUPPORTED_CAPS=false
SE_NODE_OVERRIDE_MAX_SESSIONS=false
HOME=/home/seluser
SE_HTTP_LOGS=false
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
LANG_WHERE=US
SE_NODE_HEARTBEAT_PERIOD=30
SE_EVENT_BUS_SUBSCRIBE_PORT=4443
SE_OFFLINE=true
ENCODING=UTF-8
SE_NODE_GRID_URL=http://localhost:4444
GENERATE_CONFIG=true
SE_BROWSER_LEFTOVERS_INTERVAL_SECS=3600
SE_SUPERVISORD_PID_FILE=/tmp/supervisord.pid
SE_ENABLE_TRACING=true
SE_STRUCTURED_LOGS=false
SE_EVENT_BUS_PUBLISH_PORT=4442
TERM=xterm
SE_OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED=true
SE_DRAIN_AFTER_SESSION_COUNT=0
DEBCONF_NONINTERACTIVE_SEEN=true
SE_NODE_SESSION_TIMEOUT=180
SE_VNC_PORT=5900
SE_BIND_HOST=false
SE_BROWSER_LEFTOVERS_TEMPFILES_DAYS=1
SE_SUPERVISORD_AUTO_RESTART=true
DISPLAY=:99.0
SHLVL=1
SE_OTEL_SERVICE_NAME=selenium-node-edge
SEL_USER=seluser
SE_SUPERVISORD_LOG_LEVEL=info
SE_SCREEN_HEIGHT=1080
SE_START_VNC=true
SE_LOG_LEVEL=INFO
SE_OTEL_TRACES_EXPORTER=otlp
SE_LOG_TIMESTAMP_FORMAT=%Y-%m-%d %H:%M:%S,%3N
LANG_WHICH=en
SE_SUPERVISORD_LOG_FILE=/tmp/supervisord.log
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SE_EVENT_BUS_HOST=selenium-hub
SE_SCREEN_DEPTH=24
SE_JAVA_SSL_TRUST_STORE_PASSWORD=/opt/selenium/secrets/server.pass
DBUS_SESSION_BUS_ADDRESS=/dev/null
SE_JAVA_DISABLE_HOSTNAME_VERIFICATION=true
SE_SUPERVISORD_START_RETRIES=5
DEBIAN_FRONTEND=noninteractive
SEL_GID=1201
SE_HTTPS_PRIVATE_KEY=/opt/selenium/secrets/tls.key
_=/usr/bin/printenv

[VietND96]

In the changesets, it has an improvement on container graceful shutdown for recorder.
I saw the env var SE_VIDEO_FILE_NAME=video.mp4, so are you using static file name? Have you ever tried to use SE_VIDEO_FILE_NAME=auto yet? It will listen on Node endpoint and start recording for any session starting.
I also saw env var enable internal upload. In your case, the container will be stopped without enough time for uploader to transfer file to remote storage, right?

[MJB222398]

@VietND96 Ah OK I guess I was relying on a behaviour that I shouldn't have been and that has now changed. I was using static filename yes, as I was not aware of 'auto'. Perhaps that is relatively new. I think I should probably use that. Let me try to do this migration and see if my issue is resolved.

I was not aware of SE_VIDEO_INTERNAL_UPLOAD variable either, so that was just the default value of it you saw, I was not setting it. I will explicitly set that to false then as I think I won't need it - I just use docker CLI to pull the file from the video container to my machine.

Looking at the docs I see SE_VIDEO_RECORD_STANDALONE is mandatory but I am not seeing it in my environment variables? What does it mean/do anyway - that is not documented.

What effect does specifying SE_NODE_GRID_URL have for the video container? I was not doing this.

[VietND96]

SE_VIDEO_RECORD_STANDALONE is needed in case you want to record a standalone container, where Hub & Node in same (e.g standalone-chrome). If you are using node-chrome and a separate Hub container, it isn't required. Check out the example: https://github.com/SeleniumHQ/docker-selenium/blob/trunk/docker-compose-v3-video-upload-standalone.yml

SE_NODE_GRID_URL is nice to have if you are using SE_VIDEO_FILE_NAME=auto with support to set dynamic video file name via setting capabilities in tests while creating the Remote WebDriver. You can check out example: https://github.com/SeleniumHQ/docker-selenium/blob/trunk/docker-compose-v3-video-upload.yml and README: https://github.com/SeleniumHQ/docker-selenium?tab=readme-ov-file#video-recording-with-dynamic-file-name-based-on-metadata-in-tests

[MJB222398]

Thanks I shall take a look at that. Using auto is largely working for me. The only problem is is that when I dispose the driver and then pull the video the file is not a valid video. This is because the video buffer was not yet flushed. So I have to wait for this. Is there a way for me to tell when it is ready rather than just waiting an arbitrary time?

[VietND96]

Waiting for an arbitrary time that you mentioned here is the stop_grace_period defined for container in docker compose file, isn't it?

docker-selenium/docker-compose-v3-video-upload-standalone.yml

Line 74 in 2741b81

stop_grace_period: 30s

[MJB222398]

No that's for waiting for the container to stop. I'm wanting to know how to wait for the video to be fully buffered and flushed to disk when I close the web driver session.

[MJB222398]

@VietND96 Perhaps the video should be written to a temporary file until it is fully finished and then moved to the expected filename once complete? That way I can know that the video has finished before I grab it. Unless there is some kind of event or something that I am not aware of that can tell me when its done?

[VietND96]

Via video.sh script in the recorder container, FFmpeg writes to a file directly under /videos (not through any temp file). We just consider stopping the container, which triggers the termination of FFmpeg, since it might impact the process of appending EOF correctly to the file.

[VietND96]

Regarding Stopping node container now stops attached video container, with approach static file name, the FFmpeg command will be triggered, then it waits for that PID finish.

docker-selenium/Video/video.sh

Line 190 in 7216d06

wait $!

If the session stoped, Node container could be stopped (reach DRAIN_AFTER_SESSION if set any) or stopped by user proactively, it means the screen closed, it also means FFmpeg could not get stream anymore. That's time the PID finished, go to end of script and finished the program in container also.
However, if docker-compose file configure restart policy always can trigger it back and standby for next session.