Beta and dev versions in Docker Hub

[bonigarcia]

🚀 Feature Proposal

It would be amazing if we could pull the official selenium-docker images for the beta and development versions of the major browsers (i.e., Chrome, Firefox, and Edge) from Docker Hub. It would be also great if we could have a new tag for the current stable version.

Motivation

For many companies, it is key to have the possibility of checking if a web application continues working as expected in the next stable release. This process can be tricky, but it can be easily done with Docker containers. Companies like Twilio are already maintaining similar Docker images.

Example

Currently, we can pull a given version of the major browsers as follows:

docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:90.0
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-firefox:91:0
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-edge:88.0

My proposal requires to include and maintain new tags: stable, beta (typically stable+1), and dev (typically stable+2):

docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:stable
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-firefox:stable
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-edge:stable

docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:beta
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-firefox:beta
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-edge:beta

docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:dev
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-firefox:dev
docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-edge:dev

[diemol]

I think the idea makes sense. However, there are several open questions:

• Would we give any type of support to those beta and dev images? They are not actually released.
• Are the browser drivers for beta and dev versions always available? What to do if they are not?
• How often will those images be updated? Dev has very frequent updates, for example.
• What do we do if our testing pipeline does not pass for a dev/beta browser? Do we skip it or we avoid releasing it?

Thanks for sharing the Twilio link, but I imagine they do not offer any type of support for it, do they?

I am not opposing to the idea at all, only thinking how these images should work and under what conditions they can be offiered.

For now, we are focusing on Selenium 4, and probably after that, we want to have the Helm Chart merged to this repo. Which means that we do not have the bandwidth to work on this for now. Would you like to formulate a more complete proposal and then maybe a PR? What do you think?

[bonigarcia]

I suppose this could be done with a scheduled task (e.g., nightly) in your CI server to check the availability of new browser releases. When a new release is detected, the Docker container could be automatically built and deployed to Docker Hub.

Is this the process you follow for stable releases?

If so, extending it to the beta and dev channel should not be very difficult. I guess it can be done by checking the proper distribution channels.

Regarding the support of these images, if the test pipeline fails for a given version, perhaps it is better not to release the image to the Docker hub and wait until the next nightly (hopefully, the problem will be solved by the next version).

I am open to contribute, let me investigate a bit more about it.

[bonigarcia]

There are different projects that have implemented this feature. I think the process they follow is similar, i.e., installing a given Linux package (e.g., google-chrome-beta, google-chrome-unstable, etc) in the container and then push it to Docker Hub. There are public resources about it, namely:

Selenoid

As you probably know, Selenoid is a golang lightweight implementation of the Selenium Hub created and maintained by Aerokube. They also maintain a set of Docker images for the main browsers (e.g., Chrome, Firefox, Opera, Edge). I also asked this feature (i.e., publish beta/dev images in Docker Hub) some time ago (see issue). At the time of this writing, these images are not available yet in Docker Hub, although we can see how these beta/dev images images are built (see doc):

• Chrome.
• Firefox.
• Edge.
• Opera.

ElasTest

ElasTest was a EU-funded project I used to collaborate. Among other features, the ElasTest project maintained Docker images for Chrome and Firefox, including beta and dev channels (see Docker Hub). Although it seems these images are not maintained anymore, we can see the how these images were generated:

• GetVersions: script used to install and detect the versions of Chrome and Firefox beta/dev. This script is used in a Docker container (also public).

@diemol Do you think these approaches could fit in docker-selenium?

[diemol]

That makes sense, I think we could have separate GitHub workflows to build., test and push those images. Just bare in mind we won't offer Opera as the operadriver does not work in W3C mode currently.

[Luckless]

On the same vein, I can no longer tell which tags for the nodes are safe to use with a Selenium 3 grid. Seems recent ones past "latest" are all from beta 4 base but nothing indicates they are built on beta 4

ie 91.0 that my organization just tried to deploy and failed. Had to roll it back

[diemol]

On the same vein, I can no longer tell which tags for the nodes are safe to use with a Selenium 3 grid. Seems recent ones past "latest" are all from beta 4 base but nothing indicates they are built on beta 4

ie 91.0 that my organization just tried to deploy and failed. Had to roll it back

@Luckless, not at all. Images for Grid 3 are either with latest or the tag starts with 3. All other images are related to Grid 4, here you can find the explanation.

[j1elo]

A recent PSA in the discuss-webrtc forums has shown a perfect example to add into the Motivation section of this bug report:

• PSA: Firefox 96 contains major libwebrtc update. Please test!

Being able to run our Selenium tests with Firefox Nightly would have been great, in order to anticipate all problems that will probably arise from such big update on their side. So consider this a contextualized "+1"

[Brillie]

As this has gone quiet since last November... This would greatly help us. We'd like to be able to run our automated tests on the next stable version (i.e. Chrome Beta channel) in our framework, to test a version before Google pushes it to Stable; we need a standalone-chrome beta container for this. Another contextualized +1.

[diemol]

I agree it'd be great to have this, we "just" need help implementing it :)
We are very limited on time and people to work on this these days.

[jamesmortensen]

One thing I've learned by working on the multi-arch-images is that it may be easier to investigate this first on Firefox. The same geckodriver has been used since I believe Firefox 53 and still works on Firefox 98+. Mozilla hasn't released a geckodriver since September 2021. It's very likely that we'd only need to fetch the latest Firefox nightly in most cases and only need to deal with a geckodriver update perhaps once every 3 to 6 months.

To the best of my knowledge, Chrome, on the other hand, requires a chromedriver that matches the Chrome version within at least one major version. A quick Google search shows that the latest Google Chrome is v101 if installed using google-chrome-unstable. Also, the Chromedriver download page shows the latest chromedriver as v100. Maybe someone can build the Chrome images with those versions to validate if this is indeed true or not.

The corresponding versions for Chrome and Chromedriver can be passed into docker build as build arguments. Here they are in the code:

docker-selenium/NodeChrome/Dockerfile.txt

Line 18 in 0bdf761

${CHROME_VERSION:-google-chrome-stable} \

docker-selenium/NodeChrome/Dockerfile.txt

Line 36 in 0bdf761

ARG CHROME_DRIVER_VERSION

After building selenium/base and selenium/node-base, you should be able to build node-chrome and standalone-chrome using the below build commands:

$ cd NodeChrome && ./generate.sh unstable local-selenium SeleniumHQ && docker build --build-arg CHROME_VERSION=google-chrome-unstable --build-arg CHROME_DRIVER_VERSION=100.0.4896 -t local-selenium/node-chrome:unstable . 

$ cd Standalone && ./generate.sh StandaloneChrome node-chrome unstable local-selenium SeleniumHQ && cd ../StandaloneChrome && docker build -t local-selenium/standalone-chrome:unstable .

After building, start and run the container using:

$ docker run --rm -it -p 4444:4444 -p 7900:7900 -p 5900:5900 --shm-size 2g local-selenium/standalone-chrome:unstable

At the time of this writing, Chromedriver 100.0.4896 and Chrome 101.0.4929.5-1 are the latest. If someone tries to build it later, see the above resources in the second paragraph to derive what the latest versions are.

[jamesmortensen]

Some notes that may help with this implementation:

Obtaining browsers and drivers (for x86_64 platform only)

• Firefox Beta, Dev, and nightly images can be pulled from here: https://www.mozilla.org/en-US/firefox/all/#product-desktop-esr. Download links are obtained by hovering over the button and then right-click copy link. We currently don't include the beta channel in NodeFirefox's Dockerfile, but it can easily be added as an option.

• Whenever Chrome releases a new Beta major release version, they also release a chromedriver with the same version. Patches may be applied independently, so they have a download URL that works with the patch version removed. Details are here: https://chromedriver.chromium.org/downloads/version-selection. The version parsing is already handled in the Dockerfiles.

• I assume that Microsoft releases edgedriver whenever they release a patch update to Edge, looking at what's on their file server: https://msedgedriver.azureedge.net/. We can confirm whether or not that's true. Additionally, the Edge Dockerfile also contains the logic for building the container image with different release channels, such as

Possible build commands for Beta images:

The Dockerfiles already allow passing in a CHROME_VERSION and FIREFOX_VERSION at build time.

$ docker build --build-arg CHROME_VERSION=google-chrome-beta -t selenium/node-chrome:4.4.0-20220815-beta .

$ docker build --build-arg EDGE_VERSION=microsoft-edge-beta -t selenium/node-edge:4.4.0-20220815-beta .

Corresponding beta versions for Chrome, at the time of this writing:

$ docker run --rm -it selenium/node-chrome:4.4.0-20220815-chrome-beta google-chrome --version
Google Chrome 105.0.5195.28 beta

$ docker run --rm -it selenium/node-chrome:4.4.0-20220815-chrome-beta chromedriver --version
ChromeDriver 105.0.5195.19 (b9c217c128c16f53d12f9a02933fcfdec1bf49af-refs/branch-heads/5195@{#176})

Corresponding beta versions for Edge, at the time of this writing:

Beta msedgedriver version is different than Beta MS Edge browser version:

$ docker run --rm -it selenium/node-edge:4.4.0-20220815-beta microsoft-edge --version
Microsoft Edge 104.0.1293.44 beta

docker run --rm -it selenium/node-edge:4.4.0-20220815-beta msedgedriver --version
Microsoft Edge WebDriver 104.0.1293.54 (cf00dc8720d894eb5f0a1b840990bb9ecf8b3589)

Unexpectedly, Beta MS Edge browser is an older version than stable:

$ docker run --rm -it selenium/node-edge:4.4.0-20220812 microsoft-edge --version 
Microsoft Edge 104.0.1293.54

But the Beta edgedriver is newer than the stable edgedriver

$ docker run --rm -it selenium/node-edge:4.4.0-20220812 msedgedriver --version 
Microsoft Edge WebDriver 104.0.1293.47 (246533a39fec5a167e7729d997877afea075bd86)

$ docker run --rm -it selenium/node-edge:4.4.0-20220815-beta msedgedriver --version 
Microsoft Edge WebDriver 104.0.1293.54 (cf00dc8720d894eb5f0a1b840990bb9ecf8b3589)

Automation can be used in tagging:

CHROME_CHANNEL=$(docker run --rm -it selenium/node-chrome:4.4.0-20220815-chrome-beta google-chrome --version
Google Chrome 105.0.5195.28 beta | awk '{print $4}')
echo $CHROME_CHANNEL # outputs "beta"

Tagging Beta Images

And tags could look like this, if we follow the existing tagging paradigm:

• 105.0-beta
• 105.0-20220815-beta
• 105.0-chromedriver-105.0-beta
• 105.0-chromedriver-105.0-20220815-beta
• 105.0-chromedriver-105.0-grid-4.4.0-20220815-beta
• 105.0.5195.28-beta
• 105.0.5195.28-20220815-beta
• 105.0.5195.28-chromedriver-105.0.5195.28-20220815-beta
• 105.0.5195.28-chromedriver-105.0.5195.28-grid-4.4.0-20220815-beta
• 4.4.0-beta
• 4.4-beta
• 4-beta
• latest-beta

But this raises some questions regarding whether or not there are any challenges with maintaining the same level of tagging granularity with Chrome beta images. The number of tags for each Selenium release or browser update increases significantly. However, releases are managed by a bot, so this approach may not be problematic in most cases.

What about when Selenium 5.0-beta is released? Do we need to differentiate between Selenium-5.0-beta but with Chrome stable? How do we communicate this in a clear manner? Tagging does indeed get complicated when we consider that different components may be independently "beta" or "stable release".

Scope of feature request - Beta Chrome and Firefox only

My thought is to scope this feature request to just the Beta channels for Chrome and Firefox, since they're more likely to be more stable than the dev/nightly channels. If it works well, and if the community continues to request nightlies, we can then take that into consideration.

We'll need to do the same assessment on the Firefox beta and stable versions that we've done on Chrome and Edge. Edge seems to be rather inconsistent, so we may want to leave Edge out for now.

[jamesmortensen]

Modify Dockerfile for NodeFirefox to include FIREFOX_VERSION "beta-latest" as an option:

RUN FIREFOX_DOWNLOAD_URL=$(if [ $FIREFOX_VERSION = "latest" ] || [ $FIREFOX_VERSION = "beta-latest" ] || [ $FIREFOX_VERSION = "nightly-latest" ] || [ $FIREFOX_VERSION = "devedition-latest" ] || [ $FIREFOX_VERSION = "esr-latest" ]; then echo "https://download.mozilla.org/?product=firefox-$FIREFOX_VERSION-ssl&os=linux64&lang=en-US"; else echo "https://download-installer.cdn.mozilla.net/pub/firefox/releases/$FIREFOX_VERSION/linux-x86_64/en-US/firefox-$FIREFOX_VERSION.tar.bz2"; fi) \

Then build the image for Firefox beta as follows:

docker build --build-arg FIREFOX_VERSION=beta-latest -t selenium/node-firefox:4.4.0-20220815-beta .

Afterwards, we confirm beta-latest is newer than latest:

$ docker run --rm -it selenium/node-firefox:4.4.0-20220815-beta firefox --version
Mozilla Firefox 104.0b9

$ docker run --rm -it selenium/node-firefox:4.4.0-20220815 firefox --version 
Mozilla Firefox 103.0.2

Unlike the Chromium-based browsers, the geckodriver update-cycle is significantly longer, so we don't need to be as concerned with matching geckodriver versions with browser versions.

Based on this information, it seems that Chrome and Firefox beta could perhaps be supported on a trial basis?

[diemol]

Thank you for the comments and research, @jamesmortensen!

I believe we could offer the three browsers on beta and dev. If Edge is not consistent on their versioning, we cannot do much about it. And in general, we won't be offering support for beta and dev images, if there are any issues related to those browser versions, well... they are beta and dev versions, issues are expected.

Regarding, image tagging. When we were doing alpha, beta and rc releases for Selenium 4, this was the convention being used: 92.0-geckodriver-0.30-grid-4.0.0-rc-2-20210930. So, if we wanted to, we could add -beta or -dev at the end.

But for simplicity, I think we should just tag the images with beta and dev, nothing else. For example:

• node-firefox:beta
• standalone-firefox:beta
• node-chrome:dev
• standalone-chrome:dev
• etc...
  and the Selenium version will be whatever is being used in the latest release.

Also, we should generate this images periodically, like every 2 days or so. Automatically. I do not think we need to create a release tag for this. They simply get generated and pushed to Docker Hub.

What do you think?

[jamesmortensen]

Hi @diemol I think what you're proposing would be a lot simpler. We don't need a release and we also don't need to maintain historical beta versions. A rolling tag sounds a lot simpler to maintain.

I'll create another workflow, which runs on a schedule, in GitHub actions to build just the base, node-base, node-chrome/firefox/edge, and the standalones for those 3 browsers. If the tests pass, then I'll run another job to push them to Docker Hub.

I'll work on these changes.

[jamesmortensen]

Hi @diemol this is setup as follows:

• Matrix build with browsers (chrome,edge,firefox) and channels (dev,beta) with fail-fast disabled, meaning if one browser and channel combination has a blocker, the others can still deploy.
• Enabled for Chrome, Firefox, and Edge on Beta and Dev channels.
• Builds and deploys every 2 days on a schedule.
• Builds from existing selenium/node-base:latest for faster builds
• Smoke tests are completed prior to deployment
• Deployments are tagged as:
  -- selenium/node-chrome:dev
  -- selenium/node-chrome:beta
  -- selenium/node-firefox:dev
  -- selenium/node-firefox:beta
  -- selenium/node-edge:dev
  -- selenium/node-edge:beta
  -- selenium/standalone-chrome:dev
  -- selenium/standalone-chrome:beta
  -- selenium/standalone-firefox:dev
  -- selenium/standalone-firefox:beta
  -- selenium/standalone-edge:dev
  -- selenium/standalone-edge:beta

Unless there's any changes, I believe this is ready to be merged.