[🐛 Bug]: org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$RwtMl7Wf cannot be cast to org.openqa.selenium.devtools.HasDevTools

[jantekb]

What happened?

I am running latest selenium-docker image:

sudo docker run -d -p 4444:4444 --shm-size="2g" -e SCREEN_WIDTH=1280 -e SCREEN_HEIGHT=720 selenium/standalone-chrome:4.1.0-20211209

In my Java application I have the following code snippet that throws ClassCastException:

ChromeOptions options = new ChromeOptions();
String driverUrl = "http://127.0.0.1:4444/wd/hub";
WebDriver driver = new RemoteWebDriver(new URL(driverUrl), options);
driver = new Augmenter().augment(driver);
DevTools devTools = ((HasDevTools) driver).getDevTools();
devTools.createSession();

As soon as the HasDevTools cast is attempted, I am getting:

org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$RwtMl7Wf cannot be cast to org.openqa.selenium.devtools.HasDevTools: java.lang.ClassCastException: org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$RwtMl7Wf cannot be cast to org.openqa.selenium.devtools.HasDevTools

I have carefully reviewed all the similar issues, but none of them seems to have concluded on this particular bug.

My relevant grade dependency is:

implementation 'org.seleniumhq.selenium:selenium-java:4.1.0'
implementation 'org.seleniumhq.selenium:selenium-devtools-v94:4.1.0'

My JDK is:

OS Version: 4.14.252-195.483.amzn2.x86_64
Java Name: OpenJDK 64-Bit Server VM
Java Vendor: Oracle Corporation
Java Version: 9.0.4
Java VM Version: 9.0.4+11

How can we reproduce the issue?

ChromeOptions options = new ChromeOptions();
String driverUrl = "http://127.0.0.1:4444/wd/hub";
WebDriver driver = new RemoteWebDriver(new URL(driverUrl), options);
driver = new Augmenter().augment(driver);
DevTools devTools = ((HasDevTools) driver).getDevTools();
devTools.createSession();

Relevant log output

2021-12-10      17:04:26        UTC     comment server  INFO    200     -       Tricky cast     _defaultVHost_  -       -       30.728  -       -       -       -       -       -       -       -       -       -       -       -       -
       -       -       -       -       -       -       -       -       -       -       -       -
2021-12-10      17:04:26        UTC     comment server  ERROR   500     -       Error initializing stack: org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$RwtMl7Wf cannot be cast to org.openqa.selenium.devtools.HasDevTools: java.lang.ClassCastException: org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$RwtMl7Wf cannot be cast to org.openqa.selenium.devtools.HasDevTools|at com.mycompany.blah.Foo.lambda$new$3(Foo.java:291)|at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)|at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)|at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)|at java.base/java.lang.Thread.run(Unknown Source)|    _defaultVHost_  -       -       30.73   -

Operating System

Amazon Linux

Selenium version

4.1.0

What are the browser(s) and version(s) where you see this issue?

Chrome

What are the browser driver(s) and version(s) where you see this issue?

the one in your docker image

Are you using Selenium Grid?

4.1.0

[github-actions]

@jantekb, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[pujagani]

Thank you for providing the details. I have tried running the script mentioned above in my local maven project with Java Version: 9.0.4 and I am unable to reproduce the error after multiple attempts when running on IDE. If I use maven-jar-plugin then it works as expected. If I use maven-assembly-plugin without a custom assembly file then I see the error you are facing. But if I add a custom assembly file that prevents overwriting for metaInf-services, then it works as expected.
Can you please provide a sample project and instructions to run the same? It will help understand the project configuration, dependencies, and hopefully, reproduce the problem locally. Thank you! Appreciate it.

[pujagani]

The issue might be potentially similar to #10010, in terms of overwriting the meta-inf services. The Augmenter uses ServiceLoader to load and implement the interfaces. It uses AugmentProvider to load the classes, the implementation classes information is stored in the meta-inf services. Based on how the Java jar is bundled, this information might be overwritten by default when using maven-assembly-plugin. Hence, the class cast exception.

[hellfish46]

Hello, I have absolutely the same issue using Selenoid v1.10.7. I use jdk 11.0.4. I don't use maven-assembly-plugin.
I use plugins:

<build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.6.1</version>
                <configuration>
                    <fork>true</fork>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M4</version>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                    <suiteXmlFiles>${file}</suiteXmlFiles>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
            <plugin>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-maven</artifactId>
                <version>2.10.0</version>
                <configuration>
                    <reportVersion>2.8.1</reportVersion>
                </configuration>
            </plugin>
        </plugins>
    </build>

Also I use bytebuddy for Augmenter

<dependency>
    <groupId>net.bytebuddy</groupId>
    <artifactId>byte-buddy</artifactId>
    <version>1.12.4</version>
</dependency>

Maybe I need to add maven-assembly-plugin with specific config ?

[pujagani]

@hellfish46 Thank you for providing the details. How are you running the code? Are you running a java jar or running on IDE?

[hellfish46]

I run it on IDE:
IntelliJ IDEA 2021.1 (Community Edition)
Build #IC-211.6693.111, built on April 6, 2021
Runtime version: 11.0.10+9-b1341.35 x86_64.

[jantekb]

I don't use maven either, I use Gradle and shadowJar and in my final jar file these are the selenium related service entries:

META-INF\services\org.openqa.selenium.WebDriverInfo
META-INF\services\org.openqa.selenium.devtools.CdpInfo
META-INF\services\org.openqa.selenium.remote.AdditionalHttpCommands
META-INF\services\org.openqa.selenium.remote.AugmenterProvider
META-INF\services\org.openqa.selenium.remote.http.HttpClient$Factory
META-INF\services\org.openqa.selenium.remote.locators.CustomLocator
META-INF\services\org.openqa.selenium.remote.service.DriverService$Builder

Interestingly each one of these files contain the below single line, which does not match what I can see in the original jar files. Thanks for the hint, I'll investigate it and report back on my results

org.openqa.selenium.chrome.ChromeDriverInfo

[jantekb]

Confirming the fix: by adding the below to my build.gradle shadow related configuration, the issue is now gone because the services files are properly merged:

plugins {
    id 'java'
    id 'com.github.johnrengelman.shadow' version '5.2.0'
}

// dependencies, etc.

shadowJar  {
    mergeServiceFiles()
}

Thank you very much!

[hellfish46]

Can't understand how to fix this issue and what the problem was here

[diemol]

@hellfish46 Can you provide a GitHub repo with a project we can use to reproduce the issue?

[hellfish46]

@diemol Hello, tried to make it )
https://github.com/hellfish46/SeleniumIssue

Want to emphasize that this issue appears when I use Selenoid. When I use Selenium Grid standalone there was another issue - somethins with websokets connection. Maybe you already fixed it in #9803

[boris779]

@hellfish46 That was the root reason why I created #9803 because we are running a moon-cluster (kubernetes version of selenoid). To find a reason why it is not working I tried to get it run with the selenium-grid just to find out where the problem is.

Today I tried many versions of the grid, except the standalone solution I don't get the CDP running, and we really want to use that. Also tried the docker-selenium-grid version (with the docker-compose) but still get it not running :-(

I can provide a lot of examples, to help to fix it.

[diemol]

Thank you, @jantekb, for sharing how the issue got fixed!

@hellfish46 not sure how this can get done with Selenoid. You would need to ask them over there. We have tested this with Grid 4 and I believe some cloud vendors work as well.

@boris779, #9803 was fixed and I gave context on why it was failing. The key part to have it working with the official Docker images is to set the SE_NODE_GRID_URL with the public facing url of your Grid. Feel free to drop by our community channels and I'd be happy to help more over there.

Closing as the original reported issue is now fixed.

[jhainzlschmid]

Confirming the fix: by adding the below to my build.gradle shadow related configuration, the issue is now gone because the services files are properly merged:

plugins {
    id 'java'
    id 'com.github.johnrengelman.shadow' version '5.2.0'
}

// dependencies, etc.

shadowJar  {
    mergeServiceFiles()
}

Thank you very much!

For the people using Maven: You need to use the maven shade plugin (maven assembly won't work) and configure a ServicesResourceTransformer. Heres an example maven shade configuration for an executable uber jar:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>YourMainClass</mainClass>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.