Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[🐛 Bug]: HasAuthentication is not working for Chromium browsers when they are executed in Selenium Grid #9934

Closed
NikolayStanoev opened this issue Oct 15, 2021 · 19 comments
Labels

Comments

@NikolayStanoev
Copy link

What happened?

I am able to use register method from HasAuthentication interface using chromium drivers running on local machine. But when i try to use chromium browser (edge/chrome) which is part of selenium grid I am getting the following error when I try to use HasAuthentication interface.

Code where the test fails:
((HasAuthentication) webDriver).register(() -> new UsernameAndPassword("admin", "admin"));

Error:

java.lang.ClassCastException: class org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$gPlLDnEY cannot be cast to class org.openqa.selenium.HasAuthentication (org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$gPlLDnEY is in unnamed module of loader net.bytebuddy.dynamic.loading.ByteArrayClassLoader @77681ce4; org.openqa.selenium.HasAuthentication is in unnamed module of loader 'app')

How can we reproduce the issue?

How to reproduce the problem:
* Run a selenium grid (standalone / hub&node mode - does not matter)
`java -jar selenium-server.jar standalone --detect-drivers true`
* Execute the code below using selenium-java 4

Code Example
`WebDriver webDriver = RemoteWebDriver
                 .builder()
                 .augmentUsing(new Augmenter())
                 .oneOf(new ChromeOptions())
                 .address(new URL("http://localhost:4444"))
                 .build();
((HasAuthentication) webDriver).register(() -> new UsernameAndPassword("admin", "admin"));
webDriver.get("https://the-internet.herokuapp.com/basic_auth");`

Relevant log output

java.lang.ClassCastException: class org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$E2NldVid cannot be cast to class org.openqa.selenium.HasAuthentication (org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$E2NldVid is in unnamed module of loader net.bytebuddy.dynamic.loading.ByteArrayClassLoader @2d38edfd; org.openqa.selenium.HasAuthentication is in unnamed module of loader 'app')

	at com.isobar.commerce.selenium.tests.FeaturesTests.test(FeaturesTests.java:175)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.testng.TestRunner.privateRun(TestRunner.java:764)
	at org.testng.TestRunner.run(TestRunner.java:585)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
	at org.testng.TestNG.runSuites(TestNG.java:1069)
	at org.testng.TestNG.run(TestNG.java:1037)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)

Operating System

Windows 10 / MacOS BigSur 11.3.1

Selenium version

4.0.0

What are the browser(s) and version(s) where you see this issue?

Chrome 94

What are the browser driver(s) and version(s) where you see this issue?

chromedriver 94.0.4606.61

Are you using Selenium Grid?

Yes - 4.0.0

@NikolayStanoev
Copy link
Author

NikolayStanoev commented Oct 15, 2021

Issue is reproducible with the following code as well

WebDriver webDriver = new RemoteWebDriver(new URL("http://localhost:4444"), new ChromeOptions())

webDriver = new Augmenter().augment(webDriver);

((HasAuthentication) webDriver).register(() -> new UsernameAndPassword("admin", "admin"));

webDriver.get("https://the-internet.herokuapp.com/basic_auth");

@jantekb
Copy link

jantekb commented Oct 18, 2021

Same applies to the HasDevTools interface in a very similar scenario

@pujagani
Copy link
Contributor

Thank you for providing the details. I was able to reproduce the error you are facing. The RemoteDriver needs to be augmented with the HasDevTools interface first. Then it needs to be augmented with the HasAuthentication interface. This gets the RemoteWebDriver ready to handle and intercept authentication requests. Adding a working example below. I recommend going through it and trying it out and providing feedback.

public class ChromeDevToolsAugmenterAuthentication {

  public static void main(String[] args) throws MalformedURLException, InterruptedException {
    ChromeOptions chromeOptions = new ChromeOptions();
    WebDriver driver = new RemoteWebDriver(new URL("http://<grid>-url"), chromeOptions);

    Augmenter augmenter = new Augmenter();

    driver = augmenter.augment(driver);

    DevTools devTools = ((HasDevTools) driver).getDevTools();
    devTools.createSession();

    driver = augmenter.
      addDriverAugmentation("chrome", HasAuthentication.class, (caps, exec) -> (whenThisMatches, useTheseCredentials) -> devTools.getDomains().network().addAuthHandler(whenThisMatches, useTheseCredentials)).augment(driver);

    ((HasAuthentication) driver).register(UsernameAndPassword.of("admin", "admin"));
    driver.get("https://the-internet.herokuapp.com/basic_auth");
    System.out.println(driver.findElement(By.tagName("p")).getText());
    driver.quit();
  }
}

@NikolayStanoev
Copy link
Author

Hi @pujagani ,

Thank you for the help. I can confirm that code example above solved the problem.

@pujagani
Copy link
Contributor

Thank you for trying it out. Will take this opportunity to add the example to the Selenium documentation.

@NikolayStanoev
Copy link
Author

Hello @pujagani ,

Based on the solution above driver needs to be augmented in order to get access to HasDevTools interface. After that devtools session must be created and then driver must be augmented one more time. I guess same should be applied for the rest of the interfaces part of the ChromiumDriver.

Are there any plans to make the second augmentation one idea easier by providing driver + active devtools instance and this to unlock all other interfaces?

PS: Feel free to add the example above in the documentation. It would help a lot of people!

@pujagani pujagani reopened this Oct 18, 2021
@pujagani
Copy link
Contributor

pujagani commented Oct 18, 2021

@NikolayStanoev You make a valid case. The whole idea of Augmentation is to be able to add interfaces and implementation as required. A similar error is faced while using HasLogEvents interface as well. Since most CDP-related interfaces require HasDevTools, I think it might be a good idea to chain this in a manner that is easier to use. I will look into how we can achieve that and bring it up with the core contributors to identify the best way to solve this.

@titusfortner
Copy link
Member

Augmentation is only needed for RemoteWebDriver instances because the applicable browser drivers themselves implement the interface. This doesn't actually address making the second call easier, but with respect to getting an augmented driver in the first place, I wanted to point out that you can use RemoteWebDriverBuilder to automatically augment. Here's an example:

https://github.com/saucelabs-training/demo-java/blob/docs-1.2/selenium-examples/src/test/java/com/saucedemo/selenium/se4newfeatures/RemoteWebDriverBuilderTest.java

Just thought I'd mention it since there are so few resources out there right now discussing the Augmenter, and it's super powerful.

@pujagani
Copy link
Contributor

pujagani commented Oct 20, 2021

Thank you @titusfortner for pointing this out. It helped me get some insights into how certain AugmentationProviders are loaded by the Java class loader. Also, the code example now looks a tad cleaner :)
The updated example is shared below @NikolayStanoev.

public class ChromeDevToolsAugmenterAuthentication {

  public static void main(String[] args) throws MalformedURLException, InterruptedException {
    ChromeOptions chromeOptions = new ChromeOptions();
    WebDriver driver = RemoteWebDriver.builder()
      .oneOf(chromeOptions)
      .address("http://localhost:4444")
      .build();

    DevTools devTools = ((HasDevTools) driver).getDevTools();
    devTools.createSession();

    Augmenter augmenter = new Augmenter();

    driver = augmenter
      .addDriverAugmentation("chrome",
                             HasAuthentication.class,
                             (caps, exec) -> (whenThisMatches, useTheseCredentials) ->
                               devTools.getDomains()
                                 .network()
                                 .addAuthHandler(whenThisMatches, useTheseCredentials))
      .augment(driver);

    ((HasAuthentication) driver).register(UsernameAndPassword.of("admin", "admin"));
    driver.get("https://the-internet.herokuapp.com/basic_auth");
    System.out.println(driver.findElement(By.tagName("p")).getText());
    driver.quit();
  }
}
public class ChromeDevToolsAugmenterAuthenticationSimplify {

  public static void main(String[] args) throws MalformedURLException, InterruptedException {
    ChromeOptions chromeOptions = new ChromeOptions();
    AtomicReference<DevTools> devToolsAtomicReference = new AtomicReference<>();

    WebDriver driver = RemoteWebDriver.builder()
      .augmentUsing(new Augmenter().addDriverAugmentation("chrome",
                                                          HasAuthentication.class,
                                                          (caps, exec) -> (whenThisMatches, useTheseCredentials) -> {
                                                            devToolsAtomicReference.get().createSessionIfThereIsNotOne();
                                                            devToolsAtomicReference.get().getDomains()
                                                              .network()
                                                              .addAuthHandler(whenThisMatches, useTheseCredentials);
                                                          }))
      .oneOf(chromeOptions)
      .address("http://localhost:4444")
      .build();

    devToolsAtomicReference.set(((HasDevTools) driver).getDevTools());

    ((HasAuthentication) driver).register(UsernameAndPassword.of("admin", "admin"));
    driver.get("https://the-internet.herokuapp.com/basic_auth");
    System.out.println(driver.findElement(By.tagName("p")).getText());
    driver.quit();
  }
}

@NikolayStanoev
Copy link
Author

Hi @pujagani,

I am not able to execute the example above.

It fails during compilation on that line kind.initializeListener(finalDriver);
java: incompatible types: org.openqa.selenium.WebDriver cannot be converted to org.openqa.selenium.logging.HasLogEvents

If I update the line to kind.initializeListener((HasLogEvents) finalDriver); and execute the test it fails with

Exception in thread "main" java.lang.ClassCastException: class org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$jkzQPf7S cannot be cast to class org.openqa.selenium.logging.HasLogEvents (org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$jkzQPf7S is in unnamed module of loader net.bytebuddy.dynamic.loading.ByteArrayClassLoader @369c9bb; org.openqa.selenium.logging.HasLogEvents is in unnamed module of loader 'app')
	at com.isobar.commerce.selenium.tests.ChromeDevToolsAugmenterDOMMutation$1.onLogEvent(ChromeDevToolsAugmenterDOMMutation.java:41)
	at org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$jkzQPf7S$ByteBuddy$i8qYic92.onLogEvent(Unknown Source)
	at com.isobar.commerce.selenium.tests.ChromeDevToolsAugmenterDOMMutation.main(ChromeDevToolsAugmenterDOMMutation.java:48)

The full code with all imports looks this way:

import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.HasDevTools;
import org.openqa.selenium.devtools.events.DomMutationEvent;
import org.openqa.selenium.logging.EventType;
import org.openqa.selenium.logging.HasLogEvents;
import org.openqa.selenium.remote.Augmenter;
import org.openqa.selenium.remote.RemoteWebDriver;

import java.net.MalformedURLException;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicReference;

import static org.openqa.selenium.devtools.events.CdpEventTypes.domMutation;

public class ChromeDevToolsAugmenterDOMMutation {

    public static void main(String[] args) throws MalformedURLException, InterruptedException {
        ChromeOptions chromeOptions = new ChromeOptions();

        WebDriver driver = RemoteWebDriver.builder()
                .oneOf(chromeOptions)
                .address("http://localhost:4444")
                .build();

        AtomicReference<DomMutationEvent> seen = new AtomicReference<>();
        CountDownLatch latch = new CountDownLatch(1);

        WebDriver finalDriver = driver;
        Augmenter augmenter = new Augmenter();
        driver = augmenter.
                addDriverAugmentation("chrome", HasLogEvents.class, (caps, exec) -> new HasLogEvents() {
                    @Override
                    public <X> void onLogEvent(EventType<X> kind) {
                        kind.initializeListener((HasLogEvents) finalDriver);
                    }
                }).augment(driver);

        DevTools devTools = ((HasDevTools) driver).getDevTools();
        devTools.createSession();

        ((HasLogEvents) driver).onLogEvent(domMutation(mutation -> {
            seen.set(mutation);
            System.out.println(mutation.getAttributeName());
            System.out.println(mutation.getCurrentValue());
            latch.countDown();
        }));

        driver.get("https://www.google.com");
        WebElement span = driver.findElement(By.cssSelector("span"));

        ((JavascriptExecutor) driver)
                .executeScript("arguments[0].setAttribute('cheese', 'gouda');", span);

        latch.await();
        driver.quit();
    }
}

Do you see a problem in the code above or something that I am missing?

@pujagani
Copy link
Contributor

pujagani commented Oct 21, 2021

Thank you for trying @NikolayStanoev. Sincere apologies, I was working on fixing the augmented chaining bit and I confused the example of HasAuthentication with HasLogEvents and shared that. I have updated the comment with a valid example. Would appreciate it if you could have a look at that, please. Thank you!

The error you faced Exception in thread "main" java.lang.ClassCastException: class is a known one for HasLogEvents and was recently fixed by a25ad32. It will be available in the next release. With the recent changes in a25ad32, the example below would be a valid one for HasLogEvents :

public class ChromeDevToolsAugmenterDOMMutation {

  public static void main(String[] args) throws MalformedURLException, InterruptedException {
    ChromeOptions chromeOptions = new ChromeOptions();
    AtomicReference<DomMutationEvent> seen = new AtomicReference<>();
    AtomicReference<WebDriver> augmentedDriver = new AtomicReference<>();
    CountDownLatch latch = new CountDownLatch(1);

    Augmenter augmenter = new Augmenter();

    WebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444"), chromeOptions);
    driver = augmenter.
      addDriverAugmentation("chrome", HasLogEvents.class, (caps, exec) -> new HasLogEvents() {
        @Override
        public <X> void onLogEvent(EventType<X> kind) {
          kind.initializeListener(augmentedDriver.get());
        }
      }).augment(driver);

    if (driver instanceof HasLogEvents) {
      augmentedDriver.set(driver);
    } else {
      System.out.println("Error error");
    }

    ((HasLogEvents) driver).onLogEvent(domMutation(mutation -> {
      seen.set(mutation);
      System.out.println(mutation.getAttributeName());
      System.out.println(mutation.getCurrentValue());
      latch.countDown();
    }));

    driver.get("https://www.google.com");
    WebElement span = driver.findElement(By.cssSelector("span"));

    ((JavascriptExecutor) driver)
      .executeScript("arguments[0].setAttribute('cheese', 'gouda');", span);

    latch.await();
    driver.quit();
  }
}

@pujagani
Copy link
Contributor

Hello @pujagani ,

Based on the solution above driver needs to be augmented in order to get access to HasDevTools interface. After that devtools session must be created and then driver must be augmented one more time. I guess same should be applied for the rest of the interfaces part of the ChromiumDriver.

Are there any plans to make the second augmentation one idea easier by providing driver + active devtools instance and this to unlock all other interfaces?

PS: Feel free to add the example above in the documentation. It would help a lot of people!

I looked into this in-depth and what I understand is, there are a set of interfaces (Example: HasDevTools), that are autoloaded (Hence, new Augmenter.augment() works for them) since they don't depend on other any other interface implementations. Few interfaces like HasAuthentication and HasLogEvents depend on the implementation of other interfaces like HasDevTools, and hence need to be explicitly added with implementation details. These implementations may directly or indirectly require HasDevTools methods. While you make a valid point, as shown in the updated examples, augmenting twice is not necessary.

From this exercise, it is definitely clear that in-depth documentation for Augmenter and how powerful it is is required. Will get started on the same.

@NikolayStanoev
Copy link
Author

Hi @pujagani ,

I can confirm that ChromeDevToolsAugmenterAuthenticationSimplify example works fine and looks simpler for sure.

Thank you a lot for spending time to update the documentation with those Augmenter implementations.

@nieperdragon
Copy link

Hi @NikolayStanoev

I see from your comment above that you have ChromeDevToolsAugmenterAuthenticationSimplify working.
I have poured over this thread for a few hours now but try as I might I simply cannot get it working.

With both ChromeDevToolsAugmenterAuthentication and ChromeDevToolsAugmenterAuthenticationSimplify my browser launches (on my Selenoid instance) but then throws the exception

Exception in thread "main" java.lang.ClassCastException: class org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$AFV6HtIP cannot be cast to class org.openqa.selenium.devtools.HasDevTools (org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$AFV6HtIP is in unnamed module of loader net.bytebuddy.dynamic.loading.ByteArrayClassLoader @7923f5b3; org.openqa.selenium.devtools.HasDevTools is in unnamed module of loader 'app') at testMain.ChromeDevToolsAugmenterAuthenticationSimplify.main(ChromeDevToolsAugmenterAuthenticationSimplify.java:41)

I'm wondering if you might be able to suggest what could be the issue I'm missing.

@NikolayStanoev
Copy link
Author

Hi @nieperdragon ,

I haven`t tried with Selenoid instance. Maybe problem comes from there.

Both solutions - ChromeDevToolsAugmenterAuthentication and ChromeDevToolsAugmenterAuthenticationSimplify works fine when they are executed against selenium grid in standalone / hub&node mode.

Can you share the code that you are using?

@nieperdragon
Copy link

nieperdragon commented Nov 3, 2021

@NikolayStanoev

So it's the same code as above with a couple of changes (some required Chrome Options) and of course the URL.
Theoretically Selenoid should just be passing on to the browser container however I will try setting up a grid in order to eliminate/confirm.

`
public class ChromeDevToolsAugmenterAuthenticationSimplify {

  public static void main(String[] args) throws MalformedURLException, InterruptedException {
    ChromeOptions chromeOptions = new ChromeOptions();
    chromeOptions.setCapability("browserName", "chrome");
	chromeOptions.setCapability("browserVersion", "90.0");
	chromeOptions.setCapability("selenoid:options", Map.<String,Object>of("enableVNC",true,"enableVideo",true));
	
    AtomicReference<DevTools> devToolsAtomicReference = new AtomicReference<>();
    
    WebDriver driver = RemoteWebDriver.builder()
    	      .augmentUsing(new Augmenter().addDriverAugmentation("chrome",
    	                                                          HasAuthentication.class,
    	                                                          (caps, exec) -> (whenThisMatches, useTheseCredentials) -> {
    	                                                            devToolsAtomicReference.get().createSessionIfThereIsNotOne();
    	                                                            devToolsAtomicReference.get().getDomains()
    	                                                              .network()
    	                                                              .addAuthHandler(whenThisMatches, useTheseCredentials);
    	                                                          }))
    	      .oneOf(chromeOptions)
    	      .address("http://localhost:4444/wd/hub")
    	      .build();
	
    devToolsAtomicReference.set(((HasDevTools) driver).getDevTools());
	

    ((HasAuthentication) driver).register(UsernameAndPassword.of("admin", "admin"));
    driver.get("https://the-internet.herokuapp.com/basic_auth");
    System.out.println(driver.findElement(By.tagName("p")).getText());`

@pujagani
Copy link
Contributor

pujagani commented Nov 8, 2021

@nieperdragon Based on what is discussed, I am sensing it could be potentially due to a Java version or due to the docker usage. It sounds very similar to #9803.

@nieperdragon
Copy link

Thank you @pujagani
I'm currently not able to try this out properly (for many reasons including tightened company restrictions and a dead computer :-( ). However I appreciate your input and I'll check out your suggestion when I'm in a position to look at it properly.

@github-actions
Copy link

github-actions bot commented Jan 2, 2022

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked and limited conversation to collaborators Jan 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

5 participants