[🐛 Bug]: Selenium Grid problem with processing requests from JDKHttpClient #14258
Comments
|
@kakliniew, thank you for creating this issue. We will troubleshoot it as soon as we can. Info for maintainersTriage this issue by using labels.
If information is missing, add a helpful comment and then
If the issue is a question, add the
If the issue is valid but there is no time to troubleshoot it, consider adding the
If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C),
add the applicable
After troubleshooting the issue, please add the Thank you! |
diemol
added
I-issue-template
|
Hi, @kakliniew. Either a complete code snippet and URL/HTML (if more than one file is needed, provide a GitHub repo and instructions to run the code), the specific versions used, or a more detailed description to help us understand the issue. Note: If you cannot share your code and URL/HTML, any complete code snippet and URL/HTML that reproduces the issue is good enough. Reply to this issue when all information is provided, thank you. |
|
@diemol ./gradlew test -D.cucumber.filter.tags=@greenDuring execution I receive the same error and session is stopped: If you need more details, please let us know. I will try other browsers/versioons on Monday. |
|
For the example project, the problem was solved by adding flags: |
|
Does the issue only happen on your Kubernetes environment? |
|
I dont know. I'm stuck in a place. GOAWAY is not random - it always throws an error in the same place for a given test. I tried different browsers and still the same thing. |
|
I tried with serenity 4.0.15 (first version with selenium 4.14.1 so jdkhttp by default) and it fails. Changed to 4.0.14 (selenium 4.13.0) and everything is fine. Added org.seleniumhq.selenium:selenium-http-jdk-client:4.13.0 and System.setProperty("webdriver.http.factory", "jdk-http-client") and it fails again. So this problem only occurs when using jdkhttpclient. I will try to reproduce it in my example project. |
kakliniew
changed the title
|
Very very strange... checkIfModalWindowIsPresent() is execution of JS scripts - for some reason twice the same script (using JavascriptExecutor). And strange steps time doesnt addup to execution time (but maybe thats bug) I tried to reproduce the problem in the new repo, but I am not able to. Any ideas? |
|
I found one function that, if commented out, makes the test go on. I don't know if whole test works yet, because it crashes on one assertion and I don't have the energy to fix it today, but when not commented out it crashes much earlier due to GOAWAY. When I look at the function, there is nothing special about it, I will analyse it tomorrow |
|
I am wondering about the http2 related error message, the selenium server should not use http2. You could try to set version http1.1 in the ClientConfig to see what happens and what error is raised. |
|
The main element of the mentioned function is findElements(), and then loop through the elements and “getAtrribute” for each element. I think the problem are some kind of limits. I tried to initializing webdriver in following way: Mentioned page is: @joerg1985 when I try to initialize webdriver like: Then I get: I tried with HTTP_2 too, but same result. |
|
Deployed new selenium grid with new helm chart and when I try to initialize the same way as above, then I get: I thought old helm chart might be a problem, because in my local cluster when I downgraded then I got this HTTP Connect timed out. |
|
Enabled debug logs for selenium hub/selenium edge node and during 3 tests I found 1500x The same debug info is available even when I run tests using selenium 4.12.0 I'm still unable to create webdriver with clientconfig version http 1_1. |
|
I was able to resolve the problem: By just removing TLS host for selenium-grid-ingress. After that change I was able to initialize driver with HTTP_1_1, and tests pass. |
|
@VietND96 have you seen this? |
|
It would be nice to have flag added like: HTTP2 is by default according to docs It seems that this problem with jdkhttpclient HTTP/2, Nginx GOAWAY is widely known and you need to handle it or use HTTP/1.1 (as described in the source reddit, but also in other places). |
|
Maybe a system property is not a bad idea. |
|
As my understanding, this case has the ingress proxy involves The error and you mentioned that
I think this relates to Server-side HTTPS enforcement through redirect - By default the controller redirects (308) to HTTPS if TLS is enabled for that ingress. In ingress-nginx docs, they also mentioned Proxy HTTP version that the Nginx reverse proxy will use to communicate with the backend. By default this is set to "1.1". So without redirect issue, HTTP version doesn't matter |
|
@VietND96 thanks for your answer. I wanted to set it up, but faced some problems. I will try again and let you know |
|
@VietND96 I configured TLS. Hub/Nodes are up and running. I used external secret. During test execution I received once again GOAWAY and there is a message in edge node's logs: Initialized driver with HTTP1_1 works fine. |
|
Yes, initialized the driver with HTTP1_1 in client config as I know it was supported via #12919 Besides that, I also tried to find some interesting info for your reference. In OpenJDK docs Introduction to the Java HTTP Client, I saw these statements
As I saw above, a comment also mentioned that the grid server side is not using HTTP/2, and suppose the client side is sending HTTP/2. That statement describes the correct behavior it should be. I also found https://stackoverflow.com/questions/67018337/is-there-any-way-to-handle-http-2-goaway-received-ioexception-in-httpclient-java Next, I found this https://stackoverflow.com/questions/55087292/how-to-handle-http-2-goaway-with-java-net-httpclient with these statements
I feel this is close to your setup, ingress nginx involves. I tried to search config key |
|
I tried setting these limits (keepalive_requests) for the local cluster (minikube) and started my local tests, but it didn't help - I couldn't reproduce the error. I can't really modify the nginx configuration in the ‘embedded cluster’, because it handles not only selenium grid and I can cause problems. JdkHttpClient by default sets http2, hence my problems when I don't explicitly specify it to use http1.1. Therefore system property would be a good solution. |
|
@VietND96 today I accidentally discovered that if I use the url for selenium grid, but protocol This makes a lot of sense, but I didn't think of it before. |
|
Not sure my understanding is correct, based on your setup mentioned before with HTTPS (it requires TLS version as well as HTTP version, which is based on the ingress controller). Probably, your setup exposes JDKHttpClient also mentioned something if H2 (needs to try and see how it helps)
|
|
At this point, setting an use-http2 to "false" looks like a good solution. I did it today and the tests work in the default config (without forcing in the code http.version http_1_1 and using Thank you so much for all your replies, they have been very helpful and motivating, as at one point I was losing hope a bit! |
…sable HTTP/2 (#2328) Fix: #2195 Docs for SeleniumHQ/selenium#14258 Signed-off-by: Viet Nguyen Duc <[email protected]>
|
This issue has been automatically locked since there has not been any recent activity since it was closed. Please open a new issue for related bugs. |
What happened?
When I use serenity 4.0.1 (selenium 4.12.0) tests work correctly, but when I try to upgrade (e.g. 4.1.4 - selenium 4.18.1) then I receive this error after few seconds (always in same step) of execution (execution works fine but then suddenly GOAWAY).
I tried debugging, changing selenium grid/hub/node versions. I tried changing selenium. I noticed that the problem occurs when I use selenium with jdkhttpclient (so 4.14 and higher). In addition, when I did a downgrade to edge node 4.15.0 I got an additional stacktrace in the logs
How can we reproduce the issue?
Serenity 4.1.4 + jdk11 Selenium grid 4.15.0 deployed in kubernetes + edge node 4.15.0Relevant log output
Operating System
windows/unix
Selenium version
4.18.1
What are the browser(s) and version(s) where you see this issue?
Tried 4.15.0, 4.21.0, 4.22.0
What are the browser driver(s) and version(s) where you see this issue?
all of them but stacktrace in 4.15.0 only
Are you using Selenium Grid?
4.15.0/4.21.0
Connected issue
SeleniumHQ/docker-selenium#2195
The text was updated successfully, but these errors were encountered: