[🐛 Bug]: Unable to register node to hub in Kubernetes Env #1645
Labels
Comments
|
Can you share the Hub logs in Debug mode? |
|
Hi, here is the full logs. I've used env variable --log-level ALL |
|
I do not see the Distributor getting the message from the Node. Probably the ports are not really open. Check ports 4442, 4443, and 4444 in the Hub are open and reachable from the Node. |
|
I did verify that all ports are open in hub and reachable from the node pod. Please find the Node Pod logs below. |
|
Maybe it can help, we encountered this issue when we had our kubernetes grid in 2 different servers. So we decided to enlarge 1 server and put the whole kubernetes there and that seemed to fix the issue. Issue might be happening when chrome-nodes try to register in a hub that its in a diferent server but same kubernetes grid |
|
Thanks for your inputs Mark. Appreciate it!! I was able to register the linux based nodes(Chrome, Firefox & EDGE) without any issue with in Kubernates cluster . I've hosted grid in single server and used NodePort as service to access cluster within Kubernates and outside of kubernates & exposed the required ports of hub 4444, 4443 & 4442. I'm able to register node b/w pods with in Kubernates cluster. When i tried to register windows node to Kubernates cluster using jar file & getting similar error. I was trying to add windows node to Grid which is hosted in Kubernates. java -jar selenium-server-4.4.0.jar node --publish-events tcp://:31654 --subscribe-events tcp://:31697 I verified that all nodeports are exposed to external traffic and not sure the root cause of this issue. Am i missing any arguments in order to register node in grid. Can some one please take a look at this issue. Thanks |
|
Hello, I was getting exact same error in a linux server (tried directly running from java, from docker, from docker compose) Then I saw the similarity with your error and my error logs. We had another server which was not having the same issues, and the difference was Dynatrace: one had Dynatrace and the other did not. And the one which had Dynatrace was having exact this same issue and error logs. |
|
Hi @eowoyn, in case server had Dynatrace, did Grid setup enable the tracing feature? Your configs contain something like |
|
Hi no my configs contain no setting like that. But it seems Dynatrace is doing that any way. Now we have also tested disabling Java Opentelemetry feature on the selenium process on the oneagent settings. That also solved the problem. |
VietND96
changed the title
|
@eowoyn, thank you for your confirmation. May I know the way you did for chromeNode:
extraEnvironmentVariables:
- name: SE_JAVA_OPTS
value: "-Dotel.javaagent.enabled=false -Dotel.metrics.exporter=none -Dotel.sdk.disabled=true" |
|
@VietND96 Hello, we were setting up this via the dynatrace admin window. But apparently one can also set the environment variable SE_ENABLE_TRACING to false for the selenium hub. In docker run, it is by And I had contact with Dynatrace support, they have verified this bug, and they are publishing a fix to this issue in the coming Oneagent release. |
|
@eowoyn , Can you please let us know the version of the Dynatrace that was causing the error. Also, Can you please share us the ticket number of the bug details that Dynatrace is working on. I have run into similar kind of issue. If you could share few detailed information about the Dynatrace issue and ticket number and fix suggested, it would be helpful. |
|
@Thomas-Personal Hi Thomas, from Dynatrace: "targeting fix with Oneagent 1.281 (early-mid January release). The current workaround is to turn OTel for selenium off by setting SE_ENABLE_TRACING environment variable to false". So the workaround is nothing but I have already mentioned previously. I hope this helps. |
|
Thank you Eowoyn. I have tried that already. But i will check it once again . Also I had tried by disabling the monitoring also for our namesapce from dynatrce montoring rules. It didnt work out. Still nodes are not getting registered to the hub. if possible, can you please explain the rootcuase and bug wit Dynatrace a little more. Is oneagent blocking any defualt port that are being used by selenium hub and node? Few more additional details would be really helpful in working with my dynatrace team |
|
@Thomas-Personal Hi Thomas, the explanation from Dynatrace: "Selenium has built its own framework around OpenTelemetry with custom context handling, which does conflict with Dynatrace OTel instrumentation." So I dont think that this is anything in the port level. It is in the java implementation, so this is why when we disable Java opentelemetry feature on the hub via setting SE_ENABLE_TRACING environment variable to false then the issue is no longer reproducable in our case. So are you experiencing exact this exception "java.lang.IllegalStateException: Context has not been changed", if so, please make sure you implement the workaround as instructed, then it should be solved, I assume :) |
|
Since Selenium 4.18.0 the workaround by setting SE_ENABLE_TRACING=false doesn't seem to work anymore. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
I'm trying to deploy seleniumgrid 4 in Openshift (Build on top of Kubernates). I created separate deployment for hub and chrome node. Also, Created services for Hub & chrome node in order to open communicate b/w pods. Hub is up and running without any issues. However, unable to register the nodes to hub. When i enable the FINE logs, i couldn't find much information about the issue with registering node. Please find the error logs below.
I'm observing this issue in both 4.2.2 & 4.3.0 version. I was using Docker images with 4.3.0 or 4.2.2 tags.
Note: I did verify network polices inorder to communicate b/w pods in Openshift Namespace.
Command used to start Selenium Grid with Docker
I did deploy the services using yml file and referred the helm chart repo to deploy it openshift environment. I make sure node ports are exposed properly in order communicate b/w pods.Relevant log output
Operating System
rhel7
Docker Selenium version (tag)
4.3.0
The text was updated successfully, but these errors were encountered: