Queue size is always high and the grid is drastically underutilized

[zzvara]

🐛 Bug Report

This is the average utilization and queue size at all times on the grid:

Sessions wait about 20 seconds in the queue, seemingly for no reason.

This problem is not present with grid version: 4.0.0-beta-4-prerelease-20210527, but present in the latest release 4.0.0-rc-1-prerelease-20210618.

To Reproduce

Detailed steps to reproduce the behavior:

Deploy the grid with the following configuration:

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: selenium-grid-advanced
  namespace: production
spec:
  releaseName: selenium-grid-advanced
  chart:
    # https://github.com/zzvara/docker-selenium/tree/selenium-grid-helm-chart-0.2.0
    repository: redacted
    name: selenium-grid
    version: 0.2.1
  values:
    global:
      seleniumGrid:
        imageTag: "4.0.0-rc-1-prerelease-20210618"
        nodesImageTag: "4.0.0-rc-1-prerelease-20210618"
    isolateComponents: false
    chromeNode:
      replicas: 60
      resources:
        limits:
          cpu: '1'
          memory: 2000Mi
        requests:
          cpu: '1'
          memory: 2000Mi
    firefoxNode:
      replicas: 0
    operaNode:
      replicas: 0
    hub:
      imageTag: "4.0.0-rc-1-prerelease-20210618"
      resources:
        requests:
          memory: "32Gi"
        limits:
          memory: "32Gi"
      extraEnvironmentVariables:
        - name: JAVA_OPTS
          # @note "POOL_MAX" not confirmed that it is working.
          value: "-Xmx20g -Xms2g -XX:+UseG1GC -DPOOL_MAX=512"
        - name: SE_OPTS
          value: "--max-threads 100 --allow-cors true"
        - name: SE_SESSION_REQUEST_TIMEOUT
          value: "30"

Expected behavior

A clear and concise description of what you expected to happen.

The queue should be close to 0 and concurrency should be high. Previous beta versions did not have this issue.

Test script or set of commands reproducing this issue

Unfortunately, scripts can not be disclosed, however, we reproduced the same problem with multiple tests/applications.

Environment

OS: Linux sigma02 5.10.43-flatcar #1 SMP Tue Jun 15 19:46:57 -00 2021 x86_64 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz GenuineIntel GNU/Linux
Grid version:
Browser: Chrome
Browser version: v.91.0
Selenium Grid version (if applicable): Grid version: 4.0.0-rc-1 (revision 23ece4f)

[Alex-Rod]

I'm experiencing the same issue. The queue grows over time, sessions timeout, driver fail to quit, and new drivers fail to be created.

[diemol]

Thank you for reporting this, I was able to reproduce it.

I am having a look soon.

[diemol]

@zzvara thank you for reporting this, we believe it has been fixed and we just did a pre-release.

Please let us know if things work as expected by commenting here, if a problem related to this issue comes up while checking the pre-release version, we can reopen this issue.

[zzvara]

It did improve. I don't know whether it is the same as it was before. @Alex-Rod what is your exp?

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.