Angular 14 karma gets stuck after executing tests inside Gitlab Kubernetes Pod runner for ChromeHeadless and does not exit

[MichaelPeter]

I checked already the troubleshooting guide. We recently upgraded from angular 12 to angular 14 and since then our pipline gets stuck on testing inside the gitlab pipline after executing all tests, but not on our local maschines.

Now if we execute ng test on our pcs, the testsuite executes, but if our CI pipline in Gitlab (which provisions Kubernets Pods) executes the tests it gets stuck after executing all tests and does not exit anymore (we waited up to an hour)

We have a angular solution with about 10 projects and every project has its own karma config, which are all quite simular.
If we even remove the tests for the last project still the same error happens.

Here is the end of the debug log.

22 06 2022 14:52:58.823:DEBUG [launcher]: google-chrome --user-data-dir=/tmp/karma-70992357 --enable-automation --no-default-browser-check --no-first-run --disable-default-apps --disable-popup-blocking --disable-translate --disable-background-timer-throttling --disable-renderer-backgrounding --disable-device-discovery-notifications http://localhost:9876/?id=70992357 --headless --disable-gpu --disable-dev-shm-usage --remote-debugging-port=9222
22 06 2022 14:52:59.144:DEBUG [web-server]: servingXXX/node_modules/karma/static/client.html
22 06 2022 14:52:59.152:DEBUG [web-server]: serving: XXX/node_modules/karma/static/karma.js
22 06 2022 14:52:59.175:DEBUG [karma-server]: A browser has connected on socket 6uyWphyJ9WJ_onP_AAAt
22 06 2022 14:52:59.177:DEBUG [web-server]: upgrade /socket.io/?EIO=4&transport=websocket&sid=lGnTmlqb2AH09oXjAAAs
22 06 2022 14:52:59.184:DEBUG [Chrome Headless 103.0.5060.53 (Linux x86_64)]: undefined -> CONNECTED
22 06 2022 14:52:59.184:INFO [Chrome Headless 103.0.5060.53 (Linux x86_64)]: Connected on socket 6uyWphyJ9WJ_onP_AAAt with id 70992357
22 06 2022 14:52:59.185:DEBUG [launcher]: BEING_CAPTURED -> CAPTURED
22 06 2022 14:52:59.185:DEBUG [launcher]: ChromeHeadless (id 70992357) captured in 0.366 secs
22 06 2022 14:52:59.185:DEBUG [Chrome Headless 103.0.5060.53 (Linux x86_64)]: CONNECTED -> CONFIGURING

...cut out..

hrome Headless 103.0.5060.53 (Linux x86_64): Executed 0 of 0 SUCCESS (0 secs / 0 secs)
22 06 2022 14:52:59.491:DEBUG [Chrome Headless 103.0.5060.53 (Linux x86_64)]: EXECUTING -> CONNECTED
Chrome Headless 103.0.5060.53 (Linux x86_64): Executed 0 of 0 SUCCESS (0.003 secs / 0 secs)
22 06 2022 14:52:59.492:DEBUG [launcher]: CAPTURED -> BEING_KILLED
22 06 2022 14:52:59.492:DEBUG [launcher]: BEING_KILLED -> BEING_FORCE_KILLED
22 06 2022 14:52:59.492:DEBUG [Chrome Headless 103.0.5060.53 (Linux x86_64)]: CONNECTED -> DISCONNECTED
TOTAL: 0 SUCCESS
22 06 2022 14:52:59.493:DEBUG [karma-server]: Run complete, exiting.
22 06 2022 14:52:59.493:DEBUG [launcher]: Disconnecting all browsers
22 06 2022 14:52:59.493:DEBUG [launcher]: BEING_FORCE_KILLED -> BEING_FORCE_KILLED
✔ Browser application bundle generation complete.
22 06 2022 14:52:59.494:DEBUG [proxy]: Destroying proxy agents
=============================== Coverage summary ===============================
Statements   : Unknown% ( 0/0 )
Branches     : Unknown% ( 0/0 )
Functions    : Unknown% ( 0/0 )
Lines        : Unknown% ( 0/0 )
================================================================================
22 06 2022 14:52:59.502:DEBUG [coverage]: Writing coverage to /builds/XXX/artifacts/coverage/lib-layout01
22 06 2022 14:52:59.503:DEBUG [launcher]: Process ChromeHeadless exited with code null and signal SIGTERM
22 06 2022 14:52:59.503:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-70992357
22 06 2022 14:52:59.504:DEBUG [reporter.junit]: JUnit results written to "LAST_PROJECT_PATH/junit-test-results.xml".
22 06 2022 14:52:59.510:DEBUG [launcher]: Finished all browsers
22 06 2022 14:52:59.510:DEBUG [launcher]: BEING_FORCE_KILLED -> FINISHED
22 06 2022 14:52:59.510:DEBUG [launcher]: FINISHED -> FINISHED
22 06 2022 14:53:02.512:DEBUG [karma-server]: Received stop event, exiting.
22 06 2022 14:53:02.512:DEBUG [launcher]: Disconnecting all browsers
22 06 2022 14:53:02.512:DEBUG [launcher]: FINISHED -> BEING_FORCE_KILLED
✔ Browser application bundle generation complete.
22 06 2022 14:53:02.512:DEBUG [proxy]: Destroying proxy agents

As you can see it gets stuck after Destroying proxy agents

Any ideas/tips how I could fix it, work arround it?

One Idea is that our pods don't have too much memory and that the pods run out of memory, but then we still need a solution.

The docker image executing the tests is the latest version of trion/ng-cli-karma
https://hub.docker.com/r/trion/ng-cli-karma/tags

Here is the command line:

ng test --code-coverage --no-watch --browsers=ChromeHeadless --source-map=false

here the karma config of the last project ot be exeucted, but they are essentially the same

module.exports = function (config) {
    config.set({
        basePath: "",
        frameworks: ["jasmine", "@angular-devkit/build-angular"],
        plugins: [
            require("karma-jasmine"),
            require("karma-chrome-launcher"),
            require("karma-jasmine-html-reporter"),
            require("karma-junit-reporter"),
            require("karma-coverage"),
            require("@angular-devkit/build-angular/plugins/karma"),
        ],
        client: {
            jasmine: {
                // you can add configuration options for Jasmine here
                // the possible options are listed at https://jasmine.github.io/api/edge/Configuration.html
                // for example, you can disable the random execution with `random: false`
                // or set a specific seed with `seed: 4321`
            },
            clearContext: false, // leave Jasmine Spec Runner output visible in browser
        },
        jasmineHtmlReporter: {
            suppressAll: true, // removes the duplicated traces
        },
        coverageReporter: {
            dir: require("path").join(__dirname, "../../artifacts/coverage/XXX"),
            subdir: ".",
            reporters: [{ type: "html" }, { type: "text-summary" }],
        },
        junitReporter: {
            outputDir: require("path").join(__dirname, "../../artifacts/tests/XXX"),
            outputFile: "junit-test-results.xml",
            useBrowserName: false,
        },
        reporters: ["progress", "kjhtml", "junit"],
        port: 9876,
        colors: true,
        logLevel: config.LOG_DEBUG,
        autoWatch: true,
        browsers: ["Chrome", "ChromeHeadless"],
        singleRun: false,
        restartOnFileChange: true,
        failOnEmptyTestSuite: false, // TODO: remove when adding tests
        customLaunchers: {
            chromeHeadless: {
                base: "ChromeHeadless",
                flags: ["--disable-gpu", "--no-sandbox", "--disable-accelerated-video-decode", "--disable-accelerated-mjpeg-decode"],
            },
        },
    });
};

and here the depedencies of our package.json

    "dependencies": {
        "@angular/animations": "^14.0.2",
        "@angular/cdk": "^14.0.2",
        "@angular/common": "^14.0.2",
        "@angular/compiler": "^14.0.2",
        "@angular/core": "^14.0.2",
        "@angular/elements": "^14.0.2",
        "@angular/flex-layout": "^13.0.0-beta.38",
        "@angular/forms": "^14.0.2",
        "@angular/material": "^14.0.2",
        "@angular/platform-browser": "^14.0.2",
        "@angular/platform-browser-dynamic": "^14.0.2",
        "@angular/router": "^14.0.2",
        "@INTERNAL_1": "0.0.11",
        "@INTERNAL_2: "0.0.12",
        "@ngx-translate/core": "13.0.0",
        "angular-split": "^13.2.0",
        "diagram-js": "^8.2.1",
        "hotkeys-js": ">=3.8.7",
        "linqts": ">=1.14.4",
        "ngx-toastr": "^14.2.4",
        "npm": "^8.1.4",
        "resize-observer": ">=1.0.2",
        "rxjs": "~6.6.0",
        "tslib": "^2.3.0",
        "zone.js": "~0.11.4"
    },
    "devDependencies": {
        "@angular-devkit/build-angular": "^14.0.2",
        "@angular-eslint/builder": "13.5.0",
        "@angular-eslint/eslint-plugin": "14.0.0-alpha.0",
        "@angular-eslint/eslint-plugin-template": "13.5.0",
        "@angular-eslint/schematics": "14.0.0-alpha.0",
        "@angular-eslint/template-parser": "14.0.0-alpha.0",
        "@angular/cli": "^14.0.2",
        "@angular/compiler-cli": "^14.0.2",
        "@playwright/test": "^1.21.1",
        "@types/jasmine": "^4.0.3",
        "@types/node": "^12.11.1",
        "@typescript-eslint/eslint-plugin": "5.3.0",
        "@typescript-eslint/parser": "5.3.0",
        "bpmn-js": "^9.0.3",
        "eslint": "^8.2.0",
        "jasmine-core": "^4.2.0",
        "karma": "^6.4.0",
        "karma-chrome-launcher": "^3.1.1",
        "karma-coverage": "^2.2.0",
        "karma-jasmine": "^5.1.0",
        "karma-jasmine-html-reporter": "^2.0.0",
        "karma-junit-reporter": "^2.0.1",
        "ng-packagr": "^14.0.2",
        "typescript": "~4.6.4",
        "webpack-bundle-analyzer": "^4.5.0"
    },
    "overrides": {
        "@angular/flex-layout": {
            "@angular/cdk": "^14.0.0",
            "@angular/common": "^14.0.0",
            "@angular/core": "^14.0.0",
            "@angular/platform-browser": "^14.0.0"
        },

As you can see we upgraded to the latest versions

Thank you very much for your time!

[MichaelPeter]

Seems to be actually a problem of the test runner container, closing this for now:

trion-development/docker-ng-cli-karma#14

[daniel-sc]

I have the same problem - when running directly via ng test (angular cli) it terminates fine, but when using an npm script it hangs. This is reproducible on my local machine with karma-browserstack-launcher and latest versions of angular (14.0.3) and karma (6.4.0).

Debug output before it hangs:

...
29 06 2022 13:10:49.487:DEBUG [karma-server]: Run complete, exiting.
29 06 2022 13:10:49.488:DEBUG [launcher]: Disconnecting all browsers
29 06 2022 13:10:49.489:DEBUG [proxy]: Destroying proxy agents
29 06 2022 13:10:49.490:DEBUG [launcher.browserstack]: Shutting down BrowserStackLocal
29 06 2022 13:10:54.570:DEBUG [launcher.browserstack]: Stopped BrowserStackLocal
29 06 2022 13:10:55.656:DEBUG [launcher.browserstack]: Chrome (Windows 10) (worker 117976169) successfully killed.
29 06 2022 13:10:55.658:DEBUG [launcher]: Finished all browsers
29 06 2022 13:10:55.658:DEBUG [launcher]: null -> FINISHED
29 06 2022 13:10:55.659:DEBUG [launcher]: FINISHED -> FINISHED
29 06 2022 13:10:55.662:DEBUG [karma-server]: Received stop event, exiting.
29 06 2022 13:10:55.663:DEBUG [launcher]: Disconnecting all browsers
29 06 2022 13:10:55.663:DEBUG [proxy]: Destroying proxy agents
29 06 2022 13:10:55.664:DEBUG [launcher.browserstack]: Shutting down BrowserStackLocal
29 06 2022 13:10:55.705:DEBUG [launcher]: FINISHED -> FINISHED

[ravasconcelos]

Hi,

I am experiencing exactly the same issue in my project after upgrading it to Angular 14. It was working fine on Angular 12.

@daniel-sc , @MichaelPeter , if you find any work around, can you please share it?

[mrs-cp]

Hi @ravasconcelos, we solved it by switching to Chrome instead of using ChromeHeadless since there is no exit script for it. So: browsers: ['Chrome']

Hope it helps you, too.

[RobinMobers97]

We have the same Issue after upgrading to Angular 14.

Is there any workaround until now ?

[ravasconcelos]

Hi @RobinMobers97 , this hack worked for me: karma-runner/karma-browserstack-launcher#195 (comment)

I hope it will be fixed soon.

[alexpalacean]

Hi,we have the same Issue after upgrading to Angular 14.

Will this be fixed soon?

[mrs-cp]

have you tried to switch from ChromeHeadless to browsers: ['Chrome'] in your karma.conf.json? This worked for us :)

[alexpalacean]

@mrs-cp yes, we tried, but it doesn't work :(

[mrs-cp]

have you updated your docker images to trion/ng-cli:14.2.1 or trion/ng-cli-karma:14.2.1, respectively?
plus we set singleRun: true + autoWatch: false in our karma.conf.json

[menzen]

@mrs-cp yes, i experienced
in trion/ng-cli-karma:13... that ng test exits but npm run test dont
but in trion/ng-cli-karma:14.2.1 no way (ng test | npm run test) exits

so we have to use trion/ng-cli-karma:13 and ng test for now

[hungerregnuh]

Had a similar issue on github... oddly enough disabling analytics in the ng cli was the trick to get this stable.

Do one of the following:

• set the env var NG_CLI_ANALYTICS=false
• ng analytics disable
• In angular.json

  "cli": {
    "analytics": false
  }

[MessiasLima]

Any update on this?

[borntodesign7]

We are also facing the same problem on angular 15.x and 16.x
We use maven frontend plugin to build our angular UI,
We also tried a new ng app(version 15 and 16) with same frontend plugin, same problem.
So It is something in the library.

We tried the hacks provided here but unfortunately, they did not work for us.
Anyone able to find some workaround, please share.
For now we are migrating our tests to Jest but that's not an ideal way.

[diegoahg]

Hi!! I fixed the problem and I post in here trion-development/docker-ng-cli-karma#14 (comment).

[gilmorem560]

So I've got a flavor of this happening with Karma (v6.4.3) being called by Angular's test runner in CI with the following command:

npx ng test --no-watch --browsers=ChromeHeadlessNoSandbox

In researching the problem, I found: karma-runner/karma-browserstack-launcher#195 (comment)

The suggestion therein is simply to add a process exit at the end of the close event on the web server. That would be here:

karma/lib/server.js

Line 473 in 84f85e7

webServer.close(() => {

The "hacky workaround" therein was to add

process.nextTick(() => process.exit(code || 0));

after the removeAllListeners() call. I tried this on my own machine (not the CI host) and it didn't seem to break anything, although I've never had this hangup on my host, just CI.

Looking above, I wonder if the

done(code || 0)

already up there on line 468 would work similarly.

Just food for thought, I'll probably try this out in the next few days, a few folks in the linked thread suggested this helped them with troubles in this area.