Scenarios failing due to evaluation of 'karate-config.js' javascript function call failed: null

[Gokuroro]

When running a project with multiple (upwards of 1000) scenarios and a few functions set in karate-config.js, including a cached login call using the karate.callSingle process, after around 80% or so of the tests are run, all scenarios start failing.

Tried reproducing the issue in an isolated environment/project, but I hit a wall while trying to do so. Tried (based on the original project):

• almost 3 times the amount of scenarios and feature files;
• callable scenarios that call JavaScript functions created in the karate-config.js file;
• great amount of text printed per scenario;
• with and without sleep timer between calls (to simulate the original's longer execution time);
• tried using the same karate-config.js file;

[Gokuroro]

I will keep trying to find a way to reproduce this issue and, depending on your and my managers' availability, we can try to work with you as consultant for this issue.

[ptrthomas]

@Gokuroro sorry, promptly closing this. we need a way to replicate. try taking a heap dump and get others to look at it if needed. try the following:

• set log level to WARN
• if any file upload / download tests are present, remove them
• there is a configure setting to printEnabled
• don't use the logPrettyRequest and logPrettyResponse options
• run with fewer parallel threads
• if you know of any scenarios with huge response payloads, comment them out
• the karate-config.js issue can happen with JDK versions being incompatible, upgrade your CI JDK

[ptrthomas]

@Gokuroro hey could it be that after 10 minutes your cached login actually expires ?

[Gokuroro]

• current log levels:
  • <logger name="com.intuit" level="DEBUG"/>;
  • <root level="info"> ... </root>;
• no upload scenarios present;
• grepping for printEnabled returns nothing;
• grepping for logPretty returns nothing;
• parallel set to 1 (if I understood correctly): Results results = Runner.parallel(getClass(), 1, "target/surefire-reports");;
• karate-config.js is read correctly for most of the run. Also, tried with both OpenJDK8 and OracleJDK9;

As for the scenarios with huge response payloads, this looks like a pretty good hint, since there are multiple cases like this. I'll go through with some more tests and let you know if I can reproduce going forward.

As always, thanks a lot for the support.

[Gokuroro]

@ptrthomas I have rolled this issue to my client's management so they may work with you as consultant if the need arrives. I was able to remove some tests to make the suite work, but that's not a good solution for the project and, as stated, I was not able to reproduce it in a controlled environment.
Given there is classified information in the tests I'm working on, the decision is on management on how to proceed going forward.

Thank you very much for all the support!

[ptrthomas]

@Gokuroro ok. I would say it is unlikely to be a karate issue :) did you see my comment where my guess is your one-time auth expires after 5-10 minutes which explains the behavior you are seeing

[Gokuroro]

@ptrthomas yes, I thought I had answered it, sorry about that. Our auth token does not have a (short) expiration time, with some of my experiences it working almost 24h after the login at least. Either way, the issue observed is evaluation of 'karate-config.js' javascript function call failed: null which seems to be during the karate-config.js file parsing stage.
I'll do some tests on the token expiration time just in case.

[ptrthomas]

@Gokuroro are the karate tests packaged into a JAR file by any chance before you run the tests ?

[Gokuroro]

This is possible. Does it have anything to do with the <packaging>jar</packaging> item in pom.xml? I was not the one who setup the karate environment for the project, so I'm more aware of creating tests with it than the setup.

FYI: I'm running the tests by basically running mvn test.

Edit: I removed the <packaging>jar</packaging> field from pom.xml and it looks like the run was successful.

As always, thank you very much for your support. I'll add this change to our project and push the code and check the result in our CI environment.

[ptrthomas]

@Gokuroro great, thanks for the update.

attention @celcius112 can you take a look at the changes in #833 are causing the stream to be null under load when in a JAR ? please help !

[Gokuroro]

@ptrthomas it ran fine the first time locally, but did not work in the subsequent runs. I'm testing other packaging formats right now. war also failed and others (pom, maven-plugin) didn't even run. I'll look a bit more into it. :(

Worst case, I'll have to leave further contact to my managers, since I'll be leaving this project soon anyway.

Currently trying to force some different versions for the references in pom.xml just to make sure it is nothing related to that.

[celcius112]

@ptrthomas I'll take a look after my vacations 😅 🌴

[celcius112]

@ptrthomas the changes in #833 should not apply here since it is not released yet, am I right ?

[Gokuroro]

@ptrthomas thank you guys very much for all the effort, but since we're on testing phase, we'll just split the CI jobs in 2 based on tags for now. As I mentioned previously, I'll be leaving the project and if this is an issue going forward for the team, they'll contact you guys with better directions.

@celcius112 this is happening in 0.9.5.RC1, if this information helps.

[ptrthomas]

@celcius112 we did an RC release - but hey no worries, see you after your vacation !

[celcius112]

Ah nice ! I did not see this RC.

@Gokuroro Is there a stacktrace somewhere ? I don't remember whether there's a more precise log when evaluating with Nashorn. And is it failing at a precise moment ? Like precisely after 10 minutes ? And last question (maybe 😛), are you running the tests in parallel ?

[ptrthomas]

@celcius112 just one comment, based on some past experience and a brain-dump for those who need to maintain this code in the future. when we did the re-write of the file / resource handling when we moved off cucumber, one of the "last mile" problems was - under concurrent load, an attempt to read a stream (especially when the source was a JAR) would result in an NPE

so you can see a couple of places in the code where we cache and lock stuff.

here we lock threads to create that horrible nio file-system: https://github.com/intuit/karate/blob/develop/karate-core/src/main/java/com/intuit/karate/FileUtils.java#L545

and herewe cache the stream if tit comes from a JAR file, again with a lock and hashmap: https://github.com/intuit/karate/blob/develop/karate-core/src/main/java/com/intuit/karate/Resource.java#L122

and this test actually was very useful to prove the above worked.
https://github.com/intuit/karate/blob/develop/karate-junit4/src/test/java/com/intuit/karate/junit4/files/JarLoadingTest.java#L78

so my request to you is to review your work and see if you need to add some of these defenses. if you can add a test (with the concurrent executor) like the example above, that would give some measure of confidence

[celcius112]

@ptrthomas ok thanks for the resources, i'll take a look