Functional Tests not running in parallel

[Carthanial]

Description :

Currently the Functional Test stage is not running in parallel and therefore is taking over 2 hours to run.

Task :

• Develop script to run Functional Tests in parallel to reduce run time by >90%

ETA : EOD Tue 12/21

[Carthanial]

When utilizing the function functionalDynamicParallelSteps(image) it loops through the CiGroups and runs the functional test scripts in a new stage with image.inside. However when that is run for each loop both the docker inspect and docker pull fail with the following errors :

• docker inspect -f . bfs6.7.2_test-test-image:97

/var/lib/jenkins/workspace/Kibana_bfs6.7.2_test_2@tmp/durable-fdaaae32/script.sh: 1: /var/lib/jenkins/workspace
/Kibana_bfs6.7.2_test_2@tmp/durable-fdaaae32/script.sh: docker: not found

• docker pull -f . bfs6.7.2_test-test-image:97

/var/lib/jenkins/workspace/Kibana_bfs6.7.2_test_2@tmp/durable-fdaaae32/script.sh: 1: /var/lib/jenkins/workspace
/Kibana_bfs6.7.2_test_2@tmp/durable-fdaaae32/script.sh: docker: not found

Note :

• The image generated in the Build Container Image stage is assigned the tag bfs6.7.2_test-test-image:97 and is being called seemingly correctly here however we see the above errors. Any insight on this would be appreciated.
• I believe we have come across this issue before and it was due to some configuration in Jenkins however I am not convinced that is the issue here.

Reference : Jenkins Job #97

[Carthanial]

When utilizing the function functionalDynamicParallelSteps(image) it loops through the CiGroups and runs the functional test scripts in a new stage with image.inside. However when that is run for each loop both the docker inspect and docker pull fail with the following errors :

• docker inspect -f . bfs6.7.2_test-test-image:97

/var/lib/jenkins/workspace/Kibana_bfs6.7.2_test_2@tmp/durable-fdaaae32/script.sh: 1: /var/lib/jenkins/workspace
/Kibana_bfs6.7.2_test_2@tmp/durable-fdaaae32/script.sh: docker: not found

• docker pull -f . bfs6.7.2_test-test-image:97

/var/lib/jenkins/workspace/Kibana_bfs6.7.2_test_2@tmp/durable-fdaaae32/script.sh: 1: /var/lib/jenkins/workspace
/Kibana_bfs6.7.2_test_2@tmp/durable-fdaaae32/script.sh: docker: not found

Note :

* The image generated in the `Build Container Image` stage is assigned the tag `bfs6.7.2_test-test-image:97` and is being called seemingly correctly here however we see the above errors. Any insight on this would be appreciated.

* I believe we have come across this issue before and it was due to some configuration in Jenkins however I am not convinced that is the issue here.

Reference : Jenkins Job #97

Fixed with syntax error : Functional Test stage being placed inside image.inside{} when it needed to be after it. This solved this bug.

[Carthanial]

Update :

• The config.js file that is called for these tests utilizes a snapshot of 6.7.2 that is no longer supported therefore the following error occurs :

Snapshots for 6.7.2/latest are not available

[zhyuanqi]

Hi. When we running functional test in parallel, we encountered the issue where only one elasticsearch was able to connect successfully, others failed. #9 as reference.
After logging into the instance, and run the commend, see error below:

[2022-01-06T19:19:23,242][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/jenkins/workspace/Kibana_bfs6.7.2_zhyuanqi/elasticsearch-6.7.2/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
   at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.7.2.jar:6.7.2]
   at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.7.2.jar:6.7.2]
   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.7.2.jar:6.7.2]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/jenkins/workspace/Kibana_bfs6.7.2_zhyuanqi/elasticsearch-6.7.2/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
   at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.2.jar:6.7.2]
   at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.2.jar:6.7.2]

elasticsearch-6.7.2/data folder has lock which allow allows one cluster to write at a time. In order to solve this issue, using #57 (comment) this one as reference, I tried to mount extra data files for it to work.

def functionalDynamicParallelSteps(image){
    ciGroupsMap = [:]
    for (int i = 1; i <= 12; i++) {
        def currentCiGroup = "ciGroup${i}";
        def currentStep = i;
        ciGroupsMap["${currentCiGroup}"] = {
            sh "rm -rf ${env.WORKSPACE}/data/${currentCiGroup}"
            sh "mkdir -p ${env.WORKSPACE}/data/${currentCiGroup}"
            stage("${currentCiGroup}") {
                withEnv([
                    "TEST_BROWSER_HEADLESS=1",
                    "CI=1",
                    "GCS_UPLOAD_PREFIX=fake",
                    "TEST_ES_PORT=9200",
                    "TEST_KIBANA_PORT=5601",
                    "TEST_KIBANA_PROTOCOL=http",
                    "TEST_ES_HOSTNAME=localhost",
                    "CACHE_DIR=${currentCiGroup}"
                ]) {
                    image.inside("-v \'${env.WORKSPACE}/data/${currentCiGroup}:${env.WORKSPACE}/elasticsearch-6.7.2/data\'"){
                       sh './elasticsearch-6.7.2/bin/elasticsearch & '
                       
                       echo "Starting Kibana..."
                       sh "./bin/kibana --no-base-path &"
                       
                       sh "sleep 180"
                       sh "curl localhost:9200"
                       sh "curl localhost:5601"
                       echo "Start Functional Tests"
                       sh "node scripts/functional_test_runner --config test/functional/config.js --include-tag ${currentCiGroup}"
                    }
                }
            }
        }
    }
    parallel ciGroupsMap
}

Reference run: https://jenkins.bfs.sichend.people.aws.dev/job/Kibana/job/bfs6.7.2_zhyuanqi/16/console