DAOS-16167 test: update soak test to use internal job scheduler #14775

mjean308 · 2024-07-16T14:30:35Z

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: soak_smoke

Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2024-07-16T14:31:19Z

Ticket title is 'Soak: update soak test to use internal job scheduler instead of depending on slurm'
Status is 'In Progress'
Labels: 'daos_framework,soak'
https://daosio.atlassian.net/browse/DAOS-16167

daosbuild1 · 2024-07-16T14:39:53Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/1/execution/node/415/log

daosbuild1 · 2024-07-16T14:41:28Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/1/execution/node/313/log

daosbuild1 · 2024-07-16T14:42:21Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/1/execution/node/357/log

daosbuild1 · 2024-07-16T14:45:47Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/373/log

daosbuild1 · 2024-07-16T14:48:57Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/414/log

daosbuild1 · 2024-07-16T14:50:07Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/364/log

daosbuild1 · 2024-07-16T14:51:12Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/360/log

daosbuild1 · 2024-07-16T14:51:39Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/361/log

daosbuild1 · 2024-07-16T14:56:17Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/2/execution/node/367/log

daosbuild1 · 2024-07-16T15:31:46Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/383/log

daosbuild1 · 2024-07-16T15:31:47Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/387/log

daosbuild1 · 2024-07-16T15:36:25Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/363/log

daosbuild1 · 2024-07-16T15:37:32Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/365/log

daosbuild1 · 2024-07-16T15:38:16Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/364/log

daosbuild1 · 2024-07-16T15:41:56Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/3/execution/node/353/log

daosbuild1 · 2024-07-16T20:39:27Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/359/log

daosbuild1 · 2024-07-16T20:45:14Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/341/log

daosbuild1 · 2024-07-16T20:47:45Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/344/log

daosbuild1 · 2024-07-16T20:52:02Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/467/log

daosbuild1 · 2024-07-16T20:52:04Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/398/log

daosbuild1 · 2024-07-16T20:53:08Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/4/execution/node/336/log

daosbuild1 · 2024-07-18T21:34:50Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/383/log

daosbuild1 · 2024-07-18T21:34:57Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/351/log

daosbuild1 · 2024-07-18T21:36:14Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/350/log

daosbuild1 · 2024-07-18T21:38:35Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/357/log

daosbuild1 · 2024-07-18T21:40:30Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/518/log

daosbuild1 · 2024-07-18T21:42:28Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/6/execution/node/352/log

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Required-githooks: true Signed-off-by: Maureen Jean <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Signed-off-by: Maureen Jean <[email protected]>

daosbuild1 · 2024-10-03T22:04:33Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/43/execution/node/936/log

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Required-githooks: true Skipped-githooks: flake,pylint Signed-off-by: Maureen Jean <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Required-githooks: true Skipped-githooks: flake,pylint Signed-off-by: Maureen Jean <[email protected]>

Signed-off-by: Maureen Jean <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

daosbuild1 · 2024-11-06T18:58:48Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/49/execution/node/825/log

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

daosbuild1 · 2024-11-13T17:46:13Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/50/execution/node/936/log

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Signed-off-by: Maureen Jean <[email protected]>

daosbuild1 · 2024-11-18T19:44:43Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/52/execution/node/937/log

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

daosbuild1 · 2024-12-06T16:48:29Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14775/53/execution/node/936/log

daltonbohning · 2024-12-10T16:32:23Z

src/tests/ftest/util/soak_test_base.py

-        if self.host_info.clients.partition.reservation:
-            self.srun_params["reservation"] = self.host_info.clients.partition.reservation
-        # Include test node for log cleanup; remove from client list
+        self.job_scheduler = self.params.get("job_scheduler", "/run/*", default="slurm")


When this is more tested and stable I think the default should be the internal, and phase out slurm

daltonbohning · 2024-12-10T16:35:03Z

src/tests/ftest/util/soak_test_base.py

+        lib_path = os.getenv("LD_LIBRARY_PATH")
+        path = os.getenv("PATH")
+        v_env = os.getenv("VIRTUAL_ENV")
+        env = ";".join([f"export LD_LIBRARY_PATH={lib_path}",
+                        f"export PATH={path}"])
+        if v_env:
+            env = ";".join([env, f"export VIRTUAL_ENV={v_env}"])


We should do for lib_path what we do for v_env since LD_LIBRARY_PATH might not be set

src/tests/ftest/util/soak_test_base.py

src/tests/ftest/util/soak_utils.py

daltonbohning · 2024-12-10T16:46:01Z

src/tests/ftest/util/soak_utils.py

+        job_queue.put(results)
+        # give time to update the queue before exiting
+        time.sleep(0.5)


Is this necessary? As far as I'm aware, the queue should be updated as soon as put returns

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Signed-off-by: Maureen Jean <[email protected]>

phender

Just some nits.

phender · 2024-12-11T14:09:08Z

src/tests/ftest/util/soak_test_base.py

@@ -144,7 +148,8 @@ def pre_tear_down(self):
        if self.all_failed_jobs:
            errors.append("SOAK FAILED: The following jobs failed {} ".format(
                " ,".join(str(j_id) for j_id in self.all_failed_jobs)))
-
+        # cleanup any remaining jobs
+        job_cleanup(self.log, self.hostlist_clients)


In the future it would be nice to remove the pre_tear_down method now that we can use register_cleanup to more effectively handle tearDown operations. Just a note; not a requested change.

phender · 2024-12-11T14:15:54Z

src/tests/ftest/util/soak_test_base.py

+                        job_node_list = node_list[:node_count]
+                        debug_logging(
+                            self.log,
+                            self.enable_debug_msg,
+                            f"DBG: node_list before launch_job {node_list}")


Is this supposed to be logging job_node_list? Otherwise it doesn't seem to add anything different than the previous debug message.

phender · 2024-12-11T14:27:10Z

src/tests/ftest/util/soak_utils.py

+    return next(id_counter)
+
+
+def debug_logging(log, enable_debug_msg, log_msg):


This may be better defined as a SoakTestBase method, where calls would look like self.debug_logging("some message") or test.debug_logging("some message").

phender · 2024-12-11T14:34:30Z

src/tests/ftest/util/soak_test_base.py

-                err_msg = f"Slurm failed to submit job for {script}"
-                job_id_list = []
-                raise SoakTestError(f"<<FAILED:  Soak {self.test_name}: {err_msg}>>")
+        if self.job_scheduler == "slurm":


It would have been nice to have job scheduler classes to avoid multiple if/else statements.

phender · 2024-12-11T14:37:44Z

src/tests/ftest/util/soak_test_base.py

@@ -559,7 +701,7 @@ def run_soak(self, test_param):
        resv_bytes = self.params.get("resv_bytes", test_param + "*", 500000000)
        ignore_soak_errors = self.params.get("ignore_soak_errors", test_param + "*", False)
        self.enable_il = self.params.get("enable_intercept_lib", test_param + "*", False)
-        self.sudo_cmd = "sudo" if enable_sudo else ""
+        self.sudo_cmd = "sudo -n" if enable_sudo else ""


We should consider replacing uses of self.sudo_cmd with command_as_user() in the future to simplify maintenance like this.

phender · 2024-12-11T14:39:21Z

src/tests/ftest/util/soak_utils.py

 from test_utils_container import add_container

 H_LOCK = threading.Lock()
+id_counter = count(start=1)


fwiw I agree with Dalton, especially if the id_counter is used in file names.

phender · 2024-12-11T14:48:49Z

src/tests/ftest/util/soak_utils.py

+        job_log (str): job std out
+        error_log (str): job std error
+        timeout (int): job timeout
+        test (TestObj): soak test obj


Due to line 433 this should be:

Suggested change

test (TestObj): soak test obj

test (SoakTestBase): soak test obj

phender · 2024-12-11T14:50:30Z

src/tests/ftest/util/soak_utils.py

+    if isinstance(host_list, str):
+        # assume one host in list
+        hosts = host_list
+        rhost = host_list


Why do we support a list or string here? We should keep all host information in a NodeSet object.

Update soak to support using an internal job scheduler. Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Required-githooks: true Signed-off-by: Maureen Jean <[email protected]>

…) (#15595) Update soak to support using an internal job scheduler. Signed-off-by: Maureen Jean <[email protected]> Co-authored-by: mjean308 <[email protected]>

DAOS-16167 test: update soak test to use internal job scheduler

47e9f7c

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Required-githooks: true Signed-off-by: Maureen Jean <[email protected]>

mjean308 force-pushed the mjean/DAOS-16167 branch from 015b254 to 47e9f7c Compare July 23, 2024 16:16

include feedback

950f708

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Signed-off-by: Maureen Jean <[email protected]>

Fix typo

e94e2b4

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Required-githooks: true Skipped-githooks: flake,pylint Signed-off-by: Maureen Jean <[email protected]>

mjean308 requested a review from daltonbohning October 8, 2024 20:22

mjean308 added 5 commits October 9, 2024 08:59

Merge branch 'master' into mjean/DAOS-16167

fd7d289

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

Remove server logging from preteardown

42436bf

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Required-githooks: true Skipped-githooks: flake,pylint Signed-off-by: Maureen Jean <[email protected]>

Merge branch 'master' into mjean/DAOS-16167

c7c762a

Signed-off-by: Maureen Jean <[email protected]>

Merge branch 'master' into mjean/DAOS-16167

5570550

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

Merge branch 'master' into mjean/DAOS-16167

7d6084d

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

Merge branch 'master' into mjean/DAOS-16167

86dab47

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

mjean308 added 2 commits November 18, 2024 10:08

Merge branch 'master' into mjean/DAOS-16167

5ee7f75

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

i fix liniting/isort issues

709d1a9

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Signed-off-by: Maureen Jean <[email protected]>

Merge branch 'master' into mjean/DAOS-16167

5cc4e77

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke

daltonbohning reviewed Dec 10, 2024

View reviewed changes

Include feedback

f55e1d4

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: soak_smoke Signed-off-by: Maureen Jean <[email protected]>

mjean308 requested a review from daltonbohning December 10, 2024 20:23

daltonbohning approved these changes Dec 10, 2024

View reviewed changes

phender approved these changes Dec 11, 2024

View reviewed changes

mjean308 requested a review from a team December 11, 2024 15:00

daltonbohning added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Dec 11, 2024

daltonbohning merged commit 908f20e into master Dec 11, 2024
42 checks passed

daltonbohning deleted the mjean/DAOS-16167 branch December 11, 2024 19:11

juszhan1 mentioned this pull request Dec 23, 2024

Merge release/2.6 into google/2.6 #15663

Closed

mjmac mentioned this pull request Jan 13, 2025

Merge release/2.6 into google/2.6 #15725

Merged

		return next(id_counter)


		def debug_logging(log, enable_debug_msg, log_msg):

	test (TestObj): soak test obj
	test (SoakTestBase): soak test obj

DAOS-16167 test: update soak test to use internal job scheduler #14775

DAOS-16167 test: update soak test to use internal job scheduler #14775

Conversation

mjean308 commented Jul 16, 2024

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 16, 2024

daosbuild1 commented Jul 18, 2024

daosbuild1 commented Jul 18, 2024

daosbuild1 commented Jul 18, 2024

daosbuild1 commented Jul 18, 2024

daosbuild1 commented Jul 18, 2024

daosbuild1 commented Jul 18, 2024

daosbuild1 commented Oct 3, 2024

daosbuild1 commented Nov 6, 2024

daosbuild1 commented Nov 13, 2024

daosbuild1 commented Nov 18, 2024

daosbuild1 commented Dec 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phender left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment