DAOS-16464 test: improve online_rebuild_mdtest.py #15108

daltonbohning · 2024-09-09T21:57:41Z

Run with a stonewall and stop ranks after half of the stonewall time so the timing is more reliable than arbitrarily sleeping for 30 seconds.

Test-tag: EcodOnlineRebuildMdtest
Test-repeat: 3
Skip-unit-tests: true
Skip-fault-injection-test: true

Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2024-09-09T21:57:57Z

Ticket title is 'erasurecode/online_rebuild_mdtest.py:EcodOnlineRebuildMdtest.test_ec_online_rebuild_mdtest - time out waiting for mdtest after server stop'
Status is 'In Review'
Labels: 'pr_test,testp2'
https://daosio.atlassian.net/browse/DAOS-16464

daosbuild1 · 2024-09-10T16:53:29Z

Test stage Functional Hardware Large completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15108/1/testReport/

daosbuild1 · 2024-09-10T20:13:04Z

Test stage Functional Hardware Large completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15108/2/testReport/

Run with a stonewall and stop ranks after half of the stonewall time so the timing is more reliable than arbitrarily sleeping for 30 seconds. Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Required-githooks: true Signed-off-by: Dalton Bohning <[email protected]>

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Required-githooks: true Signed-off-by: Padmanabhan <[email protected]>

daltonbohning · 2024-12-20T16:01:38Z

@rpadma2 It looks to run out of space during archiving: https://build.hpdd.intel.com/blue/rest/organizations/jenkins/pipelines/daos-stack/pipelines/daos/branches/PR-15108/runs/12/nodes/809/log/?start=0
I wonder if the problem is because the engine log_mask is DEBUG by default?

It looks like many other tests use either log_mask: DEBUG,MEM=ERR or log_mask: INFO or log_mask: ERR

rpadma2 · 2024-12-20T16:54:26Z

@rpadma2 It looks to run out of space during archiving: https://build.hpdd.intel.com/blue/rest/organizations/jenkins/pipelines/daos-stack/pipelines/daos/branches/PR-15108/runs/12/nodes/809/log/?start=0 I wonder if the problem is because the engine log_mask is DEBUG by default?

It looks like many other tests use either log_mask: DEBUG,MEM=ERR or log_mask: INFO or log_mask: ERR

Yes. We can't have DEBUG log mask... It maybe an issue. Let me change it to ERR.

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Padmanabhan <[email protected]>

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Padmanabhan <[email protected]>

daosbuild1 · 2025-01-17T22:34:51Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15108/21/execution/node/925/log

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Padmanabhan <[email protected]>

daltonbohning · 2025-01-21T17:02:01Z

src/tests/ftest/util/ec_utils.py

+        """
+        try:
+            result = self.execute_mdtest(mdtest_result_queue)
+        except (CommandFailure, DaosApiError, DaosTestError):


We need to catch all exceptions because we don't know what might go wrong

Suggested change

except (CommandFailure, DaosApiError, DaosTestError):

except Exception: # pylint: disable=broad-except

Will update it.

daltonbohning · 2025-01-21T17:03:14Z

src/tests/ftest/util/ec_utils.py

+        self.container = self.get_mdtest_container(self.pool)
+        if self.container is None:
+            self.fail("Container Create Failed")


Default behavior of get_mdtest_container/get_container will raise an exception so we don't need to check this

Suggested change

self.container = self.get_mdtest_container(self.pool)

if self.container is None:

self.fail("Container Create Failed")

self.container = self.get_mdtest_container(self.pool)

Will update it.

rpadma2 · 2025-01-21T22:38:25Z

@daltonbohning : Can you take a look at the latest changes? If things are fine, we can move this PR to "Ready for review" state.

shimizukko · 2025-01-22T00:08:08Z

src/tests/ftest/erasurecode/online_rebuild_mdtest.yaml

-  num_of_files_dirs: 10000000
-  stonewall_timer: 10
-  stonewall_statusfile: stoneWallingStatusFile
+  num_of_files_dirs: 100000009


Why is the last digit 9? It seems random.

Must be a mistake... Will fix it.

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Padmanabhan <[email protected]>

daltonbohning

code LGTM. Thanks!

rpadma2 · 2025-01-22T21:31:42Z

@daos-stack/daos-gatekeeper : This PR is ready for merge.

daosbuild1 · 2025-01-23T09:58:11Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15108/23/execution/node/925/log

daltonbohning · 2025-01-23T17:11:58Z

@rpadma2 Did you retrigger the testing? It was retriggered and failed 1 time: https://build.hpdd.intel.com/blue/organizations/jenkins/daos-stack%2Fdaos/detail/PR-15108/23/tests

rpadma2 · 2025-01-23T17:29:22Z

@rpadma2 Did you retrigger the testing? It was retriggered and failed 1 time: https://build.hpdd.intel.com/blue/organizations/jenkins/daos-stack%2Fdaos/detail/PR-15108/23/tests
Yes. I retriggered it. It looks like we are hitting pool destroy failure.... I am not sure whether we need to wait for the rebuild to complete before we do pool destroy, Still not a correct behavior to me...

rpadma2 · 2025-01-23T17:38:10Z

There is a ticket already related to this issue: https://daosio.atlassian.net/issues/DAOS-16737?jql=textfields%20~%20%22pool%20destroy%2A%22 . Looks like a known issue,

- Run with a stonewall and stop ranks after half of the stonewall time so the timing is more reliable than arbitrarily sleeping for 30 seconds. - Catch exceptions raised in the mdtest thread. - Reduce logging. - Misc refactoring improvements Signed-off-by: Dalton Bohning <[email protected]> Signed-off-by: Padmanabhan <[email protected]>

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true - Run with a stonewall and stop ranks after half of the stonewall time so the timing is more reliable than arbitrarily sleeping for 30 seconds. - Catch exceptions raised in the mdtest thread. - Reduce logging. - Misc refactoring improvements Signed-off-by: Dalton Bohning <[email protected]> Signed-off-by: Padmanabhan <[email protected]>

- Run with a stonewall and stop ranks after half of the stonewall time so the timing is more reliable than arbitrarily sleeping for 30 seconds. - Catch exceptions raised in the mdtest thread. - Reduce logging. - Misc refactoring improvements Signed-off-by: Dalton Bohning <[email protected]> Signed-off-by: Padmanabhan <[email protected]>

daltonbohning self-assigned this Sep 9, 2024

daltonbohning force-pushed the dbohning/daos-16464 branch from f37a473 to d628240 Compare September 10, 2024 17:29

daltonbohning force-pushed the dbohning/daos-16464 branch 3 times, most recently from 872cbcc to 07ef8e7 Compare September 12, 2024 18:15

daltonbohning force-pushed the dbohning/daos-16464 branch 5 times, most recently from a176256 to 4c1252a Compare December 2, 2024 17:37

daltonbohning force-pushed the dbohning/daos-16464 branch from 4c1252a to 34173c2 Compare December 2, 2024 22:13

DAOS-16494 test: Updated the read/write bytes to 1M.

a09e044

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Required-githooks: true Signed-off-by: Padmanabhan <[email protected]>

rpadma2 added 4 commits December 20, 2024 11:55

DAOS-16464 test: Merge branch 'master' into dbohning/daos-16464

0c47554

DAOS-16464 test: Set logmask to ERR for the test.

775a37b

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Padmanabhan <[email protected]>

DAOS-16464 test: Merge branch 'master' into dbohning/daos-16464

40b4a51

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true

DAOS-16464 test: Merge branch 'master' into dbohning/daos-16464

2cffbfa

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Padmanabhan <[email protected]>

rpadma2 marked this pull request as ready for review January 14, 2025 20:23

rpadma2 requested review from a team as code owners January 14, 2025 20:23

rpadma2 requested review from rpadma2, phender, shimizukko and dinghwah January 14, 2025 20:23

shimizukko previously approved these changes Jan 14, 2025

View reviewed changes

DAOS-16464 test: Updated copyright and minor fix.

9c7841b

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Padmanabhan <[email protected]>

DAOS-16464 test: Add a better exception handling mechanism.

4bd35d8

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Padmanabhan <[email protected]>

daltonbohning commented Jan 21, 2025

View reviewed changes

rpadma2 requested review from phender and shimizukko January 21, 2025 22:36

shimizukko previously approved these changes Jan 22, 2025

View reviewed changes

rpadma2 added 2 commits January 22, 2025 09:56

DAOS-16464 test: Merge branch 'master' into dbohning/daos-16464

ecf0488

DAOS-16464 test: Minor Review updates

b9bdbdd

Test-tag: EcodOnlineRebuildMdtest Test-repeat: 3 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Padmanabhan <[email protected]>

rpadma2 dismissed shimizukko’s stale review via b9bdbdd January 22, 2025 15:03

rpadma2 requested a review from shimizukko January 22, 2025 15:03

daltonbohning commented Jan 22, 2025

View reviewed changes

shimizukko approved these changes Jan 22, 2025

View reviewed changes

rpadma2 marked this pull request as ready for review January 22, 2025 21:29

rpadma2 approved these changes Jan 22, 2025

View reviewed changes

rpadma2 requested a review from a team January 22, 2025 21:31

daltonbohning removed the request for review from a team January 23, 2025 17:12

rpadma2 requested a review from a team January 23, 2025 20:26

daltonbohning merged commit fb444f2 into master Jan 23, 2025
43 of 48 checks passed

daltonbohning deleted the dbohning/daos-16464 branch January 23, 2025 23:20

jxiong mentioned this pull request Feb 26, 2025

Merge release/2.6 into google/2.6 #15985

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-16464 test: improve online_rebuild_mdtest.py #15108

DAOS-16464 test: improve online_rebuild_mdtest.py #15108

daltonbohning commented Sep 9, 2024

github-actions bot commented Sep 9, 2024 •

edited

Loading

daosbuild1 commented Sep 10, 2024

daosbuild1 commented Sep 10, 2024

daltonbohning commented Dec 20, 2024 •

edited

Loading

rpadma2 commented Dec 20, 2024

daosbuild1 commented Jan 17, 2025

daltonbohning Jan 21, 2025

rpadma2 Jan 22, 2025

rpadma2 Jan 22, 2025

daltonbohning Jan 21, 2025

rpadma2 Jan 22, 2025

rpadma2 Jan 22, 2025

rpadma2 commented Jan 21, 2025

shimizukko Jan 22, 2025

rpadma2 Jan 22, 2025

rpadma2 Jan 22, 2025

daltonbohning left a comment

rpadma2 commented Jan 22, 2025

daosbuild1 commented Jan 23, 2025

daltonbohning commented Jan 23, 2025

rpadma2 commented Jan 23, 2025

rpadma2 commented Jan 23, 2025 •

edited

Loading

	except (CommandFailure, DaosApiError, DaosTestError):
	except Exception: # pylint: disable=broad-except

DAOS-16464 test: improve online_rebuild_mdtest.py #15108

DAOS-16464 test: improve online_rebuild_mdtest.py #15108

Conversation

daltonbohning commented Sep 9, 2024

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Sep 9, 2024 • edited Loading

daosbuild1 commented Sep 10, 2024

daosbuild1 commented Sep 10, 2024

daltonbohning commented Dec 20, 2024 • edited Loading

rpadma2 commented Dec 20, 2024

daosbuild1 commented Jan 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rpadma2 commented Jan 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daltonbohning left a comment

Choose a reason for hiding this comment

rpadma2 commented Jan 22, 2025

daosbuild1 commented Jan 23, 2025

daltonbohning commented Jan 23, 2025

rpadma2 commented Jan 23, 2025

rpadma2 commented Jan 23, 2025 • edited Loading

github-actions bot commented Sep 9, 2024 •

edited

Loading

daltonbohning commented Dec 20, 2024 •

edited

Loading

rpadma2 commented Jan 23, 2025 •

edited

Loading