DAOS-11736 test: CR Pass 2 - Dangling pool map test #12517

shimizukko · 2023-06-24T05:40:25Z

Create a pool.
Stop servers.
Manually remove /mnt/daos0/<pool_uuid>/vos-0 from rank 0 node.
Enable and start the checker.
Query the checker and verify that the issue was fixed. i.e., Current status is COMPLETED.
Disable the checker.
Verify that the pool has one less target.

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_dangling_pool_map
Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

1. Create a pool. 2. Stop servers. 3. Manually remove /mnt/daos0/<pool_uuid>/vos-0 from rank 0 node. 4. Enable and start the checker. 5. Query the checker and verify that the issue was fixed. i.e., Current status is COMPLETED. 6. Disable the checker. 7. Verify that the pool has one less target. Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_dangling_pool_map Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

github-actions · 2023-06-24T05:40:44Z

Bug-tracker data:
Ticket title is 'Pass 2 Automate: No storage is allocated to engine - Dangling pool map'
Status is 'In Review'
https://daosio.atlassian.net/browse/DAOS-11736

daosbuild1

LGTM. No errors found by checkpatch.

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_dangling_pool_map Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_dangling_pool_map Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-06-25T19:24:23Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-12517/4/testReport/

shimizukko · 2023-06-27T01:10:21Z

Checker isn’t detecting/fixing the issue.

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-07-06T05:34:37Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-12517/5/testReport/

… shard Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daltonbohning · 2023-07-11T21:52:18Z

src/tests/ftest/recovery/pool_membership.py

+
+        # 3. Manually remove /mnt/daos0/<pool_uuid>/vos-0 from rank 0 node.
+        rank_0_host = NodeSet(self.server_managers[0].get_host(0))
+        rm_cmd = f"sudo rm /mnt/daos0/{pool.uuid.lower()}/vos-0"


Instead of hardcoding /mnt/daos0, we should do something like:

self.server_managers[0].get_config_value("scm_mount")

Done. Thanks.

daltonbohning · 2023-07-11T21:56:21Z

src/tests/ftest/recovery/pool_membership.py

+        total_targets = query_out["response"]["total_targets"]
+        active_targets = query_out["response"]["active_targets"]
+        diff = total_targets - active_targets
+        if diff != 1:
+            expected_targets = total_targets - 1
+            msg = (f"Unexpected number of active targets! Expected = {expected_targets}; "
+                   f"Actual = {active_targets}")
+            errors.append(msg)


minor - recommend making this check more clear

Suggested change

total_targets = query_out["response"]["total_targets"]

active_targets = query_out["response"]["active_targets"]

diff = total_targets - active_targets

if diff != 1:

expected_targets = total_targets - 1

msg = (f"Unexpected number of active targets! Expected = {expected_targets}; "

f"Actual = {active_targets}")

errors.append(msg)

total_targets = query_out["response"]["total_targets"]

active_targets = query_out["response"]["active_targets"]

expected_targets = total_targets - 1

if active_targets != expected_targets:

msg = (f"Unexpected number of active targets! Expected = {expected_targets}; "

f"Actual = {active_targets}")

errors.append(msg)

Add checker restart logic in case checker doesn't detect fault. Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 5 Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-07-14T04:56:55Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12517/7/execution/node/772/log

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 5 Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 5 Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

src/tests/ftest/recovery/pool_membership.py

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 6 Required-githooks: true

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-07-24T09:35:28Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-12517/12/testReport/

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 7 Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 7 Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 reviewed Jun 24, 2023

View reviewed changes

DAOS-11736 test: Fix pylint

a54038b

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_dangling_pool_map Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1 reviewed Jun 24, 2023

View reviewed changes

shimizukko added 2 commits June 24, 2023 05:51

DAOS-11736 test: Move the test to hw,small

1102fe0

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_dangling_pool_map Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

DAOS-11736 test: Update tag

38119a2

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1 reviewed Jun 24, 2023

View reviewed changes

DAOS-11736 test: Use hw,medium

43ee844

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1 reviewed Jun 24, 2023

View reviewed changes

DAOS-11736 test: Increase dmg check query retry count

13dfb88

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1 reviewed Jul 6, 2023

View reviewed changes

DAOS-11736 test: Increase dmg check query retry count for orphan pool…

3f41c84

… shard Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1 reviewed Jul 6, 2023

View reviewed changes

shimizukko marked this pull request as ready for review July 6, 2023 06:27

shimizukko requested a review from a team as a code owner July 6, 2023 06:27

shimizukko requested review from daltonbohning and dinghwah July 6, 2023 06:27

daltonbohning reviewed Jul 11, 2023

View reviewed changes

daosbuild1 reviewed Jul 14, 2023

View reviewed changes

DAOS-11736 test: Add check_stop()

b9b835e

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 5 Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1 reviewed Jul 14, 2023

View reviewed changes

DAOS-11736 test: Remove unnecessary comment

36938aa

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 5 Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1 reviewed Jul 16, 2023

View reviewed changes

shimizukko requested a review from daltonbohning July 16, 2023 01:13

daosbuild1 reviewed Jul 17, 2023

View reviewed changes

daltonbohning reviewed Jul 17, 2023

View reviewed changes

src/tests/ftest/recovery/pool_membership.py Outdated Show resolved Hide resolved

Merge branch 'feature/cat_recovery' into makito/DAOS-11736

34cc6ea

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 6 Required-githooks: true

daosbuild1 reviewed Jul 24, 2023

View reviewed changes

DAOS-11736 test: Sleep 3 sec between check enable and check start

6e615e9

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 7 Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1 reviewed Jul 24, 2023

View reviewed changes

DAOS-11736 test: Remove unused variable

f7329dc

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: pool_membership Test-repeat: 7 Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

daosbuild1 reviewed Jul 24, 2023

View reviewed changes

shimizukko requested a review from daltonbohning July 28, 2023 23:36

daltonbohning approved these changes Jul 31, 2023

View reviewed changes

dinghwah approved these changes Jul 31, 2023

View reviewed changes

shimizukko requested a review from a team August 2, 2023 01:08

daltonbohning merged commit 448373a into feature/cat_recovery Aug 2, 2023

daltonbohning deleted the makito/DAOS-11736 branch August 2, 2023 14:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-11736 test: CR Pass 2 - Dangling pool map test #12517

DAOS-11736 test: CR Pass 2 - Dangling pool map test #12517

shimizukko commented Jun 24, 2023

github-actions bot commented Jun 24, 2023 •

edited

Loading

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 commented Jun 25, 2023

shimizukko commented Jun 27, 2023

daosbuild1 left a comment

daosbuild1 commented Jul 6, 2023

daosbuild1 left a comment

daltonbohning Jul 11, 2023

shimizukko Jul 16, 2023

daltonbohning Jul 11, 2023

shimizukko Jul 16, 2023

daosbuild1 left a comment

daosbuild1 commented Jul 14, 2023

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 commented Jul 24, 2023

daosbuild1 left a comment

daosbuild1 left a comment

DAOS-11736 test: CR Pass 2 - Dangling pool map test #12517

DAOS-11736 test: CR Pass 2 - Dangling pool map test #12517

Conversation

shimizukko commented Jun 24, 2023

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Jun 24, 2023 • edited Loading

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jun 25, 2023

shimizukko commented Jun 27, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jul 6, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daltonbohning Jul 11, 2023

Choose a reason for hiding this comment

shimizukko Jul 16, 2023

Choose a reason for hiding this comment

daltonbohning Jul 11, 2023

Choose a reason for hiding this comment

shimizukko Jul 16, 2023

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jul 14, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jul 24, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 24, 2023 •

edited

Loading