DAOS-7485 control: Implement system reint to act on all pools #15551

tanabarr · 2024-12-03T14:00:37Z

Add dmg system reint command to reintegrate a set of storage nodes or
ranks from all the pools they belong to. Takes --ranks or --rank-hosts in
ranged format.

Shorten variable naming from Reintegrate to Reint
Don't export variables unnecessarily in cmd/dmg
Improve reporting of protobuf unmarshal errors
Add system reint to {cmd/dmg/,lib/control/,server/mgmt_}system.go
Add unit test coverage for new code

Required-githooks: true

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2024-12-03T14:00:56Z

Ticket title is 'dmg command to drain and reintegrate nodes from all pools'
Status is 'In Review'
Labels: 'triaged'
https://daosio.atlassian.net/browse/DAOS-7485

daosbuild1 · 2024-12-03T14:08:33Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/1/execution/node/357/log

daosbuild1 · 2024-12-03T14:09:03Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/1/execution/node/354/log

daosbuild1 · 2024-12-03T14:11:32Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/1/execution/node/273/log

daosbuild1 · 2024-12-03T14:14:35Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/1/execution/node/304/log

daosbuild1 · 2024-12-03T14:17:42Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/1/execution/node/480/log

daosbuild1 · 2024-12-03T14:22:31Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/1/execution/node/519/log

daosbuild1 · 2024-12-03T19:02:02Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/2/execution/node/375/log

daosbuild1 · 2024-12-03T19:02:28Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/2/execution/node/387/log

daosbuild1 · 2024-12-03T19:03:15Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/2/execution/node/360/log

daosbuild1 · 2024-12-03T19:05:05Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/2/execution/node/369/log

daosbuild1 · 2024-12-03T19:06:00Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/2/execution/node/359/log

daosbuild1 · 2024-12-03T19:10:36Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/2/execution/node/364/log

daosbuild1 · 2024-12-11T00:17:45Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15551/5/testReport/

Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

daosbuild1 · 2024-12-11T18:52:07Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15551/6/testReport/

Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

…-stack/daos into tanabarr/control-reintpools-pernode Features: control Required-githooks: true Signed-off-by: Tom Nabarro <tom.nabarrointel.com>

daosbuild1 · 2024-12-12T05:51:36Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/7/execution/node/1211/log

daltonbohning · 2024-12-12T15:47:51Z

I suspect the copyright GHA is failing because the workflow is coming from a merge of this PR + master, but the source tree used is just this PR. Something I'll need to consider in the future for new GHA. Anyway, the copyright GHA is not required so can be ignored

daosbuild1 · 2024-12-13T01:13:00Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15551/8/testReport/

…intpools-pernode Features: pool Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

daosbuild1 · 2024-12-16T11:06:30Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/9/execution/node/372/log

daosbuild1 · 2024-12-16T11:07:55Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/9/execution/node/364/log

daosbuild1 · 2024-12-16T11:09:17Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/9/execution/node/361/log

daosbuild1 · 2024-12-16T11:10:15Z

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/9/execution/node/346/log

daosbuild1 · 2024-12-16T11:14:20Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/9/execution/node/367/log

daosbuild1 · 2024-12-16T11:19:25Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/9/execution/node/521/log

Features: pool Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

daosbuild1 · 2024-12-16T21:47:38Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15551/10/execution/node/1130/log

…intpools-pernode Signed-off-by: Tom Nabarro <tom.nabarrointel.com>

Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

mjmac

Functionally, looks OK to me. I have some quibbles with the (re-)naming, though.

mjmac · 2024-12-19T15:24:27Z

src/control/cmd/dmg/pretty/system.go

-		return
-	}
-
+func printSysOsaResults(out io.Writer, results []*control.SystemOsaResult) {


This helper should have a comment. Also, what is Osa? Maybe a better name for that whole concept and family of types/functions might be PoolMembershipUpdates? Ultimately, the functionality is less about the system and more about the changes to pools, right?

mjmac · 2024-12-19T15:33:34Z

src/control/drpc/modules.go

@@ -134,7 +134,7 @@ func (m MgmtMethod) String() string {
 		MethodPoolExclude:          "PoolExclude",
 		MethodPoolDrain:            "PoolDrain",
 		MethodPoolExtend:           "PoolExtend",
-		MethodPoolReintegrate:      "PoolReintegrate",
+		MethodPoolReint:            "PoolReint",


Why the rename? The other methods generally use full words, and Reint is less clear than Reintegrate without context. Plus, it makes this PR much noisier than it would be without the rename.

it's just long and sticks out in the C code because of its length, wanted to be consistent. is it a strong enough objection that you want me to revert all the changes?

These symbols are for humans, not computers. Truncating a word in an API surface because it formats inconveniently seems strange to me.

FWIW, I think partly why long names stand out in C code is because of the conventions used in DAOS. E.g. the DAOS C code tends to do

my_return_var = some_long_function_name(some_long_argument_name1, some_long_argument_name2, some_long_argument_name3)

Whereas, IMO, for this reason and several others actually, I think this is much more reasonable

my_return_var = some_long_function_name( some_long_argument_name1, some_long_argument_name2, some_long_argument_name3)

We all come from different programming experiences and it's always about striking a balance but I generally find that people coming from C backgrounds are uncomfortable with long variable names which people from other backgrounds may be more comfortable with. So when interacting with C engine code I tend to try to be considerate to the preferences of the relevant code owners. "Reint" is an abbreviation that's been used variously around
the code base and is relatively intuitive and unambiguous. @mjmac are you requesting that I revert the reame?

Fair point with regard to the C code, but why does the Go code need to be truncated, too? Go norms focus on readability and maintainability. I don't have a problem with the C code conforming to the style in that codebase, but I do think the Go code should be reverted to the more verbose and readable conventions.

mjmac · 2024-12-19T15:38:44Z

src/control/lib/control/system.go

@@ -566,41 +566,49 @@ func SystemExclude(ctx context.Context, rpcClient UnaryInvoker, req *SystemExclu
 	return resp, convertMSResponse(ur, resp)
 }

+// SystemOsaResult describes the result of an OSA operation on a pool's ranks.


As noted elsewhere, this is not a very human-friendly type name. It requires the reader to go figure out what Osa or OSA is. That's more of an internal DAOS name, IMO. Better to describe the types and functions in terms of why and how they would be used, e.g. SystemPoolMembershipUpdate or something along those lines. Yes, it's more verbose, but this is a public API, and I don't think brevity is a virtue in public types and documentation.

Membership is an overloaded term IMO but if you insist I can change to SystemPoolMembershipUpdate, is that required?

Put yourself in the shoes of an API user who has not been hacking on DAOS for a couple of years. What the heck is a/an Osa? Sure, it can be looked up, but the bar for adding to an API surface should be higher than "it compiles and lets me move on to the next bug, go figure it out", right?

I'm re-factoring to reduce duplication as per suggestion from @wangshilong and will rename appropriately

…failures Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

…intpools-pernode Features: control Signed-off-by: Tom Nabarro <[email protected]>

knard38

LGTM

tanabarr added the control-plane work on the management infrastructure of the DAOS Control Plane label Dec 3, 2024

tanabarr self-assigned this Dec 3, 2024

Base automatically changed from tanabarr/control-drainpools-pernode to master December 3, 2024 18:56

tanabarr mentioned this pull request Dec 3, 2024

DAOS-7485 control: Implement dmg system drain to act on all hosts #15506

Merged

18 tasks

tanabarr and others added 8 commits December 11, 2024 18:18

DAOS-7485 control: Add system cmd to reint ranks to all pools

83635af

Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

update drpc method names

4fe74fb

Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

protobuf related changes

1d0919d

Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

renaming changes reintegrate->reint

d06ee0e

Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

address leftover comments from kjacque on PR-15506

ba498c8

Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

DRY-up refactor

a67fc8d

Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

add SystemReint internals

79a3513

Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

refactor to reduce repetition

d96bbfd

Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

tanabarr force-pushed the tanabarr/control-reintpools-pernode branch from 45f3e40 to d96bbfd Compare December 11, 2024 18:18

tanabarr added 2 commits December 12, 2024 00:21

refactor to reduce repetition

b6a73b2

Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

Merge branch 'tanabarr/control-reintpools-pernode' of github.com:daos…

2446825

…-stack/daos into tanabarr/control-reintpools-pernode Features: control Required-githooks: true Signed-off-by: Tom Nabarro <tom.nabarrointel.com>

Merge remote-tracking branch 'origin/master' into tanabarr/control-re…

79e6c4c

…intpools-pernode Features: pool Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

tanabarr marked this pull request as ready for review December 16, 2024 11:03

tanabarr requested review from a team as code owners December 16, 2024 11:03

resolve merge conference

c3e19e5

Features: pool Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

tanabarr added 2 commits December 18, 2024 12:27

Merge remote-tracking branch 'origin/master' into tanabarr/control-re…

f53f8bd

…intpools-pernode Signed-off-by: Tom Nabarro <tom.nabarrointel.com>

add missing unit test coverage for new system-reint code

5c5b8ac

Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

tanabarr requested review from mjmac, kjacque and knard38 December 18, 2024 12:32

trivial changes to command results table output

e8c7622

Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

mjmac reviewed Dec 19, 2024

View reviewed changes

tanabarr added 2 commits December 27, 2024 15:12

consolidate reintegrate methods into drain and coalesce invalid rank …

7fb508b

…failures Features: control Required-githooks: true Signed-off-by: Tom Nabarro <[email protected]>

Merge remote-tracking branch 'origin/master' into tanabarr/control-re…

10f42ec

…intpools-pernode Features: control Signed-off-by: Tom Nabarro <[email protected]>

knard38 approved these changes Dec 30, 2024

View reviewed changes

DAOS-7485 control: Implement system reint to act on all pools #15551

Are you sure you want to change the base?

DAOS-7485 control: Implement system reint to act on all pools #15551

Conversation

tanabarr commented Dec 3, 2024 • edited Loading

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Dec 3, 2024 • edited Loading

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 3, 2024

daosbuild1 commented Dec 11, 2024

daosbuild1 commented Dec 11, 2024

daosbuild1 commented Dec 12, 2024

daltonbohning commented Dec 12, 2024

daosbuild1 commented Dec 13, 2024

daosbuild1 commented Dec 16, 2024

daosbuild1 commented Dec 16, 2024

daosbuild1 commented Dec 16, 2024

daosbuild1 commented Dec 16, 2024

daosbuild1 commented Dec 16, 2024

daosbuild1 commented Dec 16, 2024

daosbuild1 commented Dec 16, 2024

mjmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjmac Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knard38 left a comment

Choose a reason for hiding this comment

tanabarr commented Dec 3, 2024 •

edited

Loading

github-actions bot commented Dec 3, 2024 •

edited

Loading

mjmac Dec 19, 2024 •

edited

Loading