Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge master into CR at 20240305 #13604

Closed
wants to merge 249 commits into from
Closed

Merge master into CR at 20240305 #13604

wants to merge 249 commits into from

Conversation

Nasf-Fan
Copy link
Contributor

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate watchers.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

Nasf-Fan and others added 30 commits March 29, 2022 13:35
@Nasf-Fan @mjmac
DAOS-9590 chk: interface data definition
c4a9fdd 
Define the shared data for the interface between control plane and DAOS
check engine, include the following:

DAOS global inconsistency class, the action to repair inconsistency,
DAOS check scan phases, instance status, pool status, and so on.

Signed-off-by: Fan Yong <[email protected]>
@Nasf-Fan @mjmac
DAOS-9590 chk: interface for controlling DAOS check engine
17ad3d2 
Define the dRPC protocol that are used by control plane to control
DAOS check engine for the following use cases:

1) Start check - DRPC_METHOD_MGMT_CHK_START
2) Stop check - DRPC_METHOD_MGMT_CHK_STOP
3) Query check progress - DRPC_METHOD_MGMT_CHK_QUERY
4) Get check parameters and property - DRPC_METHOD_MGMT_CHK_PROP
5) Execute the action to repair the specified inconsistency under
   interation mode - DRPC_METHOD_MGMT_CHK_ACT

Signed-off-by: Fan Yong <[email protected]>
@Nasf-Fan @mjmac
DAOS-9590 chk: interface for check engine to talk with control plane
913c568 
Define new dRPC upcall to control plane for the following use cases:

DRPC_METHOD_CHK_LIST_POOL: obtain the known pools list from MS.

DRPC_METHOD_CHK_REG_POOL: register the (orphan) pool to MS.

DRPC_METHOD_CHK_DEREG_POOL: deregister the (dangling) pool from MS.

DRPC_METHOD_CHK_REPORT: DAOS check engine reports to the control
plane with the found inconsistency and the repair result. If the
repair action is CIA_INTERACT, then notify the control plane to
interact with the admin for the repair decision.

Signed-off-by: Fan Yong <[email protected]>
@Nasf-Fan @mjmac
DAOS-9583 chk: start DAOS engine with check mode
97cd326 
The check module infrastructure and bootstrap sequence.
New options for start engine with check mode:

-C|--check:     Start engine with check mode, global consistency check.

Signed-off-by: Fan Yong <[email protected]>
@mjmac
DAOS-9586 chk: Add support for starting ranks in checker mode
12bbbf2 
Adds a new --checker flag to `dmg system start` which will
start the ranks in a special checker mode. All ranks must
first be stopped before starting in checker mode.

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
DAOS-9585 chk: Add CR operator interface
6f56eee 
Adds new dmg commands to manage/query CR:
  * dmg check start
  * dmg check stop
  * dmg check query
  * dmg check prop

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
DAOS-9576 chk: Update packaging for libchk.so
fdf70aa 
Ensure that deb/rpm packages know what to do with the file.

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
DAOS-9584 chk: Add srv handlers for checker upcalls (#8579)
57204f0 
Implement control plane handlers for the following
engine checker dRPC upcalls:
  * CheckerListPools
  * CheckerRegisterPool
  * CheckerDeregisterPool

Also fixes a slow test.

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
Merge branch 'master' into feature/cat_recovery
5511b9a
@mjmac
Revert LeaderQuery unit test change
ed855a4 
Shouldn't have been merged into previous commit; causes
problems when trying to integrate with other branches.

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
DAOS-10019 control: Move raft into its own package
75c8748 
Refactor the system package to remove the direct dependencies
on external raft/grpc packages in order to avoid bringing
them in for unrelated tools (e.g. daos, dmg, etc).

Features: control

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
Adapt checker infrastructure for raft package
4c8a5a0 
Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
Merge branch 'master' into feature/cat_recovery
7969d1e
@mjmac
Merge branch 'master' into feature/cat_recovery
1c53d11
@mjmac
Merge branch 'master' into feature/cat_recovery
515ae02
@mjmac
Merge branch 'master' into feature/cat_recovery
36d7850
@mjmac
Merge branch 'master' into feature/cat_recovery
a0d4cda
@mjmac
DAOS-9586 control: Fix race in control RPCs (#8716)
1f32324 
Setting the system name in the RPC callback is racy.

Signed-off-by: Michael MacDonald <[email protected]>
@ryon-jensen
DAOS-10197 ddb: Introduction of daos debug tool (ddb) (#8665)
d7dc272 
The DAOS Debug Tool (ddb) is a new executable that allows a user to
navigate through a file in the VOS format. It is similar to debugfs
for ext2/3/4 and offers both a command line and interactive shell
mode. This commit is the introduction of the tool with just a couple
commands supported. For more details about the tool see
src/ddb/README.md

Signed-off-by: Ryon Jensen <[email protected]>
@mjmac
Merge branch 'master' into feature/cat_recovery
8214c13
@mjmac
DAOS-9279 chk: Update dmg check commands to use cmdutil
5daed3e 
Broken after master was merged in.

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
DAOS-9590 chk: Refactor checker messages (#8683)
9952114 
* Move CheckReport to chk package
    - Separate report request from payload
    - Add Actions and Details lists to allow the checker to
      specify defined actions that could be taken in response
      to the inconsistency report.
    - Rename chk/check -> chk/chk to ensure unique namespaces.
  * Adjust srv/mgmt messages to use chk types directly

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
DAOS-9584 chk: Add Checker database (#8684)
fa98a10 
Provide a central repository for storing the system
checker progress. Add handler for NotifyReport upcall
to store checker reports for display to the admin. Add
new `dmg check repair` command to allow the admin to
select a repair action for an inconsistency.

Changes to chk.CheckReport message:
  * Actions -> ActChoices
  * Details -> ActDetails
  * Adds ActMsgs array for human-formatted action descriptions

TODO: Add test coverage once this settles down.

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
Merge branch 'master' into feature/cat_recovery
1825e91
@mjmac
DAOS-9799 chk: Add fault injection capability for MS checker (#8685)
15ab565 
Add a new `dmg faults add-checker-report` command to allow
manual injection of checker reports for prototyping and testing.
The command and associated RPCs are not compiled into release builds.

Signed-off-by: Michael MacDonald <[email protected]>
@mjmac
Merge branch 'master' into feature/cat_recovery
066b747
@ryon-jensen
DAOS-9590 chk: Update SConscript to export chk_pb (#8870)
0b02c40 
Small fix to always export the chk_pb variable so it
can be used by server and client.

Signed-off-by: Ryon Jensen <[email protected]>
@mjmac
Merge branch 'master' into feature/cat_recovery
fe6d356
@ryon-jensen
DAOS-10224 ddb: dump, load, rm commands (#8781)
6b3c70e 
  * Introduces the dump, dump_ilog, dump_dtx, load, and rm commands for the
    ddb tool.
  * Reworked the construction of the unit test lists so that the
    test function name is printed in stead of a separate description.
  * Added some filtering to the unit tests.
  * Abstracted out the printing of the commands so that what is
    printed is more easily testable and to clean up the
    command functions a little.

Signed-off-by: Ryon Jensen <[email protected]>
@mjmac
Merge branch 'master' into feature/cat_recovery
d8e92ac
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/17/execution/node/1173/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/18/execution/node/1405/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/18/execution/node/1451/log

@Nasf-Fan Nasf-Fan force-pushed the Nasf-Fan/CR_baseline_20231208 branch 2 times, most recently from ea25607 to 923dba0 Compare March 3, 2024 06:44
@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/20/execution/node/1150/log

@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/21/execution/node/1173/log

@Nasf-Fan Nasf-Fan force-pushed the Nasf-Fan/CR_baseline_20231208 branch from 923dba0 to ec21d5a Compare March 5, 2024 09:48
@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/22/execution/node/288/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/22/execution/node/291/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/22/execution/node/350/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/22/execution/node/353/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/23/execution/node/319/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/23/execution/node/369/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/23/execution/node/330/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/23/execution/node/316/log

liw and others added 4 commits March 5, 2024 21:13
@liw @Nasf-Fan
Refactor map_dist
9174c1c 
Signed-off-by: Li Wei <[email protected]>
Required-githooks: true
@liw @Nasf-Fan
Fix server_init/fini
1c7b49b 
Signed-off-by: Li Wei <[email protected]>
Required-githooks: true
@Nasf-Fan
DAOS-15084 tests: add sleep in CR test when set_param
8d2b639 
For lower layer primary group initialization.

Signed-off-by: Fan Yong <[email protected]>
@Nasf-Fan
Merge branch 'master' into Nasf-Fan/CR_baseline_20231208
dcc83f2 
Signed-off-by: Fan Yong <[email protected]>
@Nasf-Fan Nasf-Fan force-pushed the Nasf-Fan/CR_baseline_20231208 branch from ec21d5a to 191e761 Compare March 5, 2024 13:39
@Nasf-Fan Nasf-Fan changed the title Merge master into CR at 20240130 Merge master into CR at 20240305 Mar 5, 2024
@Nasf-Fan
DAOS-15084 utils: new changelog entry for CR
2b16bc4 
Bump version and add new changelog entry for CR.

Signed-off-by: Fan Yong <[email protected]>
@Nasf-Fan Nasf-Fan force-pushed the Nasf-Fan/CR_baseline_20231208 branch from 191e761 to 8b4d4c4 Compare March 5, 2024 16:51
@Nasf-Fan
DAOS-15084 tests: fix some CR test scripts issues
48c6d6f 
1. For test_lost_majority_ps_replicas, remove "rdb-pool" from two
   ranks that contain the pool service replica.

2. For test_dangling_rank_entry, the rebuild process after CR is
   not related with CR logic but may cause test timeout, drop it.

3. More log messages when update CR bookmark.

4. More class tags for ddb tests.

Signed-off-by: Fan Yong <[email protected]>
@Nasf-Fan Nasf-Fan force-pushed the Nasf-Fan/CR_baseline_20231208 branch from 8b4d4c4 to 48c6d6f Compare March 5, 2024 17:01
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13604/26/execution/node/1452/log

@Nasf-Fan
Copy link
Contributor Author

Nasf-Fan commented Mar 6, 2024

rebuild_simple timeout because of DAOS-15290.

@Nasf-Fan Nasf-Fan closed this Apr 11, 2024
@Nasf-Fan Nasf-Fan deleted the Nasf-Fan/CR_baseline_20231208 branch April 11, 2024 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@Nasf-Fan @daosbuild1 @mjmac @ryon-jensen @liw @shimizukko @jolivier23 @kjacque