Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-16653 pool: Batch crt events (#15230) #15302

Merged
merged 1 commit into from
Oct 18, 2024
Merged

DAOS-16653 pool: Batch crt events (#15230) #15302

merged 1 commit into from
Oct 18, 2024

Conversation

liw
Copy link
Contributor

@liw liw commented Oct 11, 2024

When multiple engines become unavailable around the same time, if a pool cannot tolerate the unavailability of those engines, it is sometimes desired that the pool would not exclude any of the engines. Hence, this patch introduces a CaRT event delay, tunable via the server-side environment variable, CRT_EVENT_DELAY, so that the events signaling the unavailability of those engines will be handled in hopefully one batch, giving pool_svc_update_map_internal a chance to reject the pool map update based on the RF check.

When the RF check rejects a pool map change, we should revisit the corresponding events later, rather than simply throwing them away. This patch improves this case by returning the events back to the event queue, and pause the queue handling until next new event or pool map update.

  • Introduce event sets: pool_svc_event_set. Now the event queue can be simplified to just one event set.

  • Add the ability to pause and resume the event handling: pse_paused.

  • Track the time when the latest event was queued: pse_time.

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 11, 2024 •
edited
Loading

Ticket title is 'PS to wait for group stabilization before handling events'
Status is 'Awaiting backport'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-16653

@liw liw force-pushed the liw/ps-events-2.6 branch from 787d26f to c8a191e Compare October 15, 2024 00:58
@github-actions github-actions bot added the priority Ticket has high priority (automatically managed) label Oct 15, 2024
@liw liw added clean-cherry-pick Cherry-pick from another branch that did not require additional edits approved-to-merge PR has received release branch merge approval release-2.6.2 Targeted for release 2.6.2 and removed priority Ticket has high priority (automatically managed) labels Oct 15, 2024
@liw liw force-pushed the liw/ps-events-2.6 branch from c8a191e to d7ebf20 Compare October 15, 2024 23:19
@github-actions github-actions bot added the priority Ticket has high priority (automatically managed) label Oct 15, 2024
@liw
DAOS-16653 pool: Batch crt events (#15230)
b4d115d 
* DAOS-16653 pool: Batch crt events

When multiple engines become unavailable around the same time, if a pool
cannot tolerate the unavailability of those engines, it is sometimes
desired that the pool would not exclude any of the engines. Hence, this
patch introduces a CaRT event delay, tunable via the server-side
environment variable, CRT_EVENT_DELAY, so that the events signaling the
unavailability of those engines will be handled in hopefully one batch,
giving pool_svc_update_map_internal a chance to reject the pool map
update based on the RF check.

When the RF check rejects a pool map change, we should revisit the
corresponding events later, rather than simply throwing them away. This
patch improves this case by returning the events back to the event
queue, and pause the queue handling until next new event or pool map
update.

  - Introduce event sets: pool_svc_event_set. Now the event queue can be
    simplified to just one event set.

  - Add the ability to pause and resume the event handling: pse_paused.

  - Track the time when the latest event was queued: pse_time.

Skip-nlt: true
Signed-off-by: Li Wei <[email protected]>
Required-githooks: true
@liw liw force-pushed the liw/ps-events-2.6 branch from d7ebf20 to b4d115d Compare October 17, 2024 23:22
@liw liw marked this pull request as ready for review October 18, 2024 07:21
@liw liw requested review from a team as code owners October 18, 2024 07:21
@liw
Copy link
Contributor Author

liw commented Oct 18, 2024

Skipped NLT because build 1 to 3 were all blocked by the NLT warning on cont_open.

@gnailzenh gnailzenh requested a review from liuxuezhao October 18, 2024 07:54
@liw liw requested a review from a team October 18, 2024 08:12
@gnailzenh gnailzenh merged commit 60d4b5d into release/2.6 Oct 18, 2024
50 of 53 checks passed
@gnailzenh gnailzenh deleted the liw/ps-events-2.6 branch October 18, 2024 08:39
kccain pushed a commit that referenced this pull request Oct 18, 2024
@liw @kccain
DAOS-16653 pool: Batch crt events (#15230) (#15302)
8484ce1 
* DAOS-16653 pool: Batch crt events

When multiple engines become unavailable around the same time, if a pool
cannot tolerate the unavailability of those engines, it is sometimes
desired that the pool would not exclude any of the engines. Hence, this
patch introduces a CaRT event delay, tunable via the server-side
environment variable, CRT_EVENT_DELAY, so that the events signaling the
unavailability of those engines will be handled in hopefully one batch,
giving pool_svc_update_map_internal a chance to reject the pool map
update based on the RF check.

When the RF check rejects a pool map change, we should revisit the
corresponding events later, rather than simply throwing them away. This
patch improves this case by returning the events back to the event
queue, and pause the queue handling until next new event or pool map
update.

  - Introduce event sets: pool_svc_event_set. Now the event queue can be
    simplified to just one event set.

  - Add the ability to pause and resume the event handling: pse_paused.

  - Track the time when the latest event was queued: pse_time.

Signed-off-by: Li Wei <[email protected]>
mjean308 pushed a commit that referenced this pull request Oct 23, 2024
@liw @mjean308
DAOS-16653 pool: Batch crt events (#15230) (#15302)
0ef23e6 
* DAOS-16653 pool: Batch crt events

When multiple engines become unavailable around the same time, if a pool
cannot tolerate the unavailability of those engines, it is sometimes
desired that the pool would not exclude any of the engines. Hence, this
patch introduces a CaRT event delay, tunable via the server-side
environment variable, CRT_EVENT_DELAY, so that the events signaling the
unavailability of those engines will be handled in hopefully one batch,
giving pool_svc_update_map_internal a chance to reject the pool map
update based on the RF check.

When the RF check rejects a pool map change, we should revisit the
corresponding events later, rather than simply throwing them away. This
patch improves this case by returning the events back to the event
queue, and pause the queue handling until next new event or pool map
update.

  - Introduce event sets: pool_svc_event_set. Now the event queue can be
    simplified to just one event set.

  - Add the ability to pause and resume the event handling: pse_paused.

  - Track the time when the latest event was queued: pse_time.

Signed-off-by: Li Wei <[email protected]>
daltonbohning pushed a commit that referenced this pull request Oct 23, 2024
@liw @daltonbohning
DAOS-16653 pool: Batch crt events (#15230) (#15302)
e7aea30 
Skip-build: true

When multiple engines become unavailable around the same time, if a pool
cannot tolerate the unavailability of those engines, it is sometimes
desired that the pool would not exclude any of the engines. Hence, this
patch introduces a CaRT event delay, tunable via the server-side
environment variable, CRT_EVENT_DELAY, so that the events signaling the
unavailability of those engines will be handled in hopefully one batch,
giving pool_svc_update_map_internal a chance to reject the pool map
update based on the RF check.

When the RF check rejects a pool map change, we should revisit the
corresponding events later, rather than simply throwing them away. This
patch improves this case by returning the events back to the event
queue, and pause the queue handling until next new event or pool map
update.

  - Introduce event sets: pool_svc_event_set. Now the event queue can be
    simplified to just one event set.

  - Add the ability to pause and resume the event handling: pse_paused.

  - Track the time when the latest event was queued: pse_time.

Signed-off-by: Li Wei <[email protected]>
Signed-off-by: Dalton Bohning <[email protected]>
daltonbohning pushed a commit that referenced this pull request Oct 23, 2024
@liw @daltonbohning
DAOS-16653 pool: Batch crt events (#15230) (#15302)
50d1457 
* DAOS-16653 pool: Batch crt events

When multiple engines become unavailable around the same time, if a pool
cannot tolerate the unavailability of those engines, it is sometimes
desired that the pool would not exclude any of the engines. Hence, this
patch introduces a CaRT event delay, tunable via the server-side
environment variable, CRT_EVENT_DELAY, so that the events signaling the
unavailability of those engines will be handled in hopefully one batch,
giving pool_svc_update_map_internal a chance to reject the pool map
update based on the RF check.

When the RF check rejects a pool map change, we should revisit the
corresponding events later, rather than simply throwing them away. This
patch improves this case by returning the events back to the event
queue, and pause the queue handling until next new event or pool map
update.

  - Introduce event sets: pool_svc_event_set. Now the event queue can be
    simplified to just one event set.

  - Add the ability to pause and resume the event handling: pse_paused.

  - Track the time when the latest event was queued: pse_time.

Signed-off-by: Li Wei <[email protected]>
@mjean308 mjean308 mentioned this pull request Oct 24, 2024
Closed
18 tasks
@juszhan1 juszhan1 mentioned this pull request Oct 28, 2024
Closed
This was referenced Oct 28, 2024
Closed
Closed
@mjmac mjmac mentioned this pull request Nov 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liuxuezhao liuxuezhao liuxuezhao approved these changes

@gnailzenh gnailzenh gnailzenh approved these changes

Assignees
No one assigned
Labels
approved-to-merge PR has received release branch merge approval clean-cherry-pick Cherry-pick from another branch that did not require additional edits priority Ticket has high priority (automatically managed) release-2.6.2 Targeted for release 2.6.2
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@liw @liuxuezhao @gnailzenh