DAOS-4881 sched: CPU relaxing on idle #4332

NiuYawei · 2021-01-19T08:38:45Z

Scheduler will put CPU relaxing for a short interval when the
xstream is idle. The default relax mode is that scheduler calls
sleep("short_timeout") on idle, if the cart progress interface
is able to relax CPU (depends on network provider), env var
SCHED_RELAX_MODE=net could be set to make the scheduler calls
crt_progress("short_timeout") to realx while waiting on request.

Following changes are made to support this feature:

Introduced sched_cond_wait(), it's called by ULTs being blocked
on a condition for a long time (like RDB daemons), so that
scheduler will be able to know how many ULTs are blocked on a
non-immediate event.
Introduced sched_create_thread/task() to bump 'busy timestamp'
once a ULT is created for certain xstream, so that scheduler is
able to know if the xstream is busy recently.
Intrdouced DSS_ULT_FL_PERIODIC flag, it'll be used by the thread
creation caller who doesn't want 'busy timestamp' being bumped.
It's mainly for EC aggregation periodical collective calls.
Necessary changes to DTX and EC aggregation ULTs to avoid keeping
the xstream 'busy'.
Added few env variables:
1. SCHED_PRIO_DISABLED to disable scheduler priority queue.
2. SCHED_RELAX_DISABLED to disable CPU relaxing on idle.
3. SCHED_RELAX_INTVL to configure the short relaxing interval.
  (in milli-seconds, 1 msec by default)
4. SCHED_STATS_INTVL to periodically print relaxing stats on
  console (in seconds).
5. SCHED_RELAX_MODE to change relax mode, "sleep" is the default
  mode, set to any other string for "block cart progress" mode.

Signed-off-by: Niu Yawei [email protected]

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

NiuYawei · 2021-01-22T06:56:19Z

The patch hasn't been fully tested yet, but I think you may have an early review if you have time. Thanks.

Scheduler will put CPU relaxing for a short interval when the xstream is idle. The default relax mode is that scheduler calls sleep("short_timeout") on idle, if the cart progress interface is able to relax CPU (depends on network provider), env var SCHED_RELAX_MODE=net could be set to make the scheduler calls crt_progress("short_timeout") to realx while waiting on request. Following changes are made to support this feature: - Introduced sched_cond_wait(), it's called by ULTs being blocked on a condition for a long time (like RDB daemons), so that scheduler will be able to know how many ULTs are blocked on a non-immediate event. - Introduced sched_create_thread/task() to bump 'busy timestamp' once a ULT is created for certain xstream, so that scheduler is able to know if the xstream is busy recently. - Intrdouced DSS_ULT_FL_PERIODIC flag, it'll be used by the thread creation caller who doesn't want 'busy timestamp' being bumped. It's mainly for EC aggregation periodical collective calls. - Necessary changes to DTX and EC aggregation ULTs to avoid keeping the xstream 'busy'. - Added few env variables: 1. SCHED_PRIO_DISABLED to disable scheduler priority queue. 2. SCHED_RELAX_DISABLED to disable CPU relaxing on idle. 3. SCHED_RELAX_INTVL to configure the short relaxing interval. (in milli-seconds, 50 msecs by default) 4. SCHED_STATS_INTVL to periodically print relaxing stats on console (in seconds). 5. SCHED_RELAX_MODE to change relax mode, "sleep" is the default mode, set to any other string for "block cart progress" mode. Signed-off-by: Niu Yawei <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Merge remote-tracking branch 'github/master' into DAOS-4881-relax Conflicts: src/dtx/dtx_common.c Signed-off-by: Niu Yawei <[email protected]>

Shrink the default relax internval from 50ms to 1ms. Signed-off-by: Niu Yawei <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

ashleypittman · 2021-01-28T07:47:25Z

This had hit some assertion failures and deadlocked in CI overnight so I killed the job given the length of the queue.

NiuYawei · 2021-01-28T08:57:39Z

Not sure why the test is cancelled. Already finished tests have some failures, some are from DAOS-6544, others are test timeout, no obvious errors in the log, not sure if they are caused by DAOS-6544 as well.

daosbuild1

LGTM. No errors found by checkpatch.

NiuYawei · 2021-01-28T09:00:00Z

This had hit some assertion failures and deadlocked in CI overnight so I killed the job given the length of the queue.

@ashleypittman do you know what are the assertion failures? I didn't find them.

daosbuild1 · 2021-01-28T09:05:40Z

Test stage Build on Leap 15 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-4332/8/execution/node/389/log

daosbuild1 · 2021-01-28T09:07:36Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-4332/8/execution/node/346/log

daosbuild1 · 2021-01-28T09:08:01Z

Test stage Build RPM on CentOS 7 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-4332/8/execution/node/306/log

The DTX batched commit ULT will sleep during relax model. Under such case, after new DTX is added into the DTX CoS cache, the leader will check whehter the count of cached committable DTX exceeds the batched commit threshold, if yes, wakeup the batched commit ULT. Signed-off-by: Fan Yong <[email protected]> Co-authored-by: Niu Yawei <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

Merge remote-tracking branch 'github/master' into DAOS-4881-relax Conflicts: src/dtx/dtx_common.c Signed-off-by: Niu Yawei <[email protected]>

ashleypittman · 2021-02-03T20:56:33Z

This has conflicts that need to be resolved.

daosbuild1

LGTM. No errors found by checkpatch.

NiuYawei · 2021-02-04T02:05:36Z

Reviewers, the patch passed CI, but got a conflict, I just fixed the conflict and did another push. Please review it again. Thanks.

Merge remote-tracking branch 'github/master' into DAOS-4881-relax Conflicts: src/iosrv/srv.c Signed-off-by: Niu Yawei <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

liw

Might also want to document the new env vars (what's the right place? doc/admin/env_variables.md?) later.

NiuYawei · 2021-02-04T15:16:41Z

@daos-stack/daos-gatekeeper All tests passed, but three new NLT errors showed up after merging PR 4490. Could we land it and solve the NLT errors in follow-on patch?

ashleypittman · 2021-02-04T15:17:31Z

The NLT error is "dss_module_key_get() Assertion 'dtls != NULL' failed" and if landed as-is would cause all master builds and PRs to fail.

Fix a merge error. Signed-off-by: Niu Yawei <[email protected]>

NiuYawei · 2021-02-04T16:01:17Z

@ashleypittman oh, right, that's caused by a merge conflict. Thanks. sigh, I have to push a commit again.

daosbuild1

LGTM. No errors found by checkpatch.

sylviachanoiyee · 2021-02-04T16:16:35Z

Niu, I have restarted your build with a higher priority, running build 19 now. Build 18 should be canceled.

daosbuild1

LGTM. No errors found by checkpatch.

ashleypittman · 2021-02-04T16:42:02Z

@daos-stack/daos-gatekeeper All tests passed, but three new NLT errors showed up after merging PR 4490. Could we land it and solve the NLT errors in follow-on patch?

I just had a second look, the other two are in master anyway as of https://build.hpdd.intel.com/job/daos-stack/job/daos/job/master/3744/ this PR must have been merged with master and built before the relevent master build completed to be selected as a reference. A simple re-trigger or rebuilding of just the stage should have been enough to pass.

daosbuild1 reviewed Jan 19, 2021

View reviewed changes

NiuYawei force-pushed the DAOS-4881-relax branch from 4cf26e9 to e80b097 Compare January 20, 2021 08:44

daosbuild1 reviewed Jan 20, 2021

View reviewed changes

NiuYawei force-pushed the DAOS-4881-relax branch from e80b097 to 56e7814 Compare January 21, 2021 09:56

daosbuild1 reviewed Jan 21, 2021

View reviewed changes

NiuYawei force-pushed the DAOS-4881-relax branch from 56e7814 to 80e48bb Compare January 21, 2021 10:05

daosbuild1 reviewed Jan 21, 2021

View reviewed changes

NiuYawei force-pushed the DAOS-4881-relax branch from 80e48bb to 3d89b19 Compare January 22, 2021 06:39

daosbuild1 reviewed Jan 22, 2021

View reviewed changes

NiuYawei requested review from johannlombardi, liw, gnailzenh, Nasf-Fan, jgmoore-or and wangdi1 January 22, 2021 06:53

NiuYawei force-pushed the DAOS-4881-relax branch from 3d89b19 to fd670ac Compare January 25, 2021 05:51

daosbuild1 reviewed Jan 25, 2021

View reviewed changes

NiuYawei added 2 commits January 27, 2021 13:35

DAOS-4881 sched: merge master

9baf564

Merge remote-tracking branch 'github/master' into DAOS-4881-relax Conflicts: src/dtx/dtx_common.c Signed-off-by: Niu Yawei <[email protected]>

DAOS-4881 sched: shrink default relax interval

4d67d81

Shrink the default relax internval from 50ms to 1ms. Signed-off-by: Niu Yawei <[email protected]>

daosbuild1 reviewed Jan 27, 2021

View reviewed changes

Merge remote-tracking branch 'github/master' into DAOS-4881-relax

d6a5d88

daosbuild1 reviewed Jan 28, 2021

View reviewed changes

daosbuild1 reviewed Feb 3, 2021

View reviewed changes

DAOS-4881 sched: merge master

a25f28a

Merge remote-tracking branch 'github/master' into DAOS-4881-relax Conflicts: src/dtx/dtx_common.c Signed-off-by: Niu Yawei <[email protected]>

NiuYawei requested review from liw and wangdi1 February 4, 2021 00:45

daosbuild1 reviewed Feb 4, 2021

View reviewed changes

DAOS-4881 sched: merge master

0c2ad6b

Merge remote-tracking branch 'github/master' into DAOS-4881-relax Conflicts: src/iosrv/srv.c Signed-off-by: Niu Yawei <[email protected]>

daosbuild1 reviewed Feb 4, 2021

View reviewed changes

liw previously approved these changes Feb 4, 2021

View reviewed changes

wangdi1 previously approved these changes Feb 4, 2021

View reviewed changes

NiuYawei requested a review from a team February 4, 2021 15:15

DAOS-4881 sched: fix merge error

9eaf376

Fix a merge error. Signed-off-by: Niu Yawei <[email protected]>

NiuYawei dismissed stale reviews from wangdi1 and liw via 9eaf376 February 4, 2021 16:04

NiuYawei requested review from liw and wangdi1 February 4, 2021 16:05

daosbuild1 reviewed Feb 4, 2021

View reviewed changes

jgmoore-or approved these changes Feb 4, 2021

View reviewed changes

jolivier23 approved these changes Feb 4, 2021

View reviewed changes

jolivier23 merged commit fbf5746 into master Feb 4, 2021

jolivier23 deleted the DAOS-4881-relax branch February 4, 2021 19:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-4881 sched: CPU relaxing on idle #4332

DAOS-4881 sched: CPU relaxing on idle #4332

NiuYawei commented Jan 19, 2021 •

edited

Loading

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

NiuYawei commented Jan 22, 2021

daosbuild1 left a comment

daosbuild1 left a comment

ashleypittman commented Jan 28, 2021

NiuYawei commented Jan 28, 2021

daosbuild1 left a comment

NiuYawei commented Jan 28, 2021

daosbuild1 commented Jan 28, 2021

daosbuild1 commented Jan 28, 2021

daosbuild1 commented Jan 28, 2021

daosbuild1 left a comment

daosbuild1 left a comment

ashleypittman commented Feb 3, 2021

daosbuild1 left a comment

NiuYawei commented Feb 4, 2021

daosbuild1 left a comment

liw left a comment

NiuYawei commented Feb 4, 2021

ashleypittman commented Feb 4, 2021

NiuYawei commented Feb 4, 2021

daosbuild1 left a comment

sylviachanoiyee commented Feb 4, 2021 •

edited

Loading

daosbuild1 left a comment

ashleypittman commented Feb 4, 2021

DAOS-4881 sched: CPU relaxing on idle #4332

DAOS-4881 sched: CPU relaxing on idle #4332

Conversation

NiuYawei commented Jan 19, 2021 • edited Loading

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

NiuYawei commented Jan 22, 2021

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

ashleypittman commented Jan 28, 2021

NiuYawei commented Jan 28, 2021

daosbuild1 left a comment

Choose a reason for hiding this comment

NiuYawei commented Jan 28, 2021

daosbuild1 commented Jan 28, 2021

daosbuild1 commented Jan 28, 2021

daosbuild1 commented Jan 28, 2021

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

ashleypittman commented Feb 3, 2021

daosbuild1 left a comment

Choose a reason for hiding this comment

NiuYawei commented Feb 4, 2021

daosbuild1 left a comment

Choose a reason for hiding this comment

liw left a comment

Choose a reason for hiding this comment

NiuYawei commented Feb 4, 2021

ashleypittman commented Feb 4, 2021

NiuYawei commented Feb 4, 2021

daosbuild1 left a comment

Choose a reason for hiding this comment

sylviachanoiyee commented Feb 4, 2021 • edited Loading

daosbuild1 left a comment

Choose a reason for hiding this comment

ashleypittman commented Feb 4, 2021

NiuYawei commented Jan 19, 2021 •

edited

Loading

sylviachanoiyee commented Feb 4, 2021 •

edited

Loading