feat: update default size of bgworkers, add hbworkers

[sunng87]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

#4128

What's changed and what's your intention?

This patch changes default bg_worker runtime to half of cpu_num, to avoid cpu-intensive tasks to fill up all compute resources

It also use dedicated runtime to host heartbeat tasks, which is of higher priority.

Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR requires documentation updates.

[sunng87]

It seems we need to update sample configuration as well. But I just need to make sure this looks good to you before moving on.

[evenyag]

This should be a workaround. We might need to implement some other approaches to limit the resources used by background tasks.

[evenyag]

There is an issue related to the config test: #4126

Options related to the number of cores and the host memory are unstable in tests. You might encounter the same problem.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.60%. Comparing base (d8b51cf) to head (f969a46).
Report is 22 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4129      +/-   ##
==========================================
- Coverage   85.40%   84.60%   -0.80%     
==========================================
  Files         994     1022      +28     
  Lines      174087   179017    +4930     
==========================================
+ Hits       148677   151466    +2789     
- Misses      25410    27551    +2141     

[sunng87]

Just updated defaults and sample configuration. PTAL @evenyag @MichaelScofield

[evenyag]

Just updated defaults and sample configuration. PTAL @evenyag @MichaelScofield

@sunng87 I realize that we don't need to make the size of the heartbeat runtime configurable. If sending the heartbeat occupies one core for the whole time, there must be something wrong...... I'd suggest not exposing hb_rt_size.

[sunng87]

@evenyag Sounds good. We need to keep to UI as minimal as possible. We only expose it when there is a real case.