prepare for nydusd daemonless #540

kevinXYin · 2022-06-30T03:33:02Z

This patchset is used for support nydusd daemonless for fscache scenario. After all blob data has been loaded to local storage, data flow is handled in kernel, nydusd daemon can exit.

Thit patchset basically does these following things:

improve blob prefetch, add retry mechanism to make sure all prefetch requests can be handled correctly.
add a periodic task to check all BlobCacheMgr's status , if all data is ready stops the prefetch workers, and mark this BlobCacheMgr is "data ready".
add data_all_ready field in metrics, then user can get data status of BlobCacheMgr, and kill nydusd after all data is ready.

It should be noted that, when cachefiles devfd released , the fscache volume will become DEAD state , then local cache can not be accessed by erofs. For now I hold the devfd in other user daemon via usd to walk around this limitation. I will figure out in-kernel solution to support this case later.

yqleng1987 · 2022-06-30T03:33:14Z

@kevinXYin , a new test job has been submitted. Please wait in patience.

yqleng1987 · 2022-06-30T03:36:31Z

@kevinXYin , The CI test is completed, please check result:

Test Case		Test Result
merge-target-branch		✅SUCCESS
build-docker-image		✅SUCCESS
compile-nydus		❌FAIL

Sorry, your test job failed. Please get the details in the link.

yqleng1987 · 2022-06-30T10:31:28Z

@kevinXYin , your pull request has been updated. A new test job will be submitted. Please wait in patience.

yqleng1987 · 2022-06-30T10:31:32Z

@kevinXYin , the test job has been submitted. Please wait in patience.

yqleng1987 · 2022-06-30T10:35:04Z

@kevinXYin , The CI test is completed, please check result:

Test Case		Test Result
merge-target-branch		✅SUCCESS
build-docker-image		✅SUCCESS
compile-nydus		❌FAIL

Sorry, your test job failed. Please get the details in the link.

yqleng1987 · 2022-06-30T12:21:26Z

@kevinXYin , your pull request has been updated. A new test job will be submitted. Please wait in patience.

yqleng1987 · 2022-06-30T12:23:16Z

@kevinXYin , the test job has been submitted. Please wait in patience.

yqleng1987 · 2022-06-30T12:29:03Z

@kevinXYin , The CI test is completed, please check result:

Test Case		Test Result
merge-target-branch		✅SUCCESS
build-docker-image		✅SUCCESS
compile-nydus		✅SUCCESS
compile-ctr-remote		✅SUCCESS
compile-nydus-snapshotter		✅SUCCESS
start-nydus-snapshotter-config-containerd		✅SUCCESS
run-container-with-nydus-image		✅SUCCESS

Congratulations, your test job passed!

xujihui1985 · 2022-07-01T06:11:23Z

@kevinXYin if nydusdaemon was deleted after blob downloaded complete, how can you garbage collect blobcache? for example, when diskusage is high and you need to release some blobcache to reduce the diskusage.

kevinXYin · 2022-07-01T07:59:44Z

Yes , indeed in this case we will lose gc capacity for blob cache, so whether to kill nydusd depends on the needs of the user.

On the other hand , I think we need logic to restart nydusd, for example, if we need to pull a new image after nydusd exit. Maybe we should also restart it when we need gc.

xujihui1985 · 2022-07-01T08:42:05Z

@kevinXYin it's complicate that we need to decide when to restart the daemon, it's before the gc or after gc, probably before gc other wise there is chance EIO will happened, also the snapshotter should have a mechanism to know which daemon to restart, this is also complicate because you can hardly tell from the blobcache which daemon it belongs to.

kevinXYin · 2022-07-01T09:40:46Z

@kevinXYin it's complicate that we need to decide when to restart the daemon, it's before the gc or after gc, probably before gc other wise there is chance EIO will happened, also the snapshotter should have a mechanism to know which daemon to restart, this is also complicate because you can hardly tell from the blobcache which daemon it belongs to.
For fscache scenario , only one daemon is allowed , this is a limitation of kernel. what do you mean "which daemon it belongs to"? did I miss some here?

when get error during blob prefetch request processing, resend a new prefetch request. Make sure all the request can be handled correctly. Signed-off-by: Xin Yin <[email protected]>

Add a new fn check_stat() for BlobCacheMgr, this is used to check data chunk status. If all data chunks are ready stop prefetch workers belonging to the BlobCacheMgr, and mark all data ready in metrics. So far, only implement this func for fscache. Signed-off-by: Xin Yin <[email protected]>

yqleng1987 · 2022-07-11T06:34:57Z

@kevinXYin , your pull request has been updated. A new test job will be submitted. Please wait in patience.

yqleng1987 · 2022-07-11T06:36:49Z

@kevinXYin , the test job has been submitted. Please wait in patience.

yqleng1987 · 2022-07-11T06:42:22Z

@kevinXYin , The CI test is completed, please check result:

Test Case		Test Result
merge-target-branch		✅SUCCESS
build-docker-image		✅SUCCESS
compile-nydus		✅SUCCESS
compile-ctr-remote		✅SUCCESS
compile-nydus-snapshotter		✅SUCCESS
start-nydus-snapshotter-config-containerd		✅SUCCESS
run-container-with-nydus-image		✅SUCCESS

Congratulations, your test job passed!

kevinXYin · 2022-07-11T06:59:42Z

@changweige sorry for replying late~. I have updated the pull request which includes the following changes as we discussed.

use filter as rust style in get_blobs_num().
make blob prefetch retry asynchronous.
use store instead of compare_exchange in check_stat().
use an async context to fill bootstrap cache file on open cmd.

changweige · 2022-07-12T12:49:41Z

Shall we remove WIP telling reviewers it is ready to be merged or do we have further works?

kevinXYin · 2022-07-13T04:04:58Z

Shall we remove WIP telling reviewers it is ready to be merged or do we have further works?

yeah , updated , I have tested the current changes does not affect normal workflow. The further work will introduce an in-kernel solution for walk around fscache volume DEAD limitation. But I'm busy with other things now , so it may be delayed for a while.

storage/src/factory.rs

storage/src/cache/fscache/mod.rs

src/bin/nydusd/fs_cache.rs

We need a periodic task to check all the BlobCacheMgr's data cache status for fscache scenario, reuse the cache-flusher thread. Signed-off-by: Xin Yin <[email protected]>

whan handle open requests for bootstrap, feed all data at once.For supporting daemonless we need fill all cache files before nydusd exit. but bootstrap does not support prefetch. Also it may improve performance. Signed-off-by: Xin Yin <[email protected]>

yqleng1987 · 2022-07-14T07:35:57Z

@kevinXYin , your pull request has been updated. A new test job will be submitted. Please wait in patience.

yqleng1987 · 2022-07-14T07:35:58Z

@kevinXYin , your test job has passed, and no need to test again.

changweige

LGTM

kevinXYin requested review from bergwolf, imeoer, jiangliu, changweige and hsiangkao June 30, 2022 03:33

yqleng1987 added the anolis_testing label Jun 30, 2022

yqleng1987 added anolis_test_fail and removed anolis_testing labels Jun 30, 2022

kevinXYin force-pushed the daemonless branch from fcfe8dc to 565dca3 Compare June 30, 2022 10:31

yqleng1987 removed the anolis_test_fail label Jun 30, 2022

yqleng1987 added the anolis_testing label Jun 30, 2022

yqleng1987 added anolis_test_fail and removed anolis_testing labels Jun 30, 2022

kevinXYin force-pushed the daemonless branch from 565dca3 to d5b49ca Compare June 30, 2022 12:19

yqleng1987 removed the anolis_test_fail label Jun 30, 2022

yqleng1987 added the anolis_testing label Jun 30, 2022

yqleng1987 added anolis_test_pass and removed anolis_testing labels Jun 30, 2022

Xin Yin added 2 commits July 8, 2022 14:09

storage: add retry mechanism for blob prefetch

1984304

when get error during blob prefetch request processing, resend a new prefetch request. Make sure all the request can be handled correctly. Signed-off-by: Xin Yin <[email protected]>

kevinXYin force-pushed the daemonless branch from d5b49ca to e74b011 Compare July 11, 2022 06:34

yqleng1987 removed the anolis_test_fail label Jul 11, 2022

yqleng1987 added the anolis_testing label Jul 11, 2022

yqleng1987 added anolis_test_pass and removed anolis_testing labels Jul 11, 2022

hsiangkao approved these changes Jul 12, 2022

View reviewed changes

kevinXYin changed the title ~~[WIP]: prepare for nydusd daemonless~~ prepare for nydusd daemonless Jul 13, 2022