storage: introduce new backend localdisk in nydusd #972

adamqqqplay · 2022-12-21T09:15:02Z

The localdisk backend adds support for storing images in disks. In this scenario, each layer of the blob is stored in partitions, and multiple partitions are addressed in the local raw disk via the GUID partition table (GPT), which means that this disk stores the entire image.

Currently, generating a localdisk image through nydusify is not supported for the time being. You need to use the nydus-localdisk tool to complete this step.
Test document located at: https://github.com/adamqqqplay/nydus-localdisk/blob/master/README.md

This PR updates: #971

Signed-off-by: Qinqi Qu [email protected]

anolis-bot · 2022-12-21T09:16:15Z

@adamqqqplay , the title of this PR starts with "WIP" or "[WIP]", so skip testing for now.

changweige · 2022-12-22T02:37:20Z

storage/src/backend/localdisk.rs

+    }
+
+    // Disk names in GPT tables cannot store full 64-byte blob ids, so we should truncate them to 32 bytes.
+    fn truncate_blob_id(blob_id: &str) -> LocalDiskResult<&str> {


How about storing binary rather than strings? Then 32 bytes can hold the whole blob id

How about storing binary rather than strings? Then 32 bytes can hold the whole blob id

Very good suggestion, I will try to modify this.

I tried to use the storage method of Hex to String, but there are still problems in the implementation, and I will improve it later.

In the future, I will use two different methods to obtain the blob_id in get_blob() to maintain compatibility with different versions of the localdisk image.

How about storing binary rather than strings? Then 32 bytes can hold the whole blob id

Problem fixed in 7f8a808 @changweige

changweige · 2022-12-22T02:38:47Z

How to create partitions filled with blobs and run with this PR?

adamqqqplay · 2022-12-23T08:05:48Z

How to create partitions filled with blobs and run with this PR?

https://github.com/adamqqqplay/nydus-localdisk
You could use this small tools to generate a localdisk image file and then mount it to the server via a loop device. I'll be porting this feature to nydusify shortly.

anolis-bot · 2023-02-01T09:56:28Z

@adamqqqplay , the title of this PR starts with "WIP" or "[WIP]", so skip testing for now.

anolis-bot · 2023-02-01T10:48:28Z

@adamqqqplay , the title of this PR starts with "WIP" or "[WIP]", so skip testing for now.

anolis-bot · 2023-02-01T11:03:20Z

@adamqqqplay , the title has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/47122

anolis-bot · 2023-02-01T11:10:51Z

@adamqqqplay , The CI test is completed, please check result:

Test Case		Test Result
merge-target-branch		✅ SUCCESS
build-docker-image		✅ SUCCESS
compile-nydus		✅ SUCCESS
compile-ctr-remote		✅ SUCCESS
compile-nydus-snapshotter		✅ SUCCESS
start-nydus-snapshotter-config-containerd		✅ SUCCESS
run-container-with-nydus-image		✅ SUCCESS

Congratulations, your test job passed!

anolis-bot · 2023-02-02T03:13:58Z

@adamqqqplay , the title has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/47247

anolis-bot · 2023-02-02T03:22:28Z

@adamqqqplay , The CI test is completed, please check result:

Test Case		Test Result
merge-target-branch		✅ SUCCESS
build-docker-image		✅ SUCCESS
compile-nydus		✅ SUCCESS
compile-ctr-remote		✅ SUCCESS
compile-nydus-snapshotter		✅ SUCCESS
start-nydus-snapshotter-config-containerd		✅ SUCCESS
run-container-with-nydus-image		❌ FAIL

Sorry, your test job failed. Please get the details in the link.

anolis-bot · 2023-02-06T11:09:26Z

@adamqqqplay , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/48450

anolis-bot · 2023-02-06T11:17:42Z

@adamqqqplay , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		✅ SUCCESS
compile ctr remote		✅ SUCCESS
compile nydus snapshotter		✅ SUCCESS
run container with rafs		✅ SUCCESS
run container with zran		✅ SUCCESS

Congratulations, your test job passed!

imeoer · 2023-02-06T11:53:21Z

docs/nydusd.md

@@ -151,6 +151,25 @@ We are working on enabling cloud-hypervisor support for nydus.
 }
 ```

+##### Localdisk Backend
+


Can we explain what scenario will use localdisk and how we make a localdisk device here?

Ok, I'll add more description here.

imeoer · 2023-02-06T12:04:30Z

storage/src/backend/localdisk.rs

+    }
+
+    // Disk names in GPT tables cannot store full 64-byte blob ids, so we should truncate them to 32 bytes.
+    fn truncate_blob_id(blob_id: &str) -> LocalDiskResult<&str> {


If the blob id is truncated in GPT table, how do we ensure no conflicts between different blobs with the same sha256 prefix?

When the output of SHA-256 is truncated to 128 bits, its collision resistance is still 2^64, which is still enough to prevent blob collisions. And in the SHA standard, the result is also allowed to be truncated.

Please refer to:
https://security.stackexchange.com/questions/34796/truncating-the-output-of-sha256-to-128-bits
https://security.stackexchange.com/questions/72673/how-bad-is-it-to-truncate-a-hash/72685#72685

Despite this, I'll try to store the full blob id in the future, while maintaining compatibility with images generated in the past.

If the blob id is truncated in GPT table, how do we ensure no conflicts between different blobs with the same sha256 prefix?

Problem fixed in latest commit. Now the full 64-bit layer digest (blob id) stored in the Partition Name and Partition GUID respectively.

At the same time, forward and backward compatibility is maintained.

anolis-bot · 2023-02-07T09:57:27Z

@adamqqqplay , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/48844

anolis-bot · 2023-02-07T10:06:12Z

@adamqqqplay , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		✅ SUCCESS
compile ctr remote		✅ SUCCESS
compile nydus snapshotter		✅ SUCCESS
run container with rafs		✅ SUCCESS
run container with zran		✅ SUCCESS

Congratulations, your test job passed!

imeoer · 2023-02-08T01:48:19Z

docs/nydusd.md

@@ -151,6 +151,31 @@ We are working on enabling cloud-hypervisor support for nydus.
 }
 ```

+##### Localdisk Backend


Maybe mark this Localdisk Backend (Experimental) before we have supported it in nydusify.

Maybe mark this Localdisk Backend (Experimental) before we have supported it in nydusify.

OK, that's good.

anolis-bot · 2023-02-08T06:54:14Z

@adamqqqplay , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/49092

anolis-bot · 2023-02-08T07:01:44Z

@adamqqqplay , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		✅ SUCCESS
compile ctr remote		✅ SUCCESS
compile nydus snapshotter		✅ SUCCESS
run container with rafs		✅ SUCCESS
run container with zran		✅ SUCCESS

Congratulations, your test job passed!

This patch adds a new storage backend localdisk which supports reading blobs on disk. In this scenario, each layer of the blob is stored in partitions, and multiple partitions are addressed in the local raw disk via the GUID partition table (GPT), which means that this disk stores the entire image. Signed-off-by: Qinqi Qu <[email protected]>

anolis-bot · 2023-02-08T09:04:12Z

@adamqqqplay , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/49212

anolis-bot · 2023-02-08T09:16:12Z

@adamqqqplay , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		✅ SUCCESS
compile ctr remote		✅ SUCCESS
compile nydus snapshotter		✅ SUCCESS
run container with rafs		✅ SUCCESS
run container with zran		✅ SUCCESS

Congratulations, your test job passed!

changweige reviewed Dec 22, 2022

View reviewed changes

adamqqqplay force-pushed the nydusd-localdisk-backend branch from 0abd285 to b0684c1 Compare February 1, 2023 09:56

adamqqqplay requested a review from a team as a code owner February 1, 2023 09:56

adamqqqplay requested review from imeoer, hhhhsdxxxx and hsiangkao and removed request for a team February 1, 2023 09:56

adamqqqplay force-pushed the nydusd-localdisk-backend branch from b0684c1 to d2dabdc Compare February 1, 2023 10:48

dragonflyoss deleted a comment from anolis-bot Feb 1, 2023

adamqqqplay changed the title ~~[WIP] storage: introduce new backend localdisk in nydusd~~ storage: introduce new backend localdisk in nydusd Feb 1, 2023

anolis-bot added the anolis_testing label Feb 1, 2023

anolis-bot added anolis_test_pass and removed anolis_testing labels Feb 1, 2023

anolis-bot added anolis_testing and removed anolis_test_pass labels Feb 2, 2023

adamqqqplay requested review from jiangliu and changweige and removed request for hhhhsdxxxx and hsiangkao February 2, 2023 03:14

anolis-bot added anolis_test_fail and removed anolis_testing labels Feb 2, 2023

anolis-bot added anolis_test_pass and removed anolis_testing labels Feb 6, 2023

imeoer reviewed Feb 6, 2023

View reviewed changes

adamqqqplay force-pushed the nydusd-localdisk-backend branch from d1c0649 to 7f8a808 Compare February 7, 2023 09:57

anolis-bot added anolis_testing and removed anolis_test_pass labels Feb 7, 2023

anolis-bot added anolis_test_pass and removed anolis_testing labels Feb 7, 2023

imeoer reviewed Feb 8, 2023

View reviewed changes

adamqqqplay force-pushed the nydusd-localdisk-backend branch from 7f8a808 to 5b54510 Compare February 8, 2023 06:53

anolis-bot added anolis_testing and removed anolis_test_pass labels Feb 8, 2023

anolis-bot added anolis_test_pass and removed anolis_testing labels Feb 8, 2023

adamqqqplay force-pushed the nydusd-localdisk-backend branch from 5b54510 to 44ced99 Compare February 8, 2023 09:03

anolis-bot added anolis_testing and removed anolis_test_pass labels Feb 8, 2023

anolis-bot added anolis_test_pass and removed anolis_testing labels Feb 8, 2023

imeoer approved these changes Feb 8, 2023

View reviewed changes

imeoer merged commit ab07dec into dragonflyoss:master Feb 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

storage: introduce new backend localdisk in nydusd #972

storage: introduce new backend localdisk in nydusd #972

adamqqqplay commented Dec 21, 2022 •

edited

Loading

anolis-bot commented Dec 21, 2022

changweige Dec 22, 2022

adamqqqplay Dec 23, 2022

adamqqqplay Feb 3, 2023

adamqqqplay Feb 7, 2023

changweige commented Dec 22, 2022

adamqqqplay commented Dec 23, 2022

anolis-bot commented Feb 1, 2023

anolis-bot commented Feb 1, 2023

anolis-bot commented Feb 1, 2023

anolis-bot commented Feb 1, 2023

anolis-bot commented Feb 2, 2023

anolis-bot commented Feb 2, 2023

anolis-bot commented Feb 6, 2023

anolis-bot commented Feb 6, 2023

imeoer Feb 6, 2023

adamqqqplay Feb 7, 2023

imeoer Feb 6, 2023

adamqqqplay Feb 7, 2023

adamqqqplay Feb 7, 2023

anolis-bot commented Feb 7, 2023

anolis-bot commented Feb 7, 2023

imeoer Feb 8, 2023

adamqqqplay Feb 8, 2023

anolis-bot commented Feb 8, 2023

anolis-bot commented Feb 8, 2023

anolis-bot commented Feb 8, 2023

anolis-bot commented Feb 8, 2023

storage: introduce new backend localdisk in nydusd #972

storage: introduce new backend localdisk in nydusd #972

Conversation

adamqqqplay commented Dec 21, 2022 • edited Loading

anolis-bot commented Dec 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

changweige commented Dec 22, 2022

adamqqqplay commented Dec 23, 2022

anolis-bot commented Feb 1, 2023

anolis-bot commented Feb 1, 2023

anolis-bot commented Feb 1, 2023

anolis-bot commented Feb 1, 2023

anolis-bot commented Feb 2, 2023

anolis-bot commented Feb 2, 2023

anolis-bot commented Feb 6, 2023

anolis-bot commented Feb 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anolis-bot commented Feb 7, 2023

anolis-bot commented Feb 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anolis-bot commented Feb 8, 2023

anolis-bot commented Feb 8, 2023

anolis-bot commented Feb 8, 2023

anolis-bot commented Feb 8, 2023

adamqqqplay commented Dec 21, 2022 •

edited

Loading