Merge #1771

1771: Introduce new CSAL ftl bdev r=tiagolobocastro a=MaisenbacherD

For this PR to be merged the PR openebs/spdk-rs#67
for spdk-rs needs to be resolved and the submodule reference must be
updated accordingly. :)

The SPDK ftl bdev allows to create a layered device with a (fast) cache
device for buffering writes that get eventually flushed out sequentially
to a base device. SDPK ftl is also known as the Cloud Storage
Acceleration Layer (CSAL).

This kind of device potentially enables the use of emerging storage
interfaces like Zoned Namespace (ZNS) or Flexible Data Placement (FDP)
capable NVMe devices. Up to this point, those NVMe command sets are not
yet supported in SPDK upstream. However, the acceleration aspect of a
fast cache device already adds value.
With future support of new devices for SDPK ftl, Mayastor would already
be capable of utilizing those features simply by upgrading SDPK.

For now, the `ftl_mount_fs` test cases are hidden behind the `ignore`
attribute because PCIe devices are required for this test until
SPDK v24.09 is picked up, which introduces variable sector size emulation
for ftl devices.
To run these tests use the following:
```
RUST_LOG=TRACE cargo test -- --test-threads 1 --test ftl_mount_fs --nocapture --ignored
```

This patch introduces a new ftl device uri scheme:
```
ftl:///<ftl-device-name>?bbdev=<dev-uri-nested-encoding>&cbdev=<dev-uri-nested-encoding>
```

`<dev-uri-nested-encoding>` can be any already valid device uri where '?'
are replaced with '!' and '&' are replaced with '|'.
With SPDK v24.05 only PCIe devices will work where the cache device is
formatted to 4KiB+64B LBA format and the base device to 4KiB LBA format.

From SPDK v24.09 on, any device with a block size of 4KiB will work for
both cache and base device.

CSAL paper reference:
https://dl.acm.org/doi/10.1145/3627703.3629566

Co-authored-by: Dennis Maisenbacher <[email protected]>

doc/build.md

@@ -127,7 +127,7 @@ cargo build --release
 ```
 
 **Want to run or hack on Mayastor?** _You need more configuration!_ See
-[running][doc-running], then [testing][doc-testing].
+[running][doc-run], then [testing][doc-test].
 
 Whilst the nix develop will allow you to build mayastor exactly as the image build, it might not have all the necessary
 components required for testing.

doc/test.md

@@ -29,9 +29,13 @@ Mayastor's unit tests, integration tests, and documentation tests via the conven
 Mayastor uses [spdk][spdk] which is quite senistive to threading. This means tests need to run one at a time:
 
 ```bash
-cargo test -- --test-threads 1
+cd io-engine
+RUST_LOG=TRACE cargo test -- --test-threads 1 --nocapture
 ```
 
+## Testing your own SPDK version
+To test your custom SPDK version please refere to the [spdk-rs documentation](https://github.com/openebs/spdk-rs/blob/develop/README.md#custom-spdk)
+
 ## Running the end-to-end test suite
 
 Mayastor does more complete, end-to-end testing testing with [`mocha`][mocha]. It requires some extra setup.
@@ -55,6 +59,29 @@ Then, to run the tests:
 ./node_modules/mocha/bin/mocha test_csi.js
 ```
 
+## Using PCIe NVMe devices in cargo tests while developing
+
+When developing new features, testing those with real PCIe devices in the process might come in handy.
+In order to do so, the PCIe device first needs to be bound to the vfio driver:
+
+```bash
+sudo PCI_ALLOWED="<PCI-ADDRESS>" ./spdk-rs/spdk/scripts/setup.sh
+```
+
+The bdev name in the cargo test case can then follow the PCIe URI pattern:
+
+```rust
+static BDEVNAME1: &str = "pcie:///<PCI-ADDRESS>";
+```
+
+After testing the device may be rebound to the NVMe driver:
+
+```bash
+sudo PCI_ALLOWED="<PCI-ADDRESS>" ./spdk-rs/spdk/scripts/setup.sh reset
+```
+
+Please do not submit pull requests with active cargo test cases that require PCIe devices to be present.
+
 [spdk]: https://spdk.io/
 [doc-run]: ./run.md
 [mocha]: https://mochajs.org/

io-engine/Cargo.toml

@@ -13,6 +13,7 @@ extended-tests = [] # Extended I/O engine tests: not intended for daily runs.
 fault-injection = [] # Enables fault injection code.
 nexus-io-tracing = [] # Enables nexus I/O tracing code.
 spdk-async-qpair-connect = [] # Enables async qpair connection.
+nvme-pci-tests = []
 
 [[bin]]
 name = "io-engine"
@@ -77,6 +78,7 @@ merge = "0.1.0"
 nix = { version = "0.29.0", default-features = false, features = ["hostname", "net", "socket", "ioctl"] }
 once_cell = "1.20.2"
 parking_lot = "0.12.3"
+percent-encoding = "2.3.1"
 pin-utils = "0.1.0"
 prost = "0.13.3"
 prost-derive = "0.13.3"

io-engine/src/bdev/dev.rs

@@ -39,6 +39,7 @@ pub(crate) mod uri {
     use crate::{
         bdev::{
             aio,
+            ftl,
             loopback,
             lvs,
             malloc,
@@ -64,6 +65,7 @@ pub(crate) mod uri {
             "bdev" | "loopback" => {
                 Ok(Box::new(loopback::Loopback::try_from(&url)?))
             }
+            "ftl" => Ok(Box::new(ftl::Ftl::try_from(&url)?)),
             "malloc" => Ok(Box::new(malloc::Malloc::try_from(&url)?)),
             "null" => Ok(Box::new(null_bdev::Null::try_from(&url)?)),
             // keeping nvmf scheme so existing tests(if any, setting this