Skip to content

Commit

Permalink
docs: add requirements for running with ANA
Browse files Browse the repository at this point in the history
Document the prerequisites for Asymmetric Namespace Access (ANA),
including rebuilding the kernel for NixOS.
  • Loading branch information
jonathan-teh committed Mar 11, 2021
1 parent f652a5f commit 1a924d3
Showing 1 changed file with 38 additions and 6 deletions.
44 changes: 38 additions & 6 deletions doc/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ It will also need at least 512 2 MB Hugepages configured.

> Learn more about hugepages: [parts 1][hugepages-lwn-one], [2][hugepages-lwn-two],
> [3][hugepages-lwn-three], [4][hugepages-lwn-four], [5][hugepages-lwn-five].
In NixOS:

```nix
Expand Down Expand Up @@ -78,7 +78,7 @@ In order to use the full feature set of Mayastor, some or all of the following c
+ `nvme_loop`: NVMe Loop Device support

To load these on NixOS:

```nix
# /etc/nixos/configuration.nix
boot.kernelModules = [
Expand All @@ -91,7 +91,40 @@ In order to use the full feature set of Mayastor, some or all of the following c
```bash
modprobe nbd nvmet nvmet_rdma nvme_fabrics nvme_tcp nvme_rdma nvme_loop
```
* An NVMe device. (Typically via PCI-E through an standard slot or [M.2][m-dot-2] port)
* For Asymmetric Namespace Access (ANA) support (early preview), the following kernel build configuration enabled:
+ `CONFIG_NVME_MULTIPATH`: enables support for multipath access to NVMe subsystems

This is usually already enabled in distributions kernels, at least for RHEL/CentOS 8.2, Ubuntu 20.04 LTS, and SUSE Linux Enterprise 15.2.

On some distributions such as RHEL 8, the feature must be enabled manually:

```sh
# /etc/modprobe.d/nvme-multipath
options nvme_core multipath=1
```

followed by reloading the `nvme-core` module or rebooting.

To build this on NixOS:

```nix
# /etc/nixos/configuration.nix
boot.kernelPackages = pkgs.linuxPackages;
boot.kernelPatches = [ {
name = "nvme-multipath";
patch = null;
extraConfig = ''
NVME_MULTIPATH y
'';
} ];
```

followed by:

```sh
sudo nixos-rebuild boot
```
* An NVMe device. (Typically via PCI-E through a standard slot or [M.2][m-dot-2] port)
* A version of [`nix`][nix-install] configured as in the [build guide.][doc-build]

## Running binaries directly
Expand Down Expand Up @@ -158,7 +191,7 @@ Why these parameters?
- `--privileged` to allow controlling memory policies.

> **TODO:** We can use [control groups][control-groups] for this!
- `-v /dev:/dev:rw` is needed to get access to any raw device you might want to consume as local
- `-v /dev:/dev:rw` is needed to get access to any raw device you might want to consume as local
storage and huge pages
- `-v /dev/shm:/dev/shm:rw` is needed as for a circular buffer that can trace any IO operations
as they happen
Expand All @@ -184,7 +217,7 @@ nixpkgs.overlays = [
];
systemd.services.mayastor = {
wantedBy = [ "multi-user.target" ];
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
description = "A cloud native declarative data plane.";
serviceConfig = {
Expand Down Expand Up @@ -273,4 +306,3 @@ production Mayastor deployment and operation instructions.
[lxd]: https://linuxcontainers.org/
[libvirtd]: https://libvirt.org/index.html
[terraform-readme]: ./terraform/readme.adoc
[aarch64-branch]:

0 comments on commit 1a924d3

Please sign in to comment.