Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix docs and references on mayastore storage #9068

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+28 −4
Open

Fix docs and references on mayastore storage #9068

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions website/content/v1.7/kubernetes-guides/configuration/storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ To see a live demo of this section, see the video below:
#### Prep Nodes

Either during initial cluster creation or on running worker nodes, several machine config values should be edited.
(This information is gathered from the Mayastor [documentation](https://mayastor.gitbook.io/introduction/quickstart/preparing-the-cluster).)
(This information is gathered from the Mayastor [documentation](https://openebs.io/docs/Solutioning/openebs-on-kubernetes-platforms/talos).)
We need to set the `vm.nr_hugepages` sysctl and add `openebs.io/engine=mayastor` labels to the nodes which are meant to be storage nodes.
This can be done with `talosctl patch machineconfig` or via config patches during `talosctl gen config`.

Expand All @@ -96,6 +96,16 @@ First create a config patch file named `mayastor-patch.yaml` with the following
path: /machine/nodeLabels
value:
openebs.io/engine: mayastor
- op: add
path: /machine/kubelet/extraMounts
value:
- destination: /var/local
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the mount path difference /var/local vs. /var/lib/local intentional?

Copy link
Author

@blmhemu blmhemu Jul 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Openebs suggests /var/local - https://openebs.io/docs/Solutioning/openebs-on-kubernetes-platforms/talos

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that mounting under different paths is an unusual thing, as usually you want kubelet to see same path structure as the host, otherwise some features won't work

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this one of the templates

---
{{ if and .Values.etcd.persistence.enabled (eq .Values.etcd.persistence.storageClass "manual") }}
{{- range $index, $end := until (.Values.etcd.replicaCount | int) }}
apiVersion: v1
kind: PersistentVolume
metadata:
  name: etcd-volume-{{ $index }}
  labels:
    statefulset.kubernetes.io/pod-name: {{ print $.Release.Name }}-etcd-{{ $index }}
spec:
  storageClassName: manual
  # You must also delete the hostpath on the node
  persistentVolumeReclaimPolicy: {{ $.Values.etcd.persistence.reclaimPolicy }}
  capacity:
    storage: {{ $.Values.etcd.persistence.size | quote }}
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/var/local/{{ $.Release.Name }}/etcd/pod-{{ $index }}"
---
{{- end }}
{{- end }}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All instances (of /var/local) use /var/local/{{ $.Release.Name }} prefix - so i guess we can safely make it /var/local/openebs ?

type: bind
source: /var/lib/local
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While mayastore recommends /var/lib/local i think /var/lib/openebs might be better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should match I guess something in the installed helm chart?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am suggesting for the source. Lets leave destination as it is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's something broken here.

I guess there's somewhere hostPath volume mount for /var/lib/local. So on the host path is /var/lib/local. If the path on the kubelet mount is different (like in this PR), this won't work well, as for the kubelet /var/lib/local would not exist.

options:
- bind
- rshared
- rw
```

Using gen config
Expand All @@ -118,7 +128,11 @@ talosctl -n <node ip> service kubelet restart

#### Deploy Mayastor

Continue setting up [Mayastor](https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor) using the official documentation.
Continue setting up [Mayastor](https://openebs.io/docs/quickstart-guide/installation) using the official documentation.

> Note: The Mayastor helm chart uses an init container that checks for the `nvme_tcp` module.
> It does not mount `/sys` and will not be able to find it.
> Easiest solution is to disable the init container.

### Piraeus / LINSTOR

Expand Down
14 changes: 12 additions & 2 deletions website/content/v1.8/kubernetes-guides/configuration/storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ To see a live demo of this section, see the video below:
#### Prep Nodes

Either during initial cluster creation or on running worker nodes, several machine config values should be edited.
(This information is gathered from the Mayastor [documentation](https://mayastor.gitbook.io/introduction/quickstart/preparing-the-cluster).)
(This information is gathered from the Mayastor [documentation](https://openebs.io/docs/Solutioning/openebs-on-kubernetes-platforms/talos).)
We need to set the `vm.nr_hugepages` sysctl and add `openebs.io/engine=mayastor` labels to the nodes which are meant to be storage nodes.
This can be done with `talosctl patch machineconfig` or via config patches during `talosctl gen config`.

Expand All @@ -96,6 +96,16 @@ First create a config patch file named `mayastor-patch.yaml` with the following
path: /machine/nodeLabels
value:
openebs.io/engine: mayastor
- op: add
path: /machine/kubelet/extraMounts
value:
- destination: /var/local
type: bind
source: /var/lib/local
options:
- bind
- rshared
- rw
```

Using gen config
Expand All @@ -118,7 +128,7 @@ talosctl -n <node ip> service kubelet restart

#### Deploy Mayastor

Continue setting up [Mayastor](https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor) using the official documentation.
Continue setting up [Mayastor](https://openebs.io/docs/quickstart-guide/installation) using the official documentation.

> Note: The Mayastor helm chart uses an init container that checks for the `nvme_tcp` module.
> It does not mount `/sys` and will not be able to find it.
Expand Down