Expand top-level storage parameter

[vuldin]

# Persistence
storage:
  enabled: true
  # Absolute path on host to store Redpanda's data.
  # If not specified, then `emptyDir` will be used instead.
  # If specified, but `persistentVolume.enabled` is `true`, then has no effect.
  hostPath: ""
  # If `enabled` is `true` then a PersistentVolumeClaim will be created and
  # used to store Redpanda's data, otherwise `hostPath` is used.
  #
  persistentVolume:
    enabled: true
    size: 100Gi
    # If defined, then `storageClassName: <storageClass>`.
    # If set to "-", then `storageClassName: ""`, which disables dynamic
    # provisioning.
    # If undefined or empty (default), then no `storageClassName` spec is set,
    # so the default provisioner will be chosen (gp2 on AWS, standard on
    # GKE, AWS & OpenStack).
    storageClass: ""
    # Additional labels to apply to the created PersistentVolumeClaims.
    labels: {}
    # Additional annotations to apply to the created PersistentVolumeClaims.
    annotations: {}
  #
  # Shadow indexing
  cloud:
    enabled: true                             # enable cloud storage
    region: "local"                           # S3 region
    bucket: "redpanda"                        # S3 bucket
    api:
      address: "minio"                        # S3 API endpoint (will be generated for AWS if left blank)
      port: 9000                              # S3 API port (default 443)
    accessKey: "minio"                        # S3 access key
    secretKey: "minio123"                     # S3 secret key
    remoteReadEnabled: true                   # cluster-wide cloud storage read
    remoteWriteEnabled: true                  # cluster-wide cloud storage write
    maxUploadWaitIntervalSec: 30              # seconds between segment uploads (else segments uploaded when size == segment.bytes)
    tlsEnabled: true

• modify further to conform to https://github.com/bitnami/charts/blob/master/bitnami/kafka/values.yaml#L888
• make use of storage.enabled (potentially remove storage.persistentVolume.enabled in the process)
• lower default value of storage.persistentVolume.size to a more reasonable size (10 - 50 Mi)

EDIT:

This has become a much more focused issue.

The bitnami example is good, and we already have a top-level storage section which is the main benefit. We can open another ticket in the future to investigate adding new functionality to this section as needed, and when focusing on those tickets we can see if and how the bitnami chart handles those features.

We had an internal discussion around a storage.enabled parameter and decided to remove it, as we don't see a reason someone would want to disable persistence at the moment. We can revisit this in the future if it makes sense.

Lowering the default persistent volume size is still a great idea and will be included.

The last issue that wasn't initially spelled out but is being focused on is making sure the proper ownership of the persistent volume of type HostPath is set. This happens automatically in some environments, but in minikube it doesn't (see this issue). Resolving this ticket will mean setting storage.persistentVolume.enabled to true or false works in any environment.

The last change alluded to above is including shadow indexing features, and that already has another ticket and will be completed when that ticket is focused on (likely not part of the initial release of v2).