Encryption in transit broken

[Almenon]

/kind bug

What happened?
The example in https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/examples/kubernetes/encryption_in_transit/README.md does not work. It errors out with:

  Warning  FailedMount  1s (x6 over 20s)  kubelet            MountVolume.SetUp failed for volume "efs-pv" : rpc error: code = Internal desc = Could not mount "fs-0d1ea2bb1aef55da4:/" at "/var/lib/kubelet/pods/fefe7de7-00fe-4487-ac81-765e55a538a6/volumes/kubernetes.io~csi/efs-pv/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o tls fs-0d1ea2bb1aef55da4:/ /var/lib/kubelet/pods/fefe7de7-00fe-4487-ac81-765e55a538a6/volumes/kubernetes.io~csi/efs-pv/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].

What you expected to happen?
The example to work.

How to reproduce it (as minimally and precisely as possible)?
Combine linked example with https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/patterns/stateful

Specifically, we used:

module "this_cluster_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.9.1"
  ...
  enable_aws_efs_csi_driver = true
  aws_efs_csi_driver = {
    name          = "aws-efs-csi-driver"
    repository    = "https://kubernetes-sigs.github.io/aws-efs-csi-driver/"
    chart_version = "2.4.8"
    namespace     = "kube-system"
  }
  ...
}

module "efs" {
  source  = "terraform-aws-modules/efs/aws"
  version = "~> 1.1"

  name = "rstudio-packagemanager"

  # Mount targets / security group
  mount_targets = {
    for k, v in zipmap(var.vpc_azs, var.vpc_private_subnets) : k => { subnet_id = v } # I verified same subnets used for EKS
  }
  security_group_description = "rstudio-packagemanager EFS security group"
  security_group_vpc_id      = var.vpc_id
  security_group_rules = {
    vpc = {
      # relying on the defaults provided for EFS/NFS (2049/TCP + ingress)
      description = "NFS ingress from VPC private subnets"
      cidr_blocks = split(",", var.vpc_cidr_block)
    }
  }

  lifecycle_policy = {
    transition_to_ia = "AFTER_60_DAYS"
  }

  tags = var.tags
}

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version): 1.25
• Driver version: 1.5.9
• Helm chart version: 2.4.8
• Relevant: Mount issues with module aws_efs_csi_driver aws-ia/terraform-aws-eks-blueprints#1197
• This might be a bug with EKS blueprints and not aws-efs-csi-driver
• Same issue with iam instead of tls in the mount options

Please also attach debug logs to help us better diagnose
results.tgz

• Instructions to gather debug logs can be found here

[Almenon]

Workaround: the change in aws-ia/terraform-aws-eks-blueprints#1753

[mskanth972]

Hi @Almenon, Static Provisioning with encryption enabled seems working fine on default settings. Can you try to open this issue at terraform-aws-eks-blueprints if you are facing this issue still.

[Almenon]

@mskanth972 Did you try it with a EFS policy disallowing insecure transport, like in https://github.com/terraform-aws-modules/terraform-aws-efs/blob/8cdc5b65d3b92f4211271621a567e4ce6b4dc469/main.tf#L82?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.