The Amazon Elastic File System Container Storage Interface (CSI) Driver implements the CSI specification for container orchestrators to manage the lifecycle of Amazon EFS file systems.
| Amazon EFS CSI Driver \ CSI Spec Version | v0.3.0 | v1.1.0 | v1.2.0 |
|---|---|---|---|
| master branch | no | no | yes |
| v2.x.x | no | no | yes |
| v1.x.x | no | no | yes |
| v0.3.0 | no | yes | no |
| v0.2.0 | no | yes | no |
| v0.1.0 | yes | no | no |
Amazon EFS CSI driver supports dynamic provisioning and static provisioning. Currently, Dynamic Provisioning creates an access point for each PV. This mean an Amazon EFS file system has to be created manually on AWS first and should be provided as an input to the storage class parameter. For static provisioning, the Amazon EFS file system needs to be created manually on AWS first. After that, it can be mounted inside a container as a volume using the driver.
The following CSI interfaces are implemented:
- Controller Service: CreateVolume, DeleteVolume, ControllerGetCapabilities, ValidateVolumeCapabilities
- Node Service: NodePublishVolume, NodeUnpublishVolume, NodeGetCapabilities, NodeGetInfo, NodeGetId, NodeGetVolumeStats
- Identity Service: GetPluginInfo, GetPluginCapabilities, Probe
| Parameters | Values | Default | Optional | Description |
|---|---|---|---|---|
| provisioningMode | efs-ap | false | Type of volume provisioned by efs. Currently, Access Points are supported. | |
| fileSystemId | false | File System under which access points are created. | ||
| directoryPerms | false | Directory permissions for Access Point root directory creation. | ||
| uid | true | POSIX user Id to be applied for Access Point root directory creation. | ||
| gid | true | POSIX group Id to be applied for Access Point root directory creation. | ||
| gidRangeStart | 50000 | true | Start range of the POSIX group Id to be applied for Access Point root directory creation. Not used if uid/gid is set. | |
| gidRangeEnd | 7000000 | true | End range of the POSIX group Id. Not used if uid/gid is set. | |
| basePath | true | Path under which access points for dynamic provisioning is created. If this parameter is not specified, access points are created under the root directory of the file system | ||
| subPathPattern | /${.PV.name} |
true | The template used to construct the subPath under which each of the access points created under Dynamic Provisioning. Can be made up of fixed strings and limited variables, is akin to the 'subPathPattern' variable on the nfs-subdir-external-provisioner chart. Supports .PVC.name,.PVC.namespace and .PV.name |
|
| ensureUniqueDirectory | true | true | NOTE: Only set this to false if you're sure this is the behaviour you want. Used when dynamic provisioning is enabled, if set to true, appends the a UID to the pattern specified in subPathPattern to ensure that access points will not accidentally point at the same directory. |
|
| az | "" | true | Used for cross-account mount. az under storage class parameter is optional. If specified, mount target associated with the az will be used for cross-account mount. If not specified, a random mount target will be picked for cross account mount |
|
| reuseAccessPoint | false | true | When set to true, it creates the Access Point client-token from the provided PVC name. So that the AccessPoint can be replicated from a different cluster if same PVC name and storageclass configuration are used. |
Note
- Custom Posix group Id range for Access Point root directory must include both
gidRangeStartandgidRangeEndparameters. These parameters are optional only if both are omitted. If you specify one, the other becomes mandatory. - When using a custom Posix group ID range, there is a possibility for the driver to run out of available POSIX group Ids. We suggest ensuring custom group ID range is large enough or create a new storage class with a new file system to provision additional volumes.
azunder storage class parameter is not be confused with efs-utils mount optionaz. Theazmount option is used for cross-az mount or efs one zone file system mount within the same aws account as the cluster.- Using dynamic provisioning, user identity enforcement is always applied.
- When user enforcement is enabled, Amazon EFS replaces the NFS client's user and group IDs with the identity configured on the access point for all file system operations.
- The uid/gid configured on the access point is either the uid/gid specified in the storage class, a value in the gidRangeStart-gidRangeEnd (used as both uid/gid) specified in the storage class, or is a value selected by the driver is no uid/gid or gidRange is specified.
- We suggest using static provisioning if you do not wish to use user identity enforcement.
If you want to pass any other mountOptions to Amazon EFS CSI driver while mounting, they can be passed in through the Persistent Volume or the Storage Class objects, depending on whether static or dynamic provisioning is used. The following are examples of some mountOptions that can be passed:
- lookupcache: Specifies how the kernel manages its cache of directory entries for a given mount point. Mode can be one of all, none, pos, or positive. Each mode has different functions and for more information you can refer to this link.
- iam: Use the CSI Node Pod's IAM identity to authenticate with Amazon EFS.
When using the EFS CSI driver, be aware that the noresvport mount option is enabled by default. This means the client can use any available source port for communication, not just the reserved ports.
One of the advantages of using Amazon EFS is that it provides encryption in transit support using TLS. Using encryption in transit, data will be encrypted during its transition over the network to the Amazon EFS service. This provides an extra layer of defence-in-depth for applications that requires strict security compliance.
Encryption in transit is enabled by default in the master branch version of the driver. To disable it and mount volumes using plain NFSv4, set the volumeAttributes field encryptInTransit to "false" in your persistent volume manifest. For an example manifest, see the encryption in transit example.
Note
Kubernetes version 1.13 or later is required if you are using this feature in Kubernetes.
The following sections are Kubernetes specific. If you are a Kubernetes user, use this for driver features, installation steps, and examples.
| Amazon EFS CSI Driver \ Kubernetes Version | maturity | v1.11 | v1.12 | v1.13 | v1.14 | v1.15 | v1.16 | v1.17+ |
|---|---|---|---|---|---|---|---|---|
| master branch | GA | no | no | no | no | no | no | yes |
| v2.1.x | GA | no | no | no | no | no | no | yes |
| v2.0.x | GA | no | no | no | no | no | no | yes |
| v1.7.x | GA | no | no | no | no | no | no | yes |
| v1.6.x | GA | no | no | no | no | no | no | yes |
| v1.5.x | GA | no | no | no | no | no | no | yes |
| v1.4.x | GA | no | no | no | no | no | no | yes |
| v1.3.x | GA | no | no | no | no | no | no | yes |
| v1.2.x | GA | no | no | no | no | no | no | yes |
| v1.1.x | GA | no | no | no | yes | yes | yes | yes |
| v1.0.x | GA | no | no | no | yes | yes | yes | yes |
| v0.3.0 | beta | no | no | no | yes | yes | yes | yes |
| v0.2.0 | beta | no | no | no | yes | yes | yes | yes |
| v0.1.0 | alpha | yes | yes | yes | no | no | no | no |
| Amazon EFS CSI Driver Version | Image |
|---|---|
| master branch | amazon/aws-efs-csi-driver:master |
| v2.1.0 | amazon/aws-efs-csi-driver:v2.1.0 |
| v2.0.9 | amazon/aws-efs-csi-driver:v2.0.9 |
| v2.0.8 | amazon/aws-efs-csi-driver:v2.0.8 |
| v2.0.7 | amazon/aws-efs-csi-driver:v2.0.7 |
| v2.0.6 | amazon/aws-efs-csi-driver:v2.0.6 |
| v2.0.5 | amazon/aws-efs-csi-driver:v2.0.5 |
| v2.0.4 | amazon/aws-efs-csi-driver:v2.0.4 |
| v2.0.3 | amazon/aws-efs-csi-driver:v2.0.3 |
| v2.0.2 | amazon/aws-efs-csi-driver:v2.0.2 |
| v2.0.1 | amazon/aws-efs-csi-driver:v2.0.1 |
| v2.0.0 | amazon/aws-efs-csi-driver:v2.0.0 |
| v1.7.7 | amazon/aws-efs-csi-driver:v1.7.7 |
| v1.7.6 | amazon/aws-efs-csi-driver:v1.7.6 |
| v1.7.5 | amazon/aws-efs-csi-driver:v1.7.5 |
| v1.7.4 | amazon/aws-efs-csi-driver:v1.7.4 |
| v1.7.3 | amazon/aws-efs-csi-driver:v1.7.3 |
| v1.7.2 | amazon/aws-efs-csi-driver:v1.7.2 |
| v1.7.1 | amazon/aws-efs-csi-driver:v1.7.1 |
| v1.7.0 | amazon/aws-efs-csi-driver:v1.7.0 |
| v1.6.0 | amazon/aws-efs-csi-driver:v1.6.0 |
| v1.5.9 | amazon/aws-efs-csi-driver:v1.5.9 |
| v1.5.8 | amazon/aws-efs-csi-driver:v1.5.8 |
| v1.5.7 | amazon/aws-efs-csi-driver:v1.5.7 |
| v1.5.6 | amazon/aws-efs-csi-driver:v1.5.6 |
| v1.5.5 | amazon/aws-efs-csi-driver:v1.5.5 |
| v1.5.4 | amazon/aws-efs-csi-driver:v1.5.4 |
| v1.5.3 | amazon/aws-efs-csi-driver:v1.5.3 |
| v1.5.2 | amazon/aws-efs-csi-driver:v1.5.2 |
| v1.5.1 | amazon/aws-efs-csi-driver:v1.5.1 |
| v1.5.0 | amazon/aws-efs-csi-driver:v1.5.0 |
| v1.4.9 | amazon/aws-efs-csi-driver:v1.4.9 |
| v1.4.8 | amazon/aws-efs-csi-driver:v1.4.8 |
| v1.4.7 | amazon/aws-efs-csi-driver:v1.4.7 |
| v1.4.6 | amazon/aws-efs-csi-driver:v1.4.6 |
| v1.4.5 | amazon/aws-efs-csi-driver:v1.4.5 |
| v1.4.4 | amazon/aws-efs-csi-driver:v1.4.4 |
| v1.4.3 | amazon/aws-efs-csi-driver:v1.4.3 |
| v1.4.2 | amazon/aws-efs-csi-driver:v1.4.2 |
| v1.4.1 | amazon/aws-efs-csi-driver:v1.4.1 |
| v1.4.0 | amazon/aws-efs-csi-driver:v1.4.0 |
| v1.3.8 | amazon/aws-efs-csi-driver:v1.3.8 |
| v1.3.7 | amazon/aws-efs-csi-driver:v1.3.7 |
| v1.3.6 | amazon/aws-efs-csi-driver:v1.3.6 |
| v1.3.5 | amazon/aws-efs-csi-driver:v1.3.5 |
| v1.3.4 | amazon/aws-efs-csi-driver:v1.3.4 |
| v1.3.3 | amazon/aws-efs-csi-driver:v1.3.3 |
| v1.3.2 | amazon/aws-efs-csi-driver:v1.3.2 |
| v1.3.1 | amazon/aws-efs-csi-driver:v1.3.1 |
| v1.3.0 | amazon/aws-efs-csi-driver:v1.3.0 |
| v1.2.1 | amazon/aws-efs-csi-driver:v1.2.1 |
| v1.2.0 | amazon/aws-efs-csi-driver:v1.2.0 |
| v1.1.1 | amazon/aws-efs-csi-driver:v1.1.1 |
| v1.1.0 | amazon/aws-efs-csi-driver:v1.1.0 |
| v1.0.0 | amazon/aws-efs-csi-driver:v1.0.0 |
| v0.3.0 | amazon/aws-efs-csi-driver:v0.3.0 |
| v0.2.0 | amazon/aws-efs-csi-driver:v0.2.0 |
| v0.1.0 | amazon/aws-efs-csi-driver:v0.1.0 |
| Driver Version | ECR Image |
|---|---|
| v2.1.0 | public.ecr.aws/efs-csi-driver/amazon/aws-efs-csi-driver:v2.1.0 |
Note
You can find previous efs-csi-driver versions' images from here
- Static provisioning - Amazon EFS file system needs to be created manually first, then it could be mounted inside container as a persistent volume (PV) using the driver.
- Dynamic provisioning - Uses a persistent volume claim (PVC) to dynamically provision a persistent volume (PV). On Creating a PVC, kuberenetes requests Amazon EFS to create an Access Point in a file system which will be used to mount the PV.
- Mount Options - Mount options can be specified in the persistent volume (PV) or storage class for dynamic provisioning to define how the volume should be mounted.
- Encryption of data in transit - Amazon EFS file systems are mounted with encryption in transit enabled by default in the master branch version of the driver.
- Cross account mount - Amazon EFS file systems from different aws accounts can be mounted from an Amazon EKS cluster.
- Multiarch - Amazon EFS CSI driver image is now multiarch on ECR
Note
Since Amazon EFS is an elastic file system, it doesn't really enforce any file system capacity. The actual storage capacity value in persistent volume and persistent volume claim is not used when creating the file system. However, since the storage capacity is a required field by Kubernetes, you must specify the value and you can use any valid value for the capacity.
Considerations
- The Amazon EFS CSI Driver isn't compatible with Windows-based container images.
- You can't use dynamic persistent volume provisioning with Fargate nodes, but you can use static provisioning.
- Dynamic provisioning requires
1.2or later of the driver. You can statically provision persistent volumes using version1.1of the driver on any supported Amazon EKS cluster version. - Version
1.3.2or later of this driver supports the Arm64 architecture, including Amazon EC2 Graviton-based instances. - Version
1.4.2or later of this driver supports using FIPS for mounting file systems. For more information on how to enable FIPS, see Helm. - Take note of the resource quotas for Amazon EFS. For example, there's a quota of 1000 access points that can be created for each Amazon EFS file system. For more information, see https://docs.aws.amazon.com/efs/latest/ug/limits.html#limits-efs-resources-per-account-per-region.
There are potential race conditions on node startup (especially when a node is first joining the cluster) where pods/processes that rely on the EFS CSI Driver can act on a node before the EFS CSI Driver is able to startup up and become fully ready. To combat this, the EFS CSI Driver contains a feature to automatically remove a taint from the node on startup. This feature was introduced from version v1.7.2 of the EFS CSI Driver and version v2.5.2 of its Helm chart. Users can taint their nodes when they join the cluster and/or on startup, to prevent other pods from running and/or being scheduled on the node prior to the EFS CSI Driver becoming ready.
This feature is activated by default, and cluster administrators should use the taint efs.csi.aws.com/agent-not-ready:NoExecute (any effect will work, but NoExecute is recommended). For example, EKS Managed Node Groups support automatically tainting nodes.
Prerequisites
- An existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see Creating an IAM OIDC provider for your cluster.
- The AWS CLI installed and configured on your device or AWS CloudShell. To install the latest version, see Installing, updating, and uninstalling the AWS CLI and Quick configuration with
aws configurein the AWS Command Line Interface User Guide. The AWS CLI version installed in the AWS CloudShell may also be several versions behind the latest version. To update it, see Installing AWS CLI to your home directory in the AWS CloudShell User Guide. - The
kubectlcommand line tool is installed on your device or AWS CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. To install or upgradekubectl, see Installing or updatingkubectl.
Note
A Pod running on AWS Fargate automatically mounts an Amazon EFS file system, without needing the manual driver installation steps described on this page.
The driver requires IAM permission to talk to Amazon EFS to manage the volume on user's behalf. There are several methods to grant driver IAM permission:
- Using the EKS Pod Identity Add-on - Install the EKS Pod Identity add-on to your EKS cluster. This doesn't need the efs-csi-driver to be installed through EKS add-on, it can be used no matter the method of installation of the efs-csi-driver. If this installation method is used, the
AmazonEFSCSIDriverPolicypolicy has to be added to the cluster's node group's IAM role. - Using IAM role for service account – Create an IAM Role for service accounts with the required permissions in iam-policy-example.json. Uncomment annotations and put the IAM role ARN in the service-account manifest. For example steps, see Create an IAM policy and role for Amazon EKS.
- Using IAM instance profile – Grant all the worker nodes with required permissions by attaching the policy to the instance profile of the worker.
There are several options for deploying the driver. The following are some examples.
This procedure requires Helm V3 or later. To install or upgrade Helm, see Using Helm with Amazon EKS.
To install the driver using Helm
-
Add the Helm repo.
helm repo add aws-efs-csi-driver https://kubernetes-sigs.github.io/aws-efs-csi-driver/
-
Update the repo.
helm repo update aws-efs-csi-driver
-
Install a release of the driver using the Helm chart.
helm upgrade --install aws-efs-csi-driver --namespace kube-system aws-efs-csi-driver/aws-efs-csi-driver
To specify an image repository, add the following argument. Replace the repository address with the cluster's container image address.
--set image.repository=602401143452.dkr.ecr.region-code.amazonaws.com/eks/aws-efs-csi-driver
If you already created a service account by following Create an IAM policy and role for Amazon EKS, then add the following arguments.
--set controller.serviceAccount.create=false \ --set controller.serviceAccount.name=efs-csi-controller-sa
If you don't have outbound access to the Internet, add the following arguments.
--set sidecars.livenessProbe.image.repository=602401143452.dkr.ecr.region-code.amazonaws.com/eks/livenessprobe \ --set sidecars.node-driver-registrar.image.repository=602401143452.dkr.ecr.region-code.amazonaws.com/eks/csi-node-driver-registrar \ --set sidecars.csiProvisioner.image.repository=602401143452.dkr.ecr.region-code.amazonaws.com/eks/csi-provisioner
To force the Amazon EFS CSI driver to use FIPS for mounting the file system, add the following argument.
--set useFips=true
Note
hostNetwork: true (should be added under spec/deployment on kubernetes installations where AWS metadata is not reachable from pod network. To fix the following error NoCredentialProviders: no valid providers in chain this parameter should be added.)
If you want to download the image with a manifest, we recommend first trying these steps to pull secured images from the private Amazon ECR registry.
To install the driver using images stored in the private Amazon ECR registry
-
Download the manifest. Replace
release-X.Xwith your desired branch. We recommend using the latest released version. For a list of active branches, see Branches.kubectl kustomize \ "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/ecr/?ref=release-2.X" > private-ecr-driver.yamlNote
If you encounter an issue that you aren't able to resolve by adding IAM permissions, try the Manifest (public registry) steps instead. -
In the following command, replace
region-codewith the AWS Region that your cluster is in. Then run the modified command to replaceus-west-2in the file with your AWS Region.sed -i.bak -e 's|us-west-2|region-code|' private-ecr-driver.yaml -
Replace
accountin the following command with the account from Amazon container image registries for the AWS Region that your cluster is in and then run the modified command to replace602401143452in the file.sed -i.bak -e 's|602401143452|account|' private-ecr-driver.yaml -
If you already created a service account by following Create an IAM policy and role for Amazon EKS, then edit the
private-ecr-driver.yamlfile. Remove the following lines that create a Kubernetes service account.apiVersion: v1 kind: ServiceAccount metadata: labels: app.kubernetes.io/name: aws-efs-csi-driver name: efs-csi-controller-sa namespace: kube-system --- -
Apply the manifest.
kubectl apply -f private-ecr-driver.yaml
For some situations, you may not be able to add the necessary IAM permissions to pull from the private Amazon ECR registry. One example of this scenario is if your IAM principal isn't allowed to authenticate with someone else's account. When this is true, you can use the public Amazon ECR registry.
To install the driver using images stored in the public Amazon ECR registry
-
Download the manifest. Replace
release-X.Xwith your desired branch. We recommend using the latest released version. For a list of active branches, see Branches.kubectl kustomize \ "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-2.X" > public-ecr-driver.yaml -
If you already created a service account by following Create an IAM policy and role, then edit the
public-ecr-driver.yamlfile. Remove the following lines that create a Kubernetes service account.apiVersion: v1 kind: ServiceAccount metadata: labels: app.kubernetes.io/name: aws-efs-csi-driver name: efs-csi-controller-sa namespace: kube-system --- -
Apply the manifest.
kubectl apply -f public-ecr-driver.yaml
After deploying the driver, you can continue to these sections:
| Parameters | Values | Default | Optional | Description |
|---|---|---|---|---|
| vol-metrics-opt-in | false | true | Opt in to emit volume metrics. | |
| vol-metrics-refresh-period | 240 | true | Refresh period for volume metrics in minutes. | |
| vol-metrics-fs-rate-limit | 5 | true | Volume metrics routines rate limiter per file system. |
Enabling the vol-metrics-opt-in parameter activates the gathering of inode and disk usage data. This functionality, particularly in scenarios with larger file systems, may result in an uptick in memory usage due to the detailed aggregation of file system information. We advise users with large-scale file systems to consider this aspect when utilizing this feature.
| Parameters | Values | Default | Optional | Description |
|---|---|---|---|---|
| delete-access-point-root-dir | false | true | Opt in to delete access point root directory by DeleteVolume. By default, DeleteVolume will delete the access point behind Persistent Volume and deleting access point will not delete the access point root directory or its contents. | |
| tags | true | Space separated key:value pairs which will be added as tags for Amazon EFS resources. For example, '--tags=name:efs-tag-test date:Jan24' |
If you want to update to latest released version:
kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-2.0"If you want to update to a specific version, first customize the driver yaml file locally:
kubectl kustomize "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-2.0" > driver.yamlThen, update all lines referencing image: amazon/aws-efs-csi-driver to the desired version (e.g., to image: amazon/aws-efs-csi-driver:v2.1.0) in the yaml file, and deploy driver yaml again:
kubectl apply -f driver.yamlBefore following the examples, you need to:
- Get yourself familiar with how to setup Kubernetes on AWS and how to create Amazon EFS file system.
- When creating an Amazon EFS file system, make sure it is accessible from the Kubernetes cluster. This can be achieved by creating the file system inside the same VPC as the Kubernetes cluster or using VPC peering.
- Install Amazon EFS CSI driver following the Installation steps.
- Static provisioning
- Dynamic provisioning
- Encryption in transit
- Accessing the file system from multiple pods
- Consume Amazon EFS in StatefulSets
- Mount subpath
- Use Access Points
- Amazon EFS CSI driver supports using botocore to retrieve mount target ip address when dns name cannot be resolved, e.g., when user is mounting a file system in another VPC, botocore comes preinstalled on efs-csi-driver which can solve this DNS issue.
- IAM policy prerequisites to use this feature :
Allowelasticfilesystem:DescribeMountTargetsandec2:DescribeAvailabilityZonesactions in your policy attached to the Amazon EKS service account role, refer to example policy here.
-
Please go through CSI Spec and Kubernetes CSI Developer Documentation to get some basic understanding of CSI driver before you start.
-
If you are about to update iam policy file, please also update efs policy in weaveworks/eksctl https://github.com/weaveworks/eksctl/blob/main/pkg/cfn/builder/statement.go */
- Golang 1.13.4+
Dependencies are managed through go module. To build the project, first turn on go mod using export GO111MODULE=on, to build the project run: make
To execute all unit tests, run: make test
To pull logs and troubleshoot the driver, see troubleshooting/README.md.
This library is licensed under the Apache 2.0 License.