Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Opensearch repository-s3 plugin cannot read ServiceAccount token #6312

Closed
DimitrisChristinakis opened this issue Feb 14, 2023 · 14 comments · Fixed by #6390
Closed

[BUG] Opensearch repository-s3 plugin cannot read ServiceAccount token #6312

DimitrisChristinakis opened this issue Feb 14, 2023 · 14 comments · Fixed by #6390
Labels

Comments

@DimitrisChristinakis
Copy link

Describe the bug
I am running an Opensearch cluster using Opensearch Operator in AWS EKS. I have created a ServiceAccount in order to access an AWS S3 bucket to use for my snapshot but i get the following error:

"reason" : "amazon_client_exception: java.security.AccessControlException: access denied ("java.io.FilePermission" "/var/run/secrets/eks.amazonaws.com/serviceaccount/token" "read")",
"caused_by" : {
"type" : "security_exception",
"reason" : "access denied ("java.io.FilePermission" "/var/run/secrets/eks.amazonaws.com/serviceaccount/token" "read")"

The token has the following permissions:
lrwxrwxrwx 1 root root 12 Feb 13 18:21 /var/run/secrets/eks.amazonaws.com/serviceaccount/token -> ..data/token

If i copy this token under the plugin's directory it works normally:
/usr/share/opensearch/plugins/repository-s3/token

To Reproduce
Steps to reproduce the behavior:

  1. Deploy an Opensearch cluster (image version 2.5.0) with repository-s3 plugin installed
  2. Configure a Kubernetes service account to assume an IAM role
  3. Configure s3.client.default.identity_token_file: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Expected behavior
The plugin should be able to read the token

Plugins
repository-s3 plugin

Host/Environment (please complete the following information):

  • AWS EKS version 1.21
  • Opensearch cluster (image version 2.5.0) deployed with Opensearch Operator (version 2.2.0)
@DimitrisChristinakis DimitrisChristinakis added bug Something isn't working untriaged labels Feb 14, 2023
@annasunny
Copy link

I'm having the same issue.

@reta
Copy link
Collaborator

reta commented Feb 15, 2023

@DimitrisChristinakis @annasunny this is covered in documentation [1], the SecurityManager policy settings prevent s3-repository to access arbitrary locations on file system.

[1] https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/#amazon-s3

@annasunny
Copy link

@reta Thank you Andriy! Could you help explain more on this: Where is the" SecurityManager policy settings", and how I can change it?

@reta
Copy link
Collaborator

reta commented Feb 15, 2023

Sure, the policy is located here /usr/share/opensearch/plugins/repository-s3/plugin-security.policy, the basic permissions are here [1], I think if you add the following line to the grant section, it should work just fine:

grant {
    ...
    permission java.io.FilePermission "/var/run/secrets/eks.amazonaws.com/serviceaccount/token", "read";
};

Please note that service restart is needed.

[1] https://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html

@annasunny
Copy link

Thank you @reta! However, I'm not seeing the permission java.io.FilePermission from my plugin-security.policy file...
Here is my plugin-security.policy file looks like, and I'm using cat command to open it:
Screen Shot 2023-02-15 at 3 58 31 PM

Instead, I'm running into another [<my-s3-repository-name>] path [<my-s3-path>] is not accessible on cluster-manager node error. Could you take another look? - #6332 Thank you in advance!

@reta
Copy link
Collaborator

reta commented Feb 16, 2023

Here is my plugin-security.policy file looks like, and I'm using cat command to open it:

That's correct, as I mentioned, you may need to add this line manually:

permission java.io.FilePermission "/var/run/secrets/eks.amazonaws.com/serviceaccount/token", "read";

@herrbpl
Copy link

herrbpl commented Feb 20, 2023

@reta This is still a problem because:

  1. You cannot have files in path s3 plugin will be installed or installation will fail, which means you cannot really mount EKS service account token into accepted path. And you cannot preinstall plugin as it is installed on pod start.
  2. You cannot change policy config file because of same reason as 1.
    This is a problem or if there is working solution, could it please documented fully, how to make it work?

My recommendation is to add EKS default token and Kubernetes default service account token mount path to allowed files in policy, after all its AWS related product. Or allow custom step between installing plugin and starting opensearch.

@reta
Copy link
Collaborator

reta commented Feb 20, 2023

@herrbpl we do have a viable approach using soft links [1], I will update the docs and plugin (if necessary), https://www.elastic.co/guide/en/elasticsearch/reference/8.6/repository-s3.html#iam-kubernetes-service-accounts

My recommendation is to add EKS default token and Kubernetes default service account token mount path to allowed files in policy, after all its AWS related product.

The OS plugins (at least core ones) should not have access to anything outside conf or its local installation folder.

@herrbpl
Copy link

herrbpl commented Feb 20, 2023

but how to solve issue that you cannot install plugin if target folder is not empty?

@reta
Copy link
Collaborator

reta commented Feb 20, 2023

The conf folder is outside of the plugin target folder (it is controlled by OPENSEARCH_PATH_CONF env variable)

@annasunny
Copy link

annasunny commented Feb 20, 2023

Hi @reta Andriy , After I link those two:
ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "${OPENSEARCH_PATH_CONFIG}/aws-web-identity-token-file"
And refer it in opensearch.yml:
s3.client.default.identity_token_file: aws-web-identity-token-file
But I/m still seeing the same error -
Caused by: org.opensearch.common.io.stream.NotSerializableExceptionWrapper: amazon_client_exception: java.security.AccessControlException: access denied ("java.io.FilePermission" "/var/run/secrets/eks.amazonaws.com/serviceaccount/token" "read")

Looks like the symlink in Elastic community won't fix for Opensearch.
Please let me know if anyone else are able to use symlink to make it working.

(symlink: https://www.elastic.co/guide/en/elasticsearch/reference/master/repository-s3.html#iam-kubernetes-service-accounts

@reta
Copy link
Collaborator

reta commented Feb 20, 2023

Hey @annasunny ,

You need a full path here (afaik the OPENSEARCH_PATH_CONFIG won't be expanded in opensearch.yml) at the moment:

 s3.client.default.identity_token_file: /full/path/to/aws-web-identity-token-file

I have submitted the fix to make the exact case you are trying to wok just fine (#6390), but it is not yet merged. With it:

 s3.client.default.identity_token_file: aws-web-identity-token-file

would work out of the box.

@DimitrisChristinakis
Copy link
Author

Hi @reta , I am currently using Opensearch Operator to set up the cluster and I can verify that symlink works after adding it manually to every pod. Unfortunately I cannot find a way to automatically create the symlink or modify plugin-security.policy in order for this to work without modifying images, etc. I don't know if you have any ideas for this?

@reta
Copy link
Collaborator

reta commented Feb 21, 2023

Hi @reta , I am currently using Opensearch Operator to set up the cluster and I can verify that symlink works after adding it manually to every pod. Unfortunately I cannot find a way to automatically create the symlink or modify plugin-security.policy in order for this to work without modifying images, etc. I don't know if you have any ideas for this?

Thanks for confirming @DimitrisChristinakis , the modification of the plugin-security.policy should not be needed once we get #6390 in, but I sadly don't know how to instruct the Opensearch Operator to create the symlinks, you may ask the question there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants