s3 repository canned_acl is not working for some files during snapshot creation

[point911]

Elasticsearch version (bin/elasticsearch --version): 6.3.2

Plugins installed: ["repository-s3"]

JVM version (java -version): javac 1.8.0_141

OS version (uname -a if on a Unix-like system): 4.14.51-60.38.amzn1.x86_64

Description of the problem including expected versus actual behavior:

We have an ES cluster in AWS. When we want to take a snapshot we are doing into S3 bucket which is located in another account.
when we creating an S3 repository in elastic search we enabling following option:

"canned_acl": "bucket-owner-full-control"

Not all files from snapshot have ACL for full control for bucket owner.
Files which DO NOT HAVE proper permissions.
1 . All files which have index-0 in their's name
2. index.latest
3. incompatible-snapshots.

All other files have correct ACL both for snapshot creator account and bucket owner account.

Steps to reproduce:

1. Setup ES cluster in AWS account 1.
2. Setup S3 bucket in AWS account 2.
3. Grant permissions to ES to save snapshots to S3 bucket in AWS account 2.
4. Setup a repository in ES cluster with option "canned_acl": "bucket-owner-full-control"
5. Create a snapshot.
6. Check file permissions for files with names index-0, index.latest and incompatible-snapshots.
   They will not have bucket owner FULL ADMIN permissions
   aws s3api get-object-acl --bucket BUCKET_NAME --key PATH_TO_S3_OBJECT

Provide logs (if relevant):

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

@tlrx welcome back. Do you mind taking a look?

[bnm22]

+1

I am having the exact same issue.

Version 6.3.0

[bnumber1]

+1

[bnm22]

Do we have any workaround ideas for this issue?

[point911]

Any chance to look into this? @bleskes @tlrx

[maczes]

Same issue. ES:
"version" : {
"number" : "6.2.3",
"build_hash" : "0fd46e9",
"build_date" : "2018-07-16T10:43:54.041989Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
...
}

[original-brownbear]

This isn't an issue anymore. in older versions prior to #31100 we were not setting the canned acl when doing a move so the blobs that were uploaded via atomic writes (index-N and the like as mentioned in the issue description) wouldn't get the canned ACL.
-> closing