WIP: S3 client encryption

[gianrubio]

This PR is a replacement of #16843 that was merged and later rolled back.
Before I wrote the docs I'd like to receive feedbacks about the changes.

Related to #11128

How to test?

Create Index

curl -X PUT "localhost:9200/shakespeare" -H 'Content-Type: application/json' -d'
{
 "mappings": {
  "doc": {
   "properties": {
    "speaker": {"type": "keyword"},
    "play_name": {"type": "keyword"},
    "line_id": {"type": "integer"},
    "speech_number": {"type": "integer"}
   }
  }
 }
}
'

Load shakespeare example data

wget https://download.elastic.co/demos/kibana/gettingstarted/shakespeare_6.0.json
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/doc/_bulk?pretty' --data-binary @shakespeare_6.0.json

Create s3 repo

curl -XPUT 'http://localhost:9200/_snapshot/my_s3_repository' -H "Content-Type: application/json" -d '{
    "type": "s3",
    "settings": {
        "bucket": "rubio-test2",
        "client_side_encryption": "true",
        "client_public_key": "MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALb4IRgaDtmwsz+kC/bkKLno5bRWMDbRDztcM/N/VJIg+HQKA8ees6uznEOGe6cOZp+QHYrpTIXs36QB9vfsVdUCAwEAAQ==",
        "client_private_key":"MIIBUwIBADANBgkqhkiG9w0BAQEFAASCAT0wggE5AgEAAkEAtvghGBoO2bCzP6QL9uQouejltFYwNtEPO1wz839UkiD4dAoDx56zq7OcQ4Z7pw5mn5AdiulMhezfpAH29+xV1QIDAQABAkBEy/OVnmarD7e2XDZrdMqjbKDCOA4U7nKtvTODgQMJll0b7wA52bY/kG8gA8Q2aqiKVHDRl/EQ33bELr2A56NBAiEA4+7qC/TMw2V2q+FuW5Az8mzfCH6mNmyMLc/XwuZ2/V0CIQDNf9NRbZ0EZQCiq6iBjgYzkdJlv1tDt9erqVj0w0K62QIgZDE5IFhTSfDn4VYOtKEGtKG2yH0jgvjkBZ8/MKUt2OECIFSzDuJND560EqL5paZgZ2XyAIo3aOJsb9QtJKEdqe9hAiBgithEZGGQNV/pXweOtku/CezLvY2FJaSPShfeQfXjcg=="
    }
}'

Create snapshot

curl -XPUT "localhost:9200/_snapshot/my_s3_repository/test123?wait_for_completion=true" -H "Content-Type: application/json"

List snapshots

curl -XGET "localhost:9200/_snapshot/my_s3_repository/_all?pretty"

Delete index

curl -X DELETE "localhost:9200/shakespeare"

Restore snapshot

curl -XPOST "localhost:9200/_snapshot/my_s3_repository/test123/_restore?wait_for_completion=true"

[karmi]

Hi @gianrubio, we have found your signature in our records, but it seems like you have signed with a different e-mail than the one used in yout Git commit. Can you please add both of these e-mails into your Github profile (they can be hidden), so we can match your e-mails to your Github profile?

[gianrubio]

@karmi this was happening because I used an old commit from @xuzha and he didn't signed the CLA. I've pushed it again without his commits.

[elasticmachine]

Pinging @elastic/es-distributed

[tlrx]

Thanks @gianrubio for this contribution. Adding client side encryption is a feature that is regularly requested, and the AWS SDK Java library provides everything to do it.

I'm just concerned about few things:

• I'd like this feature to be added with an up to date AWS SDK version (Update AWS SDK to 1.11.340 in repository-s3 #30723)
• we're working on adding reloadable secured settings, and I think that the client keys here should be reloadable (Reloadable secure settings for plugins #29135)
• finally, I'm wondering what are the impact on client side encryption if it is activated/deactivated on an existing repository. We might want an encrypted snapshot index file and everything else encrypted, I don't know. I'd like to brainstorm a bit about this.

[gianrubio]

I'd like this feature to be added with an up to date AWS SDK version (#30723)

Ok, I can wait until it got merged

we're working on adding reloadable secured settings, and I think that the client keys here should be reloadable (#29135)

Ok, same as above

finally, I'm wondering what are the impact on client side encryption if it is activated/deactivated on an existing repository. We might want an encrypted snapshot index file and everything else encrypted, I don't know. I'd like to brainstorm a bit about this.

It'll probably break the state because the restore action will try to decrypt files that are not decrypted. Any ideas on how can we avoid users to activate/desactivate in a existing repository?

[ywelsch]

@gianrubio now that #30723 is merged, would you like to pick this PR up again? I think we should start by assuming that this setting is not dynamically enabled / disabled on a repo.

[albertzaharovits]

@gianrubio Are you still planning to work on this?
I would like to try get it done myself, if you don't mind.

[tomcallahan]

@albertzaharovits all yours