Add EC2 credential test for repository-s3 #31918

vladimirdolzhenko · 2018-07-10T11:29:27Z

Add EC2 credential test for repository-s3

Relates to #26913

elasticmachine · 2018-07-10T11:29:29Z

Pinging @elastic/es-distributed

vladimirdolzhenko · 2018-07-10T11:34:35Z

Note: there is a big difference between just a String a GString that could be evaluated to just a String - earlier evaluation leads to resolving old s3Fixture port - in contrast to settings those are lately evaluated after s3Fixture as dependency is started

alpar-t

Thanks for taking this up! I left some comments on the Gradle part.

alpar-t · 2018-07-10T11:47:51Z

buildSrc/src/main/groovy/org/elasticsearch/gradle/test/ClusterConfiguration.groovy

@@ -137,7 +137,7 @@ class ClusterConfiguration {
        this.project = project
    }

-    Map<String, String> systemProperties = new HashMap<>()
+    Map<String, Object> systemProperties = new HashMap<>()


The difference is easy to overlook , maybe it would be good to have a comment explaining why Object is needed.
We might be able to have Map<String, GString> to make the intention cleaner.

alpar-t · 2018-07-10T11:54:56Z

buildSrc/src/main/groovy/org/elasticsearch/gradle/test/ClusterFormationTasks.groovy

@@ -412,6 +413,31 @@ class ClusterFormationTasks {
        return parentTask
    }

+    /** Configure ES JAVA OPTS - adds system properties, assertion flags, remote debug etc */
+    static Task configureESJavaOpts(String parent, Project project, Task setup, NodeInfo node) {


I don't think we need a separate task for it, especially that the result is only stored in memory.
I would add this to the doFirst we already run on the start task.
We can just set ES_JAVA_OPTS before the start task, or concatenate it if already set.
The builds script could also set this. None of the currently do, but we should either concatenate to it if set, or fail if it's set if we don;t want to support it. I'm not saying we shouldn't just speculating on what the initial intent could have been.

@atorok do you mean to put it to

elasticsearch/buildSrc/src/main/groovy/org/elasticsearch/gradle/test/ClusterFormationTasks.groovy

Line 651 in db90905

project.logger.info("Starting node in ${node.clusterName} distribution: ${node.config.distribution}")

?

alpar-t · 2018-07-10T11:55:56Z

buildSrc/src/main/groovy/org/elasticsearch/gradle/test/NodeInfo.groovy

-            esJavaOpts = String.join(" ", "-ea", "-esa", esJavaOpts)
-        }
-        env = ['ES_JAVA_OPTS': esJavaOpts]
+        env = [:]


this can be removed, the map is already initialized.

nope, map is null, here is the output if I drop this line:

Cannot invoke method put() on null object

My bad, this class initializes in the constructor, this is ok as it is.

vladimirdolzhenko · 2018-07-10T12:50:29Z

@atorok thanks for your comments - i did a relevant amendments.

DaveCTurner

This is nice. I can't comment on the changes to gradle (except to say that I'm a little surprised they're as big as they are) and I asked for a handful of other changes to the fixture.

DaveCTurner · 2018-07-10T12:41:44Z

plugins/repository-s3/build.gradle

@@ -288,7 +288,9 @@ Map<String, Object> expansions = [
  'permanent_bucket': s3PermanentBucket,
  'permanent_base_path': s3PermanentBasePath,
  'temporary_bucket': s3TemporaryBucket,
-  'temporary_base_path': s3TemporaryBasePath
+  'temporary_base_path': s3TemporaryBasePath,
+  'ec2_bucket': s3TemporaryBucket,


Did you mean for this to be the same as the entry for temporary_bucket, and similarly for ec2_base_path? I think they should be different.

DaveCTurner · 2018-07-10T12:43:19Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

@@ -70,17 +72,24 @@ private AmazonS3Fixture(final String workingDir, final String permanentBucketNam

    @Override
    protected Response handle(final Request request) throws IOException {
-        final RequestHandler handler = handlers.retrieve(request.getMethod() + " " + request.getPath(), request.getParameters());
+        final String nonAuthorizedPath = "* " + request.getMethod() + " " + request.getPath();


I'm not a big fan of the * vs A thing, at least not as literals. Could you introduce constants so these things have more descriptive names?

DaveCTurner · 2018-07-10T12:46:23Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

        if (handler != null) {
            final String authorization = request.getHeader("Authorization");
            final String permittedBucket;
-            if (authorization.contains("s3_integration_test_permanent_access_key")) {
+            if (authorization != null && authorization.contains("s3_integration_test_permanent_access_key")) {


Could we handle the authorisation == null case first to avoid these null checks proliferating? It's also helpful to distinguish no authorisation header vs a bad one, which this would achieve.

DaveCTurner · 2018-07-10T12:46:47Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+                if (sessionToken.equals("securitycredentials42_TKN") == false) {
+                    return newError(request.getId(), RestStatus.FORBIDDEN, "AccessDenied", "Bad session token", "");
+                }
+                permittedBucket = temporaryBucketName;


As commented above, I think this should have its own bucket.

DaveCTurner · 2018-07-10T12:48:37Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

@@ -89,6 +98,15 @@ protected Response handle(final Request request) throws IOException {
                    return newError(request.getId(), RestStatus.FORBIDDEN, "AccessDenied", "Bad session token", "");
                }
                permittedBucket = temporaryBucketName;
+            } else if (authorization != null && authorization.contains("securitycredentials42_KEYID")) {


Could we extract securitycredentials42 out as a named constant so it's clear why it is what it is?

DaveCTurner · 2018-07-10T13:32:10Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+        // http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
+        handlers.insert("* GET /latest/meta-data/iam/security-credentials/{credentials}", (request) -> {
+            final String credentials = request.getParam("credentials");
+            return credentialResponseFunction.apply(credentials);


On deeper thought, this seems unduly lenient: it should only return credentials for the role that GET /latest/meta-data/iam/security-credentials/ returned, and should return 404 otherwise.

Also I think credentialResponseFunction can be inlined, it's only used in one place.

Also also we could prevent cheating slightly more by inventing random credentials when the service starts up, rather than synthesising them from the role name.

I extract it for doing ECS part as well - it has to return similar json,

for randomization ... I bit confused here in terms of how to reproduce it in case of failure

@DaveCTurner s3 perm/temp authorization tokens have to be randomized as well - should we ?

I think we cannot (yet) do this with auth tokens received externally because they need to be passed to the test suite as well as this fixture, so we've had to settle on just using the same (long) string in both places. It would indeed be nicer if there were no magic strings in the fixture at all.

For deterministic testing, maybe you can pass a seed in from Gradle somehow? Or just hard-code them, it's not that important. I'd rather they weren't synthesised as they are now.

it sounds like it is better to provide some kind of properties file that contains tokens, paths etc - esp. keeping in mind coming ECS test

Sure, good plan.

…s3fixture clean up

vladimirdolzhenko · 2018-07-10T14:29:33Z

thanks @DaveCTurner for quick review - I did changes - could you pls have another look ?

alpar-t

Gradle LGTM

DaveCTurner

I did another pass, and asked for a couple of changes. Not sure I like the heavy use of environment variables, but have asked for a second opinion.

DaveCTurner · 2018-07-11T11:38:44Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+        //
+        // http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
+        handlers.insert(NON_AUTH + " GET /latest/meta-data/iam/security-credentials/", (request) -> {
+            final String response = ec2Key;


Why ec2Key here? This should be the instance profile name, and can reasonably be a fixed value...

agree - it does not required to be ec2Key (I noticed that in following PR #31864) - but in fact it requires some instance profile name (it is not required to be fixed as it is 2 step process - to call /latest/meta-data/iam/security-credentials/ to get the instance profile name and /latest/meta-data/iam/security-credentials/{instance_profile_name})

DaveCTurner · 2018-07-11T11:39:02Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+        // GET
+        //
+        // http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
+        handlers.insert(NON_AUTH + " GET /latest/meta-data/iam/security-credentials/{credentials}", (request) -> {


... so that this doesn't need the {credentials} parameter in the URL ...

DaveCTurner · 2018-07-11T11:39:47Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+            if (ec2Key.equals(credentials) == false) {
+                return new Response(RestStatus.NOT_FOUND.getStatus(), new HashMap<>(), "unknown credentials".getBytes(UTF_8));
+            }
+            return credentialResponseFunction.apply(credentials, ec2Key, ec2Token);


... and this doesn't need to know it either.

DaveCTurner · 2018-07-11T11:41:15Z

plugins/repository-s3/build.gradle

+  env 'S3FIXTURE_TEMPORARY_SESSION_TOKEN', "${s3TemporarySessionToken}"
+  env 'S3FIXTURE_EC2_BUCKET_NAME', "${s3EC2Bucket}"
+  env 'S3FIXTURE_EC2_KEY', "${s3EC2AccessKey}"
+  env 'S3FIXTURE_EC2_TOKEN', "${s3EC2Token}"


Not so sure about this, compared with your proposal of configuring the fixture using a file. @tlrx what do you think?

Reason I changed to env variables - I faced some issues on creating a property file - again due to different stages of configuring / running - property file is not visible to fixture

DaveCTurner · 2018-07-11T11:42:17Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

-            throw new IllegalArgumentException(
-                "AmazonS3Fixture <working directory> <bucket for permanent creds> <bucket for temporary creds>");
+        if (args == null || args.length != 1) {
+            throw new IllegalArgumentException("AmazonS3Fixture <basedir");


Could this remain as <working directory>? Trailing > was lost too.

DaveCTurner · 2018-07-11T11:43:46Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

        final PathTrie<RequestHandler> handlers = new PathTrie<>(RestUtils.REST_DECODER);

        // HEAD Object
        //
        // https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html
-        objectsPaths("HEAD /{bucket}").forEach(path ->
+        objectsPaths(AUTH + " HEAD /{bucket}").forEach(path ->


Nit: how about putting the space into AUTH/NONAUTH rather than including it in all these strings?

…zonS3Fixture

DaveCTurner · 2018-07-11T14:43:21Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+        return Objects.requireNonNull(System.getenv(varName), "env variable '" + varName + "' is missing");
+    }
+
+    private static String nonAuthPath(Request request) {


DaveCTurner · 2018-07-11T14:55:28Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+        //
+        // http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
+        handlers.insert(nonAuthPath(HttpGet.METHOD_NAME, "/latest/meta-data/iam/security-credentials/"), (request) -> {
+            final String response = ec2ProfileName;


I really think this should a hard-coded value and not passed in from the environment. I don't think we gain much by accepting it from outside, and I envisage it being the sort of thing I have to look up each time I come across it. The BUCKET_NAME/KEY/TOKEN inputs are clearer (despite that the KEY and TOKEN used here could be generated internally if we could do so deterministically).

DaveCTurner · 2018-07-11T14:58:16Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+        this.ec2BucketName = envVar("S3FIXTURE_EC2_BUCKET_NAME");
+        this.ec2ProfileName = envVar("S3FIXTURE_EC2_PROFILE_NAME");
+        this.ec2Key = envVar("S3FIXTURE_EC2_KEY");
+        this.ec2Token = envVar("S3FIXTURE_EC2_TOKEN");


This nearly feels worthy of abstraction over the various sets of credentials, and I think that'll definitely be worth doing when the ECS-style credentials are added. Optional now, but worth thinking about.

…redentials_test

DaveCTurner

Yes, I much prefer the file-based config. There's a bit of refactoring still to do - see comments.

DaveCTurner · 2018-07-16T11:21:58Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+
+    private final Bucket permanentBucket;
+    private final Bucket temporaryBucket;
+    private final Bucket ec2Bucket;


The first two of these fields are unused. I think that's right, and we should remove them and also ec2Bucket, by generating the key and token and then passing them into the bucket's constructor.

DaveCTurner · 2018-07-16T11:22:52Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+            }
+            final Collection<Bucket> values = buckets.values();
+            String permittedBucket = null;
+            for (final Bucket bucket : values) {


I suspect it's now neater to turn this logic around - find the appropriate Bucket using request.getParam("bucket") and let the Bucket check the authorisation, look up the handler, and do the necessary.

I'd like to agree with you here - but we do have a corner case that does not require bucket parameter at all

// https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html handlers.insert(authPath(HttpPost.METHOD_NAME, "/")

We discussed this on Slack and concluded that this is an unimportant special case in which it's painful to check the authorization correctly but, moreover, we can just ignore the auth checks on this API without losing anything significant. Arguably this could just use a nonAuthPath. I think get this special case out of the way first and then neaten up the rest and move it into Bucket.

DaveCTurner · 2018-07-16T11:30:37Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+            final String response = "{"
+                 + "\"AccessKeyId\": \"" + key + "\","
+                 + "\"Expiration\": \"" + DateUtils.formatISO8601Date(expiration) + "\","
+                 + "\"RoleArn\": \"" + profileName + "_ROLE" + "\","


It should now be possible to make up the RoleArn and SecretAccessKey using the seeded RNG rather than by concatenating strings like this. They can reasonably be different each call.

DaveCTurner · 2018-07-16T11:33:43Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+            this(properties, buckets, prefix, false, random);
+        }
+
+        private Bucket(final Properties properties,


I think it'd be clearer to simply inline these overloads and spell out all the parameters at the call sites. We only make three of these objects, so four overloads of the constructor seems excessive :)

Also if Bucket were not static then you wouldn't need to pass buckets in.

makes sence

vladimirdolzhenko · 2018-07-16T15:27:25Z

test this please

…redentials_test

DaveCTurner

Couple more things following discussion yesterday.

DaveCTurner · 2018-07-17T13:38:34Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+            }
+            final Collection<Bucket> values = buckets.values();
+            String permittedBucket = null;
+            for (final Bucket bucket : values) {


We discussed this on Slack and concluded that this is an unimportant special case in which it's painful to check the authorization correctly but, moreover, we can just ignore the auth checks on this API without losing anything significant. Arguably this could just use a nonAuthPath. I think get this special case out of the way first and then neaten up the rest and move it into Bucket.

DaveCTurner · 2018-07-17T13:44:46Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

+        new Bucket("s3Fixture.temporary", true, null);
+        final Bucket ec2Bucket = new Bucket("s3Fixture.ec2", false, random);
+
+        this.handlers = defaultHandlers(buckets, ec2Bucket.name);


I was kind of thinking about creating the access key and token here and then passing them to Bucket() and defaultHandlers(), avoiding the null-or-Random parameter to Bucket, but that's ugly for other reasons. How about passing the bucket into defaultHandlers rather than passing its name and then looking it up again later?

I thought we agreed to keep original way of auth check due to that edge case - I do see a profit of turn multipart delete into non-auth and do auth check there on demand

DaveCTurner

LGTM. Thanks for the extra iterations, this is great.

DaveCTurner · 2018-07-18T09:09:39Z

plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AmazonS3Fixture.java

@@ -348,7 +344,12 @@ public static void main(final String[] args) throws Exception {

                                boolean found = false;
                                for (Bucket bucket : buckets.values()) {
-                                    if (bucket.objects.remove(objectName) != null) {
+                                    if (bucket.objects.containsKey(objectName)) {
+                                        final Response authResponse = authenticateBucket(request, bucket);


vladimirdolzhenko · 2018-07-18T10:18:28Z

thanks @DaveCTurner for review and comments 👍

Relates to #26913 (cherry picked from commit 8235b25)

* 6.x: Fix rollup on date fields that don't support epoch_millis (#31890) Revert "Introduce a Hashing Processor (#31087)" (#32179) [test] use randomized runner in packaging tests (#32109) Painless: Fix caching bug and clean up addPainlessClass. (#32142) Fix BwC Tests looking for UUID Pre 6.4 (#32158) (#32169) Call setReferences() on custom referring tokenfilters in _analyze (#32157) Add more contexts to painless execute api (#30511) Add EC2 credential test for repository-s3 (#31918) Fix CP for namingConventions when gradle home has spaces (#31914) Convert Version to Java - clusterformation part1 (#32009) Fix Java 11 javadoc compile problem Improve docs for search preferences (#32098) Configurable password hashing algorithm/cost(#31234) (#32092) [DOCS] Update TLS on Docker for 6.3 ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are Switch distribution to new style Requests (#30595) Build: Skip jar tests if jar disabled Build: Move shadow customizations into common code (#32014) Painless: Add PainlessClassBuilder (#32141) Fix accidental duplication of bwc test for script behavior Handle missing values in painless (#30975) (#31903) Build: Make additional test deps of check (#32015) Painless: Fix Bug with Duplicate PainlessClasses (#32110) Adjust translog after versionType removed in 7.0 (#32020) Disable C2 from using AVX-512 on JDK 10 (#32138) [Rollup] Add new capabilities endpoint for concrete rollup indices (#32111) Mute :qa:mixed-cluster indices.stats/10_index/Index - all’ [ML] Wait for aliases in multi-node tests (#32086) Ensure to release translog snapshot in primary-replica resync (#32045) Docs: Fix missing example script quote (#32010) Add Index UUID to `/_stats` Response (#31871) (#32113) [ML] Move analyzer dependencies out of categorization config (#32123) [ML][DOCS] Add missing 6.3.0 release notes (#32099) Updates the build to gradle 4.9 (#32087) Update monitoring template version to 6040099 (#32088) Fix put mappings java API documentation (#31955) Add exclusion option to `keep_types` token filter (#32012)

@return

* master: Painless: Simplify Naming in Lookup Package (#32177) Handle missing values in painless (#32207) add support for write index resolution when creating/updating documents (#31520) ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864) Remove indication of future multi-homing support (#32187) Rest test - allow for snapshots to take 0 milliseconds Make x-pack-core generate a pom file Rest HL client: Add put watch action (#32026) Build: Remove pom generation for plugin zip files (#32180) Fix comments causing errors with Java 11 Fix rollup on date fields that don't support epoch_millis (#31890) Detect and prevent configuration that triggers a Gradle bug (#31912) [test] port linux package packaging tests (#31943) Revert "Introduce a Hashing Processor (#31087)" (#32178) Remove empty @return from JavaDoc Adjust SSLDriver behavior for JDK11 changes (#32145) [test] use randomized runner in packaging tests (#32109) Add support for field aliases. (#32172) Painless: Fix caching bug and clean up addPainlessClass. (#32142) Call setReferences() on custom referring tokenfilters in _analyze (#32157) Fix BwC Tests looking for UUID Pre 6.4 (#32158) Improve docs for search preferences (#32159) use before instead of onOrBefore Add more contexts to painless execute api (#30511) Add EC2 credential test for repository-s3 (#31918) A replica can be promoted and started in one cluster state update (#32042) Fix Java 11 javadoc compile problem Fix CP for namingConventions when gradle home has spaces (#31914) Fix `range` queries on `_type` field for singe type indices (#31756) [DOCS] Update TLS on Docker for 6.3 (#32114) ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are Remove versionType from translog (#31945) Switch distribution to new style Requests (#30595) Build: Skip jar tests if jar disabled Painless: Add PainlessClassBuilder (#32141) Build: Make additional test deps of check (#32015) Disable C2 from using AVX-512 on JDK 10 (#32138) Build: Move shadow customizations into common code (#32014) Painless: Fix Bug with Duplicate PainlessClasses (#32110) Remove empty @param from Javadoc Re-disable packaging tests on suse boxes Docs: Fix missing example script quote (#32010) [ML] Wait for aliases in multi-node tests (#32086) [ML] Move analyzer dependencies out of categorization config (#32123) Ensure to release translog snapshot in primary-replica resync (#32045) Handle TokenizerFactory TODOs (#32063) Relax TermVectors API to work with textual fields other than TextFieldType (#31915) Updates the build to gradle 4.9 (#32087) Mute :qa:mixed-cluster indices.stats/10_index/Index - all’ Check that client methods match API defined in the REST spec (#31825) Enable testing in FIPS140 JVM (#31666) Fix put mappings java API documentation (#31955) Add exclusion option to `keep_types` token filter (#32012) [Test] Modify assert statement for ssl handshake (#32072)

Relates elastic#31918 / Closes infra/issues/6085

fixes broken build for third-party-tests (#32315) Relates #31918 / Closes infra/issues/6085

Relates #31918 / Closes infra/issues/6085 (cherry picked from commit 46709f1)

* 6.x: Security: revert to old way of merging automata (#32254) Fix a test bug in RangeQueryBuilderTests introduced in the field aliases backport. Introduce Application Privileges with support for Kibana RBAC (#32309) Undo a debugging change that snuck in during the field aliases merge. [test] port linux package packaging tests (#31943) Painless: Update More Methods to New Naming Scheme (#32305) Tribe: Add error with secure settings copied to tribe (#32298) Add V_6_3_3 version constant Add ERR to ranking evaluation documentation (#32314) [DOCS] Added link to 6.3.2 RNs [DOCS] Updates 6.3.2 release notes with PRs from ml-cpp repo (#32334) [Kerberos] Add Kerberos authentication support (#32263) [ML] Extract persistent task methods from MlMetadata (#32319) Backport - Add Snapshots Status API to High Level Rest Client (#32295) Make release notes ignore the `>test-failure` label. (#31309) [DOCS] Adds release highlights for search for 6.4 (#32095) Allow Integ Tests to run in a FIPS-140 JVM (#32316) Add support for field aliases to 6.x. (#32184) Register ERR metric with NamedXContentRegistry (#32320) fixes broken build for third-party-tests (#32315) Relates #31918 / Closes infra/issues/6085 [DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280) Rest HL client: Add put watch action (#32026) (#32191) Add WeightedAvg metric aggregation (#31037) Consistent encoder names (#29492) Switch monitoring to new style Requests (#32255) specify subdirs of lib, bin, modules in package (#32253) Rename ranking evaluation `quality_level` to `metric_score` (#32168) Add new permission for JDK11 to load JAAS libraries (#32132) Switch x-pack:core to new style Requests (#32252) Watcher: Store username on watch execution (#31873) Silence SSL reload test that fails on JDK 11 Painless: Clean up add methods in PainlessLookup (#32258) CCE when re-throwing "shard not available" exception in TransportShardMultiGetAction (#32185) Fail shard if IndexShard#storeStats runs into an IOException (#32241) Fix `range` queries on `_type` field for singe type indices (#31756) (#32161) AwaitsFix RecoveryIT#testHistoryUUIDIsGenerated Add new fields to monitoring template for Beats state (#32085) (#32273) [TEST] improve REST high-level client naming conventions check (#32244) Check that client methods match API defined in the REST spec (#31825)

add EC2 credential test

0278c88

vladimirdolzhenko added >test Issues or PRs that are addressing/adding tests :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs labels Jul 10, 2018

vladimirdolzhenko requested review from alpar-t and DaveCTurner July 10, 2018 11:29

vladimirdolzhenko added v7.0.0 v6.4.0 labels Jul 10, 2018

alpar-t reviewed Jul 10, 2018

View reviewed changes

Vladimir Dolzhenko added 2 commits July 10, 2018 14:41

excluded ec2_credentials from minio tests

a56d1cd

some gradle amendments on Alpar's review

5fe4227

DaveCTurner requested changes Jul 10, 2018

View reviewed changes

DaveCTurner reviewed Jul 10, 2018

View reviewed changes

changes due to David's review: use own bucket/basepath for EC2 test; …

30e86c8

…s3fixture clean up

clean AmazonS3Fixture from hard-coded keys

46dded9

vladimirdolzhenko requested a review from DaveCTurner July 10, 2018 19:16

vladimirdolzhenko mentioned this pull request Jul 11, 2018

ECS Task IAM profile credentials ignored in repository-s3 plugin #31864

Merged

alpar-t approved these changes Jul 11, 2018

View reviewed changes

DaveCTurner reviewed Jul 11, 2018

View reviewed changes

added s3EC2ProfileName to gradle; added nonAuthPath / authPath to Ama…

ccfa0c5

…zonS3Fixture

vladimirdolzhenko requested a review from DaveCTurner July 11, 2018 14:48

DaveCTurner requested changes Jul 11, 2018

View reviewed changes

polish of AmazonS3Fixture

c7af0b6

vladimirdolzhenko requested a review from DaveCTurner July 11, 2018 18:35

Vladimir Dolzhenko added 3 commits July 12, 2018 13:29

Merge remote-tracking branch 'remotes/origin/master' into lazy_repo_init

0d31887

moved to properties file for s3Fixture

fd28579

Merge remote-tracking branch 'remotes/origin/master' into 26913_ec2_c…

4006d19

…redentials_test

Vladimir Dolzhenko added 2 commits July 12, 2018 20:29

fix s3FixtureProperties generation

a4b3613

make fixture random again

9c4c3e7

DaveCTurner reviewed Jul 16, 2018

View reviewed changes

some code improvements

5430473

vladimirdolzhenko requested a review from DaveCTurner July 16, 2018 13:00

Merge remote-tracking branch 'remotes/origin/master' into 26913_ec2_c…

0ffae39

…redentials_test

DaveCTurner reviewed Jul 17, 2018

View reviewed changes

(hopefully) final polish of AmazonS3Fixture

161deee

vladimirdolzhenko requested a review from DaveCTurner July 17, 2018 17:49

DaveCTurner approved these changes Jul 18, 2018

View reviewed changes

vladimirdolzhenko merged commit 8235b25 into elastic:master Jul 18, 2018

vladimirdolzhenko deleted the 26913_ec2_credentials_test branch July 18, 2018 10:18

vladimirdolzhenko pushed a commit that referenced this pull request Jul 18, 2018

Add EC2 credential test for repository-s3 (#31918)

cfae31d

Relates to #26913 (cherry picked from commit 8235b25)

cbuescher mentioned this pull request Jul 19, 2018

[CI] Third-party tests for repository-s3 fails #32192

Closed

vladimirdolzhenko pushed a commit to vladimirdolzhenko/elasticsearch that referenced this pull request Jul 24, 2018

fixes broken build for third-party-tests

68ed322

Relates elastic#31918 / Closes infra/issues/6085

vladimirdolzhenko mentioned this pull request Jul 24, 2018

[CI] Reactivate 3rd party tests on CI #32315

Merged

vladimirdolzhenko added a commit that referenced this pull request Jul 24, 2018

fixes broken build for third-party-tests (#32315)

46709f1

fixes broken build for third-party-tests (#32315) Relates #31918 / Closes infra/issues/6085

vladimirdolzhenko added a commit that referenced this pull request Jul 24, 2018

fixes broken build for third-party-tests (#32315)

2bf69b1

Relates #31918 / Closes infra/issues/6085 (cherry picked from commit 46709f1)

jimczi added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Add EC2 credential test for repository-s3 #31918

Add EC2 credential test for repository-s3 #31918

Conversation

vladimirdolzhenko commented Jul 10, 2018

elasticmachine commented Jul 10, 2018

vladimirdolzhenko commented Jul 10, 2018

alpar-t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirdolzhenko Jul 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirdolzhenko commented Jul 10, 2018

DaveCTurner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirdolzhenko commented Jul 10, 2018

alpar-t left a comment

Choose a reason for hiding this comment

DaveCTurner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DaveCTurner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirdolzhenko Jul 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirdolzhenko commented Jul 16, 2018

DaveCTurner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DaveCTurner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirdolzhenko commented Jul 18, 2018

vladimirdolzhenko Jul 10, 2018 •

edited

Loading

vladimirdolzhenko Jul 16, 2018 •

edited

Loading