Conductor not compatible with Elasticsearch 5.3.0

[cquon]

I was thinking of upgrading the elasticsearch version. On the hub.docker.com for elasticsearch (https://hub.docker.com/_/elasticsearch/), it says the images are deprecated and to use the images provided by elastic.co
Using the latest image specified by docker.elastic.co/elasticsearch/elasticsearch:5.3.0, I changed the docker-compose.yml so elasticsearch could use the new image. However on upgrading the server image, but using the existing elasticsearch client gradle specifies (2.+) in redis-persistence, the elasticsearch container throws an exception:
elasticsearch_1 | java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0]

So I proceeded on upgrading the client version in the redis-persistence build.gradle from 2.+ to 5.3.0

After doing so and trying to start a "gradle build" in the redis-persistnce, some errors/warnings were thrown (I built with the -Xlint:deprecation flag since there were warnings that deprecated methods were being used)

# gradle build
Inferred project: conductor, version: 1.7.0-SNAPSHOT
Publication nebula not found in project :.
Publication named 'nebula' does not exist for project ':' in task ':artifactoryPublish'.
:conductor-common:compileJava UP-TO-DATE
:conductor-common:processResources NO-SOURCE
:conductor-common:classes UP-TO-DATE
:conductor-common:writeManifestProperties UP-TO-DATE
:conductor-common:jar UP-TO-DATE
:conductor-core:compileJava UP-TO-DATE
:conductor-core:processResources NO-SOURCE
:conductor-core:classes UP-TO-DATE
:conductor-core:writeManifestProperties UP-TO-DATE
:conductor-core:jar UP-TO-DATE
:conductor-redis-persistence:compileJava

1.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:51: error: cannot find symbol
import org.elasticsearch.indices.IndexAlreadyExistsException;
                                ^
  symbol:   class IndexAlreadyExistsException
  location: package org.elasticsearch.indices

2.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:142: error: cannot find symbol
                        } catch (IndexAlreadyExistsException ilee) {
                                 ^
  symbol:   class IndexAlreadyExistsException
  location: class ElasticSearchDAO

3.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:163: warning: [deprecation] setSource(byte[]) in PutIndexTemplateRequestBuilder has been deprecated
                                client.admin().indices().preparePutTemplate("wfe_template").setSource(templateSource).execute().actionGet();
                                                                                           ^

4.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:175: error: cannot find symbol
                        }catch(IndexAlreadyExistsException done) {}
                               ^
  symbol:   class IndexAlreadyExistsException
  location: class ElasticSearchDAO

5.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:186: warning: [deprecation] setSource(String) in PutMappingRequestBuilder has been deprecated
                                client.admin().indices().preparePutMapping(indexName).setType(WORKFLOW_DOC_TYPE).setSource(source).execute().actionGet();
                                                                                                                ^

6.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:202: warning: [deprecation] doc(byte[]) in UpdateRequest has been deprecated
                        req.doc(doc);
                           ^

7.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:203: warning: [deprecation] upsert(byte[]) in UpdateRequest has been deprecated
                        req.upsert(doc);
                           ^

8.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:221: warning: [deprecation] doc(byte[]) in UpdateRequest has been deprecated
                        req.doc(doc);
                           ^

9.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:222: warning: [deprecation] upsert(byte[]) in UpdateRequest has been deprecated
                        req.upsert(doc);
                           ^

10.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:242: warning: [deprecation] source(byte[]) in IndexRequest has been deprecated
                                request.source(om.writeValueAsBytes(taskExecLog));
                                       ^

11.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:293: warning: [deprecation] doc(byte[]) in UpdateRequest has been deprecated
                        req.doc(doc);
                           ^

12.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:294: warning: [deprecation] upsert(byte[]) in UpdateRequest has been deprecated
                        req.upsert(doc);
                           ^

13.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:342: error: cannot find symbol
                        if (!response.isFound()) {
                                     ^
  symbol:   method isFound()
  location: variable response of type DeleteResponse

14.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticSearchDAO.java:377: error: cannot find symbol
                final SearchRequestBuilder srb = client.prepareSearch(indexName).setQuery(fq).setTypes(WORKFLOW_DOC_TYPE).setNoFields().setFrom(start).setSize(size);
                                                                                                                         ^
  symbol:   method setNoFields()
  location: class SearchRequestBuilder

15.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticsearchModule.java:54: error: cannot find symbol
        Settings.Builder settings = Settings.settingsBuilder();
                                            ^
  symbol:   method settingsBuilder()
  location: class Settings

16.
/tmp/conductor2/redis-persistence/src/main/java/com/netflix/conductor/dao/index/ElasticsearchModule.java:58: error: cannot find symbol
        TransportClient tc = TransportClient.builder().settings(settings).build();
                                            ^
  symbol:   method builder()
  location: class TransportClient
7 errors
9 warnings
:conductor-redis-persistence:compileJava FAILED

FAILURE: Build failed with an exception.

I have added numbers to help identify issues.

It appears the Conductor is using a pretty old version of elasticsearch and since many API changes have occurred since.

Issue 1, 2, 4:
IndexAlreadyExistsException replaced with ResourceAlreadyExistsException
https://www.elastic.co/guide/en/elasticsearch/reference/current/release-notes-5.1.1.html
elastic/elasticsearch#21494

Issue 3, 5-12:
Methods need to accept XContentType (was going to add XContentType.JSON for additional parameters)
https://www.elastic.co/guide/en/elasticsearch/reference/5.3/release-notes-5.3.0.html
elastic/elasticsearch#23078

Issue 13:
isFound method removed
elastic/elasticsearch@119026b
elastic/elasticsearch@9ac6389

Issue 14:
fields was renamed to storedFields
elastic/elasticsearch@eb1e231
Then it was removed entirely and instead set the stored_fields == "_none"
elastic/elasticsearch@4682fc3

Issue 15:
settingsBuilder removed
elastic/elasticsearch@42526ac

Issue 16:
Transport Client moved:
https://www.elastic.co/guide/en/elasticsearch/reference/5.3/breaking_50_java_api_changes.html
https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.3/transport-client.html

Can I work on adding support for Elasticsearch 5.3.0?

[v1r3n]

@cquon we will be happy to accept a PR for ES 5.3.x support. Please send a PR against the dev branch.

[v1r3n]

Closing this issue. We can continue the further conversation on the PR.