EBMEDS-1243: update the elastic stack to the latest minor

[SkylarPhx]

EBMEDS-1243

This pull request updates the Elastic stack to the latest minor version. it changes the version number in the start.sh. Also, all the breaking changes have been reviewed.

Related changes:
api-gateway: https://github.com/ebmeds/api-gateway/pull/90
clinical-datastore: https://github.com/ebmeds/clinical-datastore/pull/83
engine: https://github.com/ebmeds/engine/pull/265

Elastic stack breaking changes from 6.2 to 6.8

Elasticsearch

• Breaking Changes in 6.3
  • The elasticsearch module of the engine required refactoring because the bulk thread pool has been renamed to the write thread pool.
• Breaking Changes in 6.4
  • No action is required.
• Breaking Changes in 6.5
  • No action is required.
• Breaking Changes in 6.6
  • No action is required.
• Breaking Changes in 6.7
  • No action is required.
• Breaking Changes in 6.8
  • There are no breaking changes for 6.8.0.

Kibana

• Breaking changes in v6.3
  • No action is required.
• Breaking changes in v6.4
  • No action is required
• Breaking changes in v6.6
  • No action is required.
• Breaking Changes in v6.7
  • No action is required.

Logstash

• Persistent Queue
  • The deployment environments required to be drained the persistent queue before upgrading.

APM

• Breaking changes in v6.4
  • Might need to refresh an APM index pattern.

APM Agent

• Upgrade to v2.x
  • No additional action is required to update the APM agent to version 2.x.

Things to resolve after upgrage:

• Could not indexe event to Elasticsearch when sending to ebmeds-logs-*:
  • Done in pull request https://github.com/ebmeds/engine/pull/269

ebmeds_logstash.1.rpb06eid5fl4@summer    | [2019-09-18T15:14:38,035][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ebmeds-logs-2019.09.18", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x7f509adf>], :response=>{"index"=>{"_index"=>"ebmeds-logs-2019.09.18", "_type"=>"doc", "_id"=>"DnbwRG0BbY7MbrsYi18-", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [body.keys] of type [text] in document with id 'DnbwRG0BbY7MbrsYi18-'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:138"}}}}}

• Logstash expecting that the body.keys field type is a text but found a type of array, therefore the parsing failed. Solution for this issue is to send the keys field value as text rather than array. Only clinical-datastore are sending this kind of information when debug log level is on.

[joonatanv]

An exemplary PR message -- it clearly conveys the relevant information in an actionable manner. Great job @SkylarPhx!