Decouple Elasticsearch from CWS, make into external dependency

[ztaylor54]

This is a rather large refactor that moves elasticsearch from a bundled software to an external dependency. As elasticsearch was previously tightly coupled with CWS's installer and REST service, this change touches a wide range of files.

In short, CWS no longer installs elasticsearch, but rather expects users to have installed it externally & configured it to their liking. In place of a bundled installation, CWS now expects the following configuration properties:

elasticsearch_host=[YourElasticsearchHost]
elasticsearch_port=9200
elasticsearch_use_auth=y
elasticsearch_username=[YourElasticsearchUsername]
elasticsearch_password=[YourElasticsearchPassword]

I have tried to provide sensible defaults where possible:

• elasticsearch_port defaults to 9200
• elasticsearch_use_auth defaults to y

Using these new configuration properties, CWS can interface with both secured and unsecured elasticsearch installations. If a user plans to simply use elasticsearch without authentication, they can specify elasticsearch_use_auth=n in their configuration.properties file, and CWS will not use authentication.

Where authentication is desired, I have refactored the WebUtils.restCall() method in cws-core to set the required Authentication header in the base64 encoded format username:password, and to only allow this for secure connections with valid certificates.

Logstash is also provided additional configuration in cws-logstash.conf to allow it to connect to a secured elasticsearch instance, as per the docs:

If using authentication:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    user => <elasticsearch_username in configuration.properties>
    password => <elasticsearch_password in configuration.properties>
  }
}

If no auth:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

I have tested this change with a locally-deployed and unsecured elasticsearch instance on my machine, and everything worked well. This still needs to be tested with a secured elasticsearch instance, but I'm submitting this PR so I can get more eyes on it in the meantime.

[jamesfwood]

Maybe it might be better to default ES to no auth so it could basically work without user changes (use defaults), just as long as they have ES running.

[jamesfwood]

Also, using ES with auth should be tested before we merge this.

[jamesfwood]

What version of ES do we support? We should probably add some kind of restriction to what version of ES the user can use. Just in case the ES API changes in the future

[jamesfwood]

Or add to the docs that we support version X.X of ES

[ztaylor54]

I have updated this PR with a commit that both bumps logstash to 7.9.0 and fixes configuration issues that were preventing it from connecting to elasticsearch.

Tested with AWS Elasticsearch v7.9, using user/pass authentication. Everything seems to be working - both the logstash connection (verified with kibana) and the REST endpoint.

So, officially, we can say that we support:

• Elasticsearch v7.9+
• Logstash v7.9.0

[jamesfwood]

It still has es_use_auth defaulting to "y" right? My only concern with this is that if users use an older config file, which a lot of people do (including me), then it won't work because it will be missing the es username and pw. If it's set to es_use_auth = n then old config files will still work.

Your thoughts on this?

[ztaylor54]

@jamesfwood Excellent point, I've amended this PR with a commit making that change!

We now default to elasticsearch_use_auth=n.

[jamesfwood]

Great, thanks! I'm glad we upgraded ES and LS in this PR also.

Nice work!