Deprecating Rivers

[kimchy]

The reasoning behind it is captured in this blog post: https://www.elastic.co/blog/deprecating_rivers

We need to @Deprecate the River classes, and deprecate them in the docs, as well as add a deprecation notice in the rivers that are under elastic.

[clintongormley]

Docs have been added to indicate that rivers are deprecated. Still need to add deprecation annotations to the code

[dharshanr]

Is there a place to have a discussion on this? Looks like the problem has been punted rather than fixed. From a customer perspective it doesn't matter if the import code runs inside elastic or not - its more about importing data.

If stability was really an issue then a better solution would be to provide a standalone daemon that will be able to able to host the rivers and allow customers to import the data instead of leaving them to fend for themselves.

[carljohnstone]

Sorry I'm not hugely familiar with the code-base itself, but would it be possible to write a non-river plugin, that pulled in data from an external source than indexed it into Elasticsearch using the bulk-data APIs? Would you expect that to be any more stable than a plugin coded against the River classes? Eg is the actual problem the fact that these are plugins running within the ES server itself, rather than rivers.

[kimchy]

@dharshanr / @carljohnstone in those cases, there really isn't any benefit in "Elasticsearch" itself having a framework for it, and its better left to be done by projects that are fully geared towards ETL such workloads, with native integration to Elasticsearch (which is where I think spending time would provide the best value). Logstash is a great example, but there are many more.

[nkavian]

I agree with @dharshanr this is punting the problem. We were enjoying using the Couchbase plugin to replicate into ES bypassing the app layer. Now what I'm understanding is the app layer now needs to copy into both places..

[dadoonet]

Couchbase plugin is a transport plugin not a river plugin so not concerned by this change.

[nik9000]

@dharshanr / @carljohnstone in those cases, there really isn't any benefit in "Elasticsearch" itself having a framework for it

I figure the advantage of rivers is that you're already deploying Elasticsearch and they are already written. As much trouble as they are, they exist.

I'm still +1 on deprecating them.

[bgiromini]

The biggest hole in my opinion is getting data from a relational database into ES. I know that some work is being down on the logstash front but would be nice if it was officially supported by Elastic.

[dadoonet]

@bgiromini in case you missed it, have a look at https://github.com/logstash-plugins/logstash-input-jdbc

[hash-include]

One quick question, If I am using river-mongodb and most of the company code is in Java. Can I push data into elasticsearch from Java API removing the river?

[dadoonet]

@hash-include This is indeed my favorite way for doing that! Pushing from the application directly :)
I wrote a blog post about it though it's based on RDBMS and not NoSQL db: http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/

[s1monw]

@dadoonet do you know what is missing here to close this?

[dadoonet]

@s1monw I think we can close it. It has been deprecated in 1.6.0 in docs and in 2.0.0 in code.
Note that we are not using the deprecation logger in master though: https://github.com/elastic/elasticsearch/blob/master/src/main/java/org/elasticsearch/river/RiversService.java#L129

Closing. If I'm missing anything, feel free to reopen.

[tchcxp]

@kimchy

I have been using mongodb river for a while. So the alternative solution is to update data in the application or other services with elastic API calls?

Thanks.

[dadoonet]

@tchcxp I answered the same question a month ago: #10345 (comment)

HTH

[tchcxp]

@dadoonet

Cool! Great help. Thanks. 👍

[syllogismos]

@tchcxp how well did updating data in elasticsearch from app layer work, instead of relying on mongodb river.

[ghost]

@kimchy I want to replicate same set of data from mongoDB server to elasticSearch server and vice
versa. What should i use?

[dadoonet]

@monikamaheshwari please ask your question on discuss.elastic.co. I'm sure many users will be able to share their experience with you. Or better, search on discuss for previous discussions regarding this.