Index is not being created in Elasticsearch

[rcastill]

Hi,

I've been trying to transport a mongodb collection to an index in Elasticsearch. It seems that my config is correct; in fact, transport exits with no errors (I tested some debug like passthrough_and_log.js, and it shows what is supposed to), problem is, when I list the available indexes in Elasticsearch, they were not created. I've been trying this from yesterday, and nothing works.

Some important notes:

• I tried the same with /tmp/foo as output sink and it worked flawlessly.
• When I use the same names for db.collection -> index.type e.g.: foo.bar -> foo.bar, transport process is slow (I was able to note this because I also tried to transport a custom collection which is larger - it took hours!). But today I tried: foo.bar -> foo.type1, and it worked ridiculously fast (11secs) compared to the former.
• I am using the transporter 0.1.1 binary release for linux x86_64 on RHEL7
• I am using Elasticsearch 5.1.1 (maybe elasticgo does not support this version?)

[jipperinbham]

@rcastill can you provide the yaml and js files used (with any credentials removed)? We're working on getting the elasticsearch adaptor stabilized due to all of the changes between ES 1.7->2.X->5.X.

[rcastill]

*Sigh*, I realized a problem thanks to your answer: I was not passing credentials.

Now, I passed credentials for the test under "simple" directory in your tests and it worked.

New things to consider:

• It looks like "foo.type1" won't work because data is not indexed in Elasticsearch.
• I'm currently running transport to index a personal collection (91268 documents) and it is slow as I stated before (very slow :c)
• In the latter indexing process I'm getting multiple:

transporter: CRITICAL: elasticsearch error (Bulk Insertion Error. Failed item count [N])

Where N has taken as values 67, 68, 69 so far

Do you still need my config files?

[jipperinbham]

It looks like "foo.type1" won't work because data is not indexed in Elasticsearch.

if you want to change the type/collection part, you'd need to use a transform function to change the namespace, something like the following should work:

module.exports = function(doc) {
  doc.ns = "foo.type1";
  return doc
}

and make sure you have the namespace set properly in the app.js file.

I'm currently running transport to index a personal collection (91268 documents) and it is slow as I stated before (very slow :c)

it's hard to say what could be the cause here but I assume you're running transporter locally? and trying to send data to a remote elasticsearch?
re: performance, it's something we hope to solve with #209

re:

transporter: CRITICAL: elasticsearch error (Bulk Insertion Error. Failed item count [N])

That looks to be an error on the elasticsearch side so would possibly need to look at the logs there.

[rcastill]

In fact both elasticsearch and mongo are local.

I finally gave up, since it may be a version incompatibility issue and started working in a tool in Python using elasticsearch and pymongo libs.

Nevertheless I'll be following this project for any update that may come.

Greetings!

[JherezTaylor]

@rcastill Bulk Insertion is most likely due to malformed _id.
You need to handle it like so:

module.exports = function(msg) { msg.data._id = msg.data._id.$oid; return msg; }

This maintains the same _id in both ES and Mongo.

[rcastill]

I tried that already with no success

[JherezTaylor]

Okay, well if you still have your configs you can attach them and we can take a look. (If you haven't fully moved on yet). Remember to redact any sensitive info

[cdechery]

Exact same problem here. There are no error in the output anywhere and I see no indexes getting created in ES.

Here are my files:

nodes:
  mongo:
    type: mongo
    uri: mongodb://dadhx03.interno:27018/TecnicoVirtual
    namespace: TecnicoVirtual.Logging
    debug: true
  es:
    type: elasticsearch
    uri: http://dadhx03.interno:9201/
    namespace: tecvirt.logging
    debug: true
  debug:
    type: file
    uri: stdout://
  stdout:
    type: file
    uri: stdout://

pipeline = Source({name:"mongo", namespace:"TecnicoVirtual.Logging", tail:true, bulk:true}).transform({filename: "transformers/passthrough_and_log.js", namespace: "Tecni
coVirtual.Logging"}).save({name:"es", namespace:"tecvirt.logging"});

[jipperinbham]

@cdechery what version of elasticsearch are you testing with and also what version of transporter?

[cdechery]

@jipperinbham ES version 2.4;, transporter 0.1.0.
I downloaded a docker image that contained a mongo, an elasticsearch and transporter, all setup to work, but it didn't. I got version problems. So I updated ES to try, and now I can see on my output all the documents from mongo being printed out (as they run through the transform) but nothing gets created on ES and I see no error messages.

[jipperinbham]

gotcha, there was a bug fix that went out with 0.1.1, would you be able to try that version?

also, we're planning on publishing official packages and images in the near future

[SYTF]

hello, I got the same problem like @cdechery case.
My ES version is 5.1.1 and transporter is 0.1.1

all documents can be printed out from mongo
but nothing created on ES too

[cdechery]

I'm finally getting the data in the index. After updating to version 0.11 and changing some settings on the transporter JS files. But it is still not perfect.

I'm getting several messages like this transporter: CRITICAL: elasticsearch error (Bulk Insertion Error. Failed item count [45]) and oddly enough, the number of documents loaded into ES is bigger than the number of documents on the source database.

Also, the process doesn't END. It gets stuck after theses error messages.

[jipperinbham]

@cdechery @SYTF @rcastill @JherezTaylor please try out the latest release, v0.1.2. I'm going to close this issue out but please feel free to re-open it or create a new one if you're seeing a problem.