Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ES bulk method incomplete execution when errors #52

Open
sorhent opened this issue Aug 9, 2017 · 1 comment
Open

ES bulk method incomplete execution when errors #52

sorhent opened this issue Aug 9, 2017 · 1 comment

Comments

@sorhent
Copy link

sorhent commented Aug 9, 2017

I mentioned this issue inside yougov/mongo-connector#446 but taking into account that it is related to https://github.com/mongodb-labs/elastic2-doc-manager repository, I mention it here again
When inside a bulk request there are failing operations the bulk method stops at failing operation even there are others operations to treat and an error is raised. These leads to inconsistency between mongo and ES.

The steps to reproduce this, are:

  1. Start full-dump of mongo-db.
  2. While dump is ongoing - delete some documents from MongoDb.
  3. As mongo-connector hasn't yet dumped mentioned document and document has been deleted in the meantime, it will not be added to ES.
  4. After dump - mongo-connector goes through oplog and it is trying to delete mentioned document again even that it has not been added to ES.

In order to solve this, I propose the folowing approch :
When ~elasticsearch.helpers.bulk or ~elasticsearch.helpers.streaming_bulk method are called set 'raise_on_error' and 'raise_on_exception' to False:
ex:
kw['raise_on_error'] = False
kw['raise_on_exception'] = False
successes, errors = bulk(self.elastic, action_buffer, **kw)
Like this, all errors of ES bulk opperations will be logged but the method call will not be interrupted by an exception.

@barbaluc
Copy link

Thanks a lot @sorhent for your solution, it works for me !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants