You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I mentioned this issue inside yougov/mongo-connector#446 but taking into account that it is related to https://github.com/mongodb-labs/elastic2-doc-manager repository, I mention it here again
When inside a bulk request there are failing operations the bulk method stops at failing operation even there are others operations to treat and an error is raised. These leads to inconsistency between mongo and ES.
The steps to reproduce this, are:
Start full-dump of mongo-db.
While dump is ongoing - delete some documents from MongoDb.
As mongo-connector hasn't yet dumped mentioned document and document has been deleted in the meantime, it will not be added to ES.
After dump - mongo-connector goes through oplog and it is trying to delete mentioned document again even that it has not been added to ES.
In order to solve this, I propose the folowing approch :
When ~elasticsearch.helpers.bulk or ~elasticsearch.helpers.streaming_bulk method are called set 'raise_on_error' and 'raise_on_exception' to False:
ex: kw['raise_on_error'] = False kw['raise_on_exception'] = False successes, errors = bulk(self.elastic, action_buffer, **kw)
Like this, all errors of ES bulk opperations will be logged but the method call will not be interrupted by an exception.
The text was updated successfully, but these errors were encountered:
I mentioned this issue inside yougov/mongo-connector#446 but taking into account that it is related to https://github.com/mongodb-labs/elastic2-doc-manager repository, I mention it here again
When inside a bulk request there are failing operations the bulk method stops at failing operation even there are others operations to treat and an error is raised. These leads to inconsistency between mongo and ES.
The steps to reproduce this, are:
In order to solve this, I propose the folowing approch :
When ~elasticsearch.helpers.bulk or ~elasticsearch.helpers.streaming_bulk method are called set 'raise_on_error' and 'raise_on_exception' to False:
ex:
kw['raise_on_error'] = False
kw['raise_on_exception'] = False
successes, errors = bulk(self.elastic, action_buffer, **kw)
Like this, all errors of ES bulk opperations will be logged but the method call will not be interrupted by an exception.
The text was updated successfully, but these errors were encountered: