Add delete-by-query plugin #11516

tlrx · 2015-06-05T16:55:31Z

This pull request adds a new plugin called "delete-by-query" which implements the now deprecated delete-by-query feature using scan/scroll/bulk requests.

Notes:

size parameter controls the scroll shard_size and the number of actions in bulk requests (defaults to 1000)
timeout parameter can be used to stop scrolling documents after a given time
response now looks like this (here a node is killed during the DBQ execution):

{  
   "took":60866,
   "timed_out":false,
   "_indices":{  
      "_all":{  
         "found":531046,
         "deleted":79901,
         "missing":0,
         "failed":301702
      },
      "disposants-2014":{  
         "found":375702,
         "deleted":74000,
         "missing":0,
         "failed":301702
      },
      "beer":{  
         "found":5901,
         "deleted":5901,
         "missing":0,
         "failed":0
      }
   },
   "failures":[  
      {  
         "shard":-1,
         "index":null,
         "reason":{  
            "type":"node_not_connected_exception",
            "reason":"[Puck][inet[/192.168.1.16:9300]] Node not connected"
         }
      }
   ]
}

Since the process involves the execution of a scan request (which can fail), then successive async scroll requests (which can also fail) we may imagine a better failure reporting. When a scroll request succeed, the scrolled documents are added to a Bulk request executed in an async manner. If the bulk fails, all documents are reported as failed documents in the counter.

Rest API documentation and test will be added later.

kimchy · 2015-06-06T18:03:25Z

...query/src/main/java/org/elasticsearch/deletebyquery/action/TransportDeleteByQueryAction.java

+                }
+                scanRequest.source(source);
+
+                logger.debug("executing scan request");


this probably need to be trace, action package has DEBUG enabled by default

applies to other logging statements here

kimchy · 2015-06-06T18:08:31Z

I left some minor comments around logging and usage of thread pool (not needed I think).

I could't follow why we need a semaphore and such, I think I a missing something. My thought was that we do search -> bulk -> search -> .... until there are no more results, so always async callback execution type chain until we are done.

tlrx · 2015-06-09T10:10:57Z

@kimchy thanks for your review! Your comments make sense, no need to use semaphore stuff... I rebased and updated the code, it is way simpler now.

I'll add some rest tests too.

kimchy · 2015-06-09T11:12:04Z

...query/src/main/java/org/elasticsearch/action/deletebyquery/TransportDeleteByQueryAction.java

+                    final String nextScrollId = scrollResponse.getScrollId();
+                    addShardFailures(scrollResponse.getShardFailures());
+
+                    if (logger.isDebugEnabled()) {


this should be trace?

tlrx · 2015-06-10T12:09:09Z

@s1monw thanks for your review. I updated the code following your comment and added a REST test. Can you please have another look if possible? Thanks :)

Documentation will be added in another PR.

s1monw · 2015-06-12T12:29:54Z

...lete-by-query/src/main/java/org/elasticsearch/action/deletebyquery/DeleteByQueryRequest.java

+            out.writeBoolean(false);
+        } else {
+            out.writeBoolean(true);
+            out.writeVLong(timeout);


if you write a VLong make sure it's not negative!

Ok, timeout has been changed to TimeValue

s1monw · 2015-06-12T12:41:08Z

...query/src/main/java/org/elasticsearch/action/deletebyquery/TransportDeleteByQueryAction.java

+        }
+
+        private boolean isTimedOut() {
+            return request.timeout() != null && (System.currentTimeMillis() >= (startTime + request.timeout().millis()));


also use the Threadpool estimations here mabye?

s1monw · 2015-06-12T12:48:35Z

looks pretty good though. I left a bunch of comments

tlrx · 2015-06-16T11:13:35Z

@s1monw thanks a lot for your review, very valuable. I updated the code following your comments, please let me know if there are still things to improve.

I'd love to have your help on writing documentation for this plugin, since I'm not sure to be able to explain all fallacies of the previous implementation.

s1monw · 2015-06-16T18:34:17Z

I'd love to have your help on writing documentation for this plugin, since I'm not sure to be able to explain all fallacies of the previous implementation.

lets get this in as is and open an issue for the documentation I will take a look at comment on it what aspects I would take into account?

s1monw · 2015-06-16T18:34:36Z

oh yeah so here is my LGTM ;)

The delete by query plugin adds support for deleting all of the documents (from one or more indices) which match the specified query. It is a replacement for the problematic delete-by-query functionality which has been removed from Elasticsearch core in 2.0. Internally, it uses the Scan/Scroll and Bulk APIs to delete documents in an efficient and safe manner. It is slower than the old delete-by-query functionality, but fixes the problems with the previous implementation. Closes elastic#7052

tlrx · 2015-06-17T13:52:03Z

@s1monw thanks!

I created #11723 for the java doc aspect.

This page is placed in a /plugins directory until we figure where to place all plugins documentation.

tlrx added v2.0.0-beta1 review labels Jun 5, 2015

kimchy reviewed Jun 6, 2015
View reviewed changes

clintongormley added >feature :Core/Infra/Plugins Plugin API and infrastructure labels Jun 8, 2015

tlrx force-pushed the delete-by-query branch from bd5d1b9 to 46ed9a1 Compare June 9, 2015 10:07

kimchy reviewed Jun 9, 2015
View reviewed changes

This was referenced Jun 10, 2015

Add documentation for delete by query plugin #11584

Merged

Reimplement delete-by-query as a bulk request #7052

Closed

s1monw reviewed Jun 12, 2015
View reviewed changes

tlrx mentioned this pull request Jun 16, 2015

Add shortcut for match_all query in the Delete-By-Query plugin #11700

Closed

tlrx force-pushed the delete-by-query branch from 5825921 to f82c7df Compare June 16, 2015 11:17

tlrx force-pushed the delete-by-query branch from f82c7df to ba35406 Compare June 17, 2015 13:39

tlrx merged commit ba35406 into elastic:master Jun 17, 2015

kevinkluge removed the review label Jun 17, 2015

tlrx mentioned this pull request Jun 17, 2015

Document TransportDeleteByQueryAction class #11723

Closed

tlrx added the :Plugin Delete By Query label Jun 18, 2015

clintongormley removed the :Core/Infra/Plugins Plugin API and infrastructure label Jun 18, 2015

tlrx added a commit to tlrx/elasticsearch that referenced this pull request Jun 23, 2015

Add documentation for delete by query plugin (see elastic#11516)

26e7827

This page is placed in a /plugins directory until we figure where to place all plugins documentation.

dadoonet mentioned this pull request Jul 8, 2015

[build] cloud-aws doesn't register s3 repos anymore #12036

Merged

tlrx deleted the delete-by-query branch May 19, 2016 09:52

lcawl added :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. and removed :Plugin Delete By Query labels Feb 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add delete-by-query plugin #11516

Add delete-by-query plugin #11516

tlrx commented Jun 5, 2015

kimchy Jun 6, 2015

kimchy Jun 6, 2015

tlrx Jun 9, 2015

kimchy commented Jun 6, 2015

tlrx commented Jun 9, 2015

kimchy Jun 9, 2015

tlrx Jun 9, 2015

tlrx commented Jun 10, 2015

s1monw Jun 12, 2015

tlrx Jun 16, 2015

s1monw Jun 12, 2015

tlrx Jun 16, 2015

s1monw commented Jun 12, 2015

tlrx commented Jun 16, 2015

s1monw commented Jun 16, 2015

s1monw commented Jun 16, 2015

tlrx commented Jun 17, 2015

Add delete-by-query plugin #11516

Add delete-by-query plugin #11516

Conversation

tlrx commented Jun 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kimchy commented Jun 6, 2015

tlrx commented Jun 9, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tlrx commented Jun 10, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1monw commented Jun 12, 2015

tlrx commented Jun 16, 2015

s1monw commented Jun 16, 2015

s1monw commented Jun 16, 2015

tlrx commented Jun 17, 2015