Percolators: Support delete-by-query

[onilton]

The error doesn't happen when I delete the queries one by one.

It only happens when you delete the percolator index itself for a index.

Gist for reproducing the error:

https://gist.github.com/1678150

[timjnh]

Can't say I tested deleting the percolator index itself but I have run across this issue when deleting _percolator queries by query rather than by id.

[ajhalani]

I have encountered this bug too.

[ajhalani]

+1, encountered it too.

[anweibel]

+1, encountered it too.

We create the percolators each night anew during re-indexing. It would be great if we could just delete the _percolator index in order to delete all percolators. This doesn't work, as described above.

Instead, our workaround is:

1. lookup all percolators
2. delete each percolator individually

That's neither elegant nor efficient, but it works so far.

[georgi0u]

+1, also experiencing this issue.

[julianhille]

Same problem here. The index as the type of the index deleting does nothing.
Its really weird, because trying to get the percolator by id says exist false. Deleting by id says exist false but also tells that deletion was successful.

[cjbottaro]

The workaround doesn't work for me. Searching for percolator documents yields nothing:

$ curl -X GET 'http://localhost:9200/_percolator/_search?pretty' -d '{}'
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

But then percolating a document yields results...

$ curl -X GET "http://localhost:9200/scholarship_manager/dev%7EApplication%3A%3AAutomatch/_percolate?pretty" -d '{"doc":{"form_field_34":4.0,"type":"dev~Application::Automatch"}}'
{
  "ok" : true,
  "matches" : [ "opportunity_46:qualification_group_1:qualification_1", "dev~opportunity_46:qualification_group_1:qualification_1" ]
}

Not sure how to get out of this state as deleting the _percolator index does nothing. I'm using Elasticsearch 0.20.6.

[spinscale]

Deleting a type from the percolator index via

curl -X DELETE localhost:9200/_percolator/myindex

internally issues a delete-by query, which succeeds, but does not update the registered percolator queries. This is the reason for both failing at the moment.

[JimminiKin]

So, once a percolator has been "deleted" the wrong way, it's stuck forever in the system?

Is there any solution to work around this issue ? Or do I have to start over on a clean setup ?

[georgi0u]

Restarting the cluster is a temporary solution.

On Thu, Aug 1, 2013 at 9:26 AM, Jimmy Thomas [email protected]:

So, once a percolator has been deleted the wrong way, it's stuck forever
in the system?

Is there any solution to work around this issue ? Or do I have to start
over on a clean setup ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1712#issuecomment-21935815
.

adamgeorgiou.com

[JimminiKin]

Oh yes, thanks !

Recreating the percolators and deleting them one by one also worked.

[martijnvg]

When a type is being deleted, under the hood a delete by query is executed to delete all document that have the type you are deleting. Unfortunately the percolator doesn't support delete by query (only normal deletes) and therefore the loaded queries aren't updated.

[cjbottaro]

When will the percolator support delete by query?

[martijnvg]

In order to update the percolator queries in realtime when a delete by query gets executed, the delete by query needs to be implemented differently. Not sure when this will be done.

The new percolator in master (soon to be 1.0 beta1), does update the in memory percolate queries in realtime when a percolate index or the _percolator type of a percolate index gets removed. (it doesn't rely on delete by query to do this)

[clintongormley]

The old percolator has been removed. Closing

[matthuhiggins]

I can still reproduce this problem with the new percolator on v1.2.1. The registered percolators are stuck in the registry, even after issuing a delete query.

curl -XPUT 'localhost:9200/my-index/.percolator/1' -d '{
    "query" : {
        "match" : {
            "message" : "bonsai tree"
        }
    }
}'

At this point, the following returns '1' in the hits:

curl -XGET 'localhost:9200/my-index/.percolator/_search' -d '{
    "query" : {
        "match_all" : {}
    }
}'

After issuing a delete all query, the percolator no longer shows up in the search:

curl -XDELETE 'localhost:9200/my-index/.percolator/_query' -d '{
    "query" : {
        "match_all" : {}
    }
}'

curl -XGET 'localhost:9200/my-index/.percolator/_search' -d '{
    "query" : {
        "match_all" : {}
    }
}'

However, the percolator continues to show up as a result in percolate runs. The following returns the percolator with id '1':

curl -XGET 'localhost:9200/my-index/message/_percolate' -d '{
    "doc" : {
        "message" : "A new bonsai tree in the office"
    }
}'

[clintongormley]

Yeah sorry - closed this without testing. Delete by query doesn't work with percolators: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-percolate.html#_important_notes
Or at least it works, but only deregisters percolators once the index has been closed.

I'll reopen

[clintongormley]

Blocked by #2230

[clintongormley]

Closing in favour of #7052

[donwalrus]

has there been a fix proposed for this, or just to avoid deleting percolators with a filter query?

version: {
number: 1.1.1
build_hash: f1585f0
build_timestamp: 2014-04-16T14:27:12Z
build_snapshot: false
lucene_version: 4.7
}