Elasticsearch 7.2 throws an error he length of [content] field of [files:61364] doc of [fulltextsearch] index has exceeded [1000000]

[e-alfred]

Elasticsearch 7.2 throws the following error:

# sudo -u www-data php occ fulltextsearch:search user searchterm
search

In Connection.php line 620:
                                                                                                                                                                                                                                       
  {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The length of [content] field of [files:61364] doc of [fulltextsearch] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This   
  maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!"}],"type":"search_phase_execution_exception","reason":"all sha  
  rds failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"fulltextsearch","node":"KVV5wbadRSm-Q1VrAZJWwQ","reason":{"type":"illegal_argument_exception","reason":"The length of [content] field of [files:613  
  64] doc of [fulltextsearch] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, ind  
  exing with offsets or term vectors is recommended!"}}],"caused_by":{"type":"illegal_argument_exception","reason":"The length of [content] field of [files:61364] doc of [fulltextsearch] index has exceeded [1000000] - maximum all  
  owed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!","caused_by":{"type  
  ":"illegal_argument_exception","reason":"The length of [content] field of [files:61364] doc of [fulltextsearch] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing  
   the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!"}}},"status":400}                                                                             
                                                                                                                                                                                                                                    

I could fix this by applying the following Curl command:

curl -XPUT "localhost:9200/fulltextsearch/_settings" -H 'Content-Type: application/json' -d' {
    "index" : {
        "highlight.max_analyzed_offset" : 60000000
    }
}
'

[gruentee]

I experienced the same error message and also fixed it by setting a higher value to highlight.max_analyzed_offset.

For large texts, indexing with offsets or term vectors is recommended!

I'm not too much into Elastic search but is there a way to change the mapping / index configuration so as to use term vectors? Or are they already used as suggested by the mapping?

[shmakovpn]

https://www.elastic.co/guide/en/elasticsearch/reference/current/term-vector.html
I changed all of 'term_vector' => 'yes' in apps/fulltextsearch_elasticsearch/lib/Service/IndexMappingService.php and in apps/fulltextsearch_elasticsearch/vendor/elasticsearch/elasticsearch/docs/index-operations.asciidoc to: 'term_vector' => 'with_positions_offsets'.
then, occ fulltextsearch:reset
then, occ fulltextsearch:index
It works!

[prolibre]

I have the same problem. I index a lot of pdf/images.
Indexing works but I have the same error when I do a search.
I tried the solution of @shmakovpn but it doesn't work for me : : I always have the same error.
On the other hand, if I modify the highlight.max_analyzed_offset it works.

Anthony

[zjxlinux]

@prolibre you can disable highlighting completely to get rid of this error message by disabling the advanced setting, doc_table:highlight (in kibana).

[Irillit]

As far as I know, Kibana could search several indexes started with specific prefix.
Can we somehow split the old index and search the key there with this app?

[balusarakesh]

Hi,
I'm also seeing the same issue, the default value for highlight.max_analyzed_offset was 100000, I changed it to 1000000 and I still see the same error.

"Caused by: java.lang.IllegalArgumentException: The length of [message.keyword] field of [Nrc0SnIBqCgzdQbvihai] doc of [filebeat-7.3.1-2020.05.25] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!",
{"type": "server", "timestamp": "2020-05-25T15:42:14,930Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-es-data-2", "message": "[619863] Failed to execute fetch phase", "cluster.uuid": "5AW4UhbMR1u4awVsVXeCRA", "node.id": "YVLJDQL-RfKOmhd6gBLgow" , 

Not really sure how to fix this issue.

• Thank you

[shmakovpn]

Hi,
I'm also seeing the same issue, the default value for highlight.max_analyzed_offset was 100000, I changed it to 1000000 and I still see the same error.

"Caused by: java.lang.IllegalArgumentException: The length of [message.keyword] field of [Nrc0SnIBqCgzdQbvihai] doc of [filebeat-7.3.1-2020.05.25] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!",
{"type": "server", "timestamp": "2020-05-25T15:42:14,930Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-es-data-2", "message": "[619863] Failed to execute fetch phase", "cluster.uuid": "5AW4UhbMR1u4awVsVXeCRA", "node.id": "YVLJDQL-RfKOmhd6gBLgow" , 

Not really sure how to fix this issue.

• Thank you

Please check your index settings:
curl http://your.elasticsearch.server:9200/your_index/ 2>/dev/null | perl -MJSON -e 'print(JSON->new->ascii->pretty->encode(decode_json(<>)))' | grep term_vector

A result should be:
"term_vector" : "with_positions_offsets", "term_vector" : "with_positions_offsets", "term_vector" : "with_positions_offsets"

Please look at the commit: 5256e3d
where the default highlight limit problem was solved

[Zhang21]

Elasticsearch 7.2 throws the following error:

# sudo -u www-data php occ fulltextsearch:search user searchterm
search

In Connection.php line 620:
                                                                                                                                                                                                                                       
  {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The length of [content] field of [files:61364] doc of [fulltextsearch] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This   
  maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!"}],"type":"search_phase_execution_exception","reason":"all sha  
  rds failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"fulltextsearch","node":"KVV5wbadRSm-Q1VrAZJWwQ","reason":{"type":"illegal_argument_exception","reason":"The length of [content] field of [files:613  
  64] doc of [fulltextsearch] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, ind  
  exing with offsets or term vectors is recommended!"}}],"caused_by":{"type":"illegal_argument_exception","reason":"The length of [content] field of [files:61364] doc of [fulltextsearch] index has exceeded [1000000] - maximum all  
  owed to be analyzed for highlighting. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!","caused_by":{"type  
  ":"illegal_argument_exception","reason":"The length of [content] field of [files:61364] doc of [fulltextsearch] index has exceeded [1000000] - maximum allowed to be analyzed for highlighting. This maximum can be set by changing  
   the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!"}}},"status":400}                                                                             
                                                                                                                                                                                                                                    

I could fix this by applying the following Curl command:

curl -XPUT "localhost:9200/index-name/_settings" -H 'Content-Type: application/json' -d' {
    "index" : {
        "highlight.max_analyzed_offset" : 6000000
    }
}
'

This can resolve problem, but not recommended to modify the configuration. It will cause kibana and es memory problem.

If you take this way, do not change this number is too large. The reason of the problem is the message is too large.

[Ark74]

IIRC this has been fixed long time ago.
Or is this very similar?