Error while loading data

[Kenzi007220]

Hi, we are using helm chart to install app using rancher over K8s
Once we started application we are not able to grab data from elasticsearch. We are not using currently security params and Kibana is able to connect

We are using ELK stack v7.7

Can you please help?

[Jeoffreybauvin]

Same issue here with Cerebro 0.9.1 and Elasticsearch 5.6.6 :

[2020-06-04T14:47:40.030+02:00][error][application              ] Error processing request [path: /cerebro/nodes, body: {"password":"xxxxxx","host":"elasticsearch","username":"jbauvin"}]
play.api.libs.json.JsResultException: JsResultException(errors:List((,List(JsonValidationError(List('attributes' is undefined on object: {"name":"node-002","transport_address":"10.2.231.40:9300","host":"10.2.231.40","ip":"10.2.231.40","version":"5.6.8","build_hash":"688ecce","roles":["master","data"],"os":{"refresh_interval":"1s","refresh_interval_in_millis":1000,"name":"Linux","arch":"amd64","version":"3.16.0-7-amd64","available_processors":3,"allocated_processors":3},"jvm":{"pid":126057,"version":"1.8.0_60","vm_name":"Java HotSpot(TM) 64-Bit Server VM","vm_version":"25.60-b23","vm_vendor":"Oracle Corporation","start_time":"2019-03-29T14:51:37.796Z","start_time_in_millis":1553871097796,"mem":{"heap_init":"1.7gb","heap_init_in_bytes":1853882368,"heap_max":"1.7gb","heap_max_in_bytes":1827733504,"non_heap_init":"2.4mb","non_heap_init_in_bytes":2555904,"non_heap_max":"0b","non_heap_max_in_bytes":0,"direct_max":"1.7gb","direct_max_in_bytes":1827733504},"gc_collectors":["ParNew","ConcurrentMarkSweep"],"memory_pools":["Code Cache","Metaspace","Compressed Class Space","Par Eden Space","Par Survivor Space","CMS Old Gen"],"using_compressed_ordinary_object_pointers":"true","input_arguments":["-Dfile.encoding=UTF-8","-Dio.netty.noKeySetOptimization=true","-Dio.netty.noUnsafe=true","-Dio.netty.recycler.maxCapacityPerThread=0","-Djava.awt.headless=true","-Djna.nosys=true","-Dlog4j.shutdownHookEnabled=false","-Dlog4j2.disable.jmx=true","-XX:+AlwaysPreTouch","-XX:+HeapDumpOnOutOfMemoryError","-XX:+PrintGCApplicationStoppedTime","-XX:+PrintGCDateStamps","-XX:+PrintGCDetails","-XX:+PrintTenuringDistribution","-XX:+UseCMSInitiatingOccupancyOnly","-XX:+UseConcMarkSweepGC","-XX:+UseGCLogFileRotation","-XX:-OmitStackTraceInFastThrow","-XX:CMSInitiatingOccupancyFraction=75","-XX:GCLogFileSize=64m","-XX:NumberOfGCLogFiles=32","-Xloggc:/var/log/elasticsearch/node-002/gc.log","-Xms1768m","-Xmx1768m","-Xss1m","-Des.path.home=/usr/share/elasticsearch"]}}),WrappedArray())))))
	at play.api.libs.json.JsReadable.$anonfun$as$2(JsReadable.scala:25)
	at play.api.libs.json.JsError.fold(JsResult.scala:66)
	at play.api.libs.json.JsReadable.as(JsReadable.scala:24)
	at play.api.libs.json.JsReadable.as$(JsReadable.scala:23)
	at play.api.libs.json.JsUndefined.as(JsLookup.scala:193)

[mirekn]

It does not seem to be similar. @Kenzi007220 forget to mention the error detail:

Error while loading data

{
  "error": "Failure for [_nodes/stats/jvm,fs,os,process?human=true]"
}

Related source code:

cerebro/app/services/overview/OverviewDataService.scala

Line 18 in fd046ee

"_nodes/stats/jvm,fs,os,process?human=true",

Elasticsearch was installed from the official Helm chart
https://hub.helm.sh/charts/elastic/elasticsearch/7.7.0

Thanks

[Kenzi007220]

@mirekn Thanks for the feedback.

• "GET _nodes/stats/jvm,fs,os,process?human=true" throw error:

{ "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "Values less than -1 bytes are not supported: -8584949760b" } ], "type" : "illegal_argument_exception", "reason" : "Values less than -1 bytes are not supported: -8584949760b", "suppressed" : [ { "type" : "illegal_state_exception", "reason" : "Failed to close the XContentBuilder", "caused_by" : { "type" : "i_o_exception", "reason" : "Unclosed object or array found" } } ] }, "status" : 400 }

• "GET /_nodes/_all/jvm,os?human=true" - working fine.

[consulthys]

It is related to this bug: elastic/elasticsearch#57312
it will be fixed in 7.7.2: elastic/elasticsearch#57718

[lmenezes]

@consulthys thanks for looking in to this.

As this is an ES issue rather than cerebro, will just close this.

[jamshid]

Is there any workaround (seeing this on 7.5.2) other than upgrading to 7.7.2 (which breaks an older client with unknown field [disable_coord])?

[lmenezes]

@jamshid Unfortunately there is no workaround. Doing so would mean trying to make things work without being able to fetch a lot of the data needed, and although possible, the end result would not be great and the effort needed to make this work is not worthwhile.