Nested field support

[Alex-Ikanow]

This is sort of a duplicate of some other issues I searched for but I haven't seen this particular aspect discussed, so I thought this was worth a separate issue.

You read the _mapping field, so you should know when a particular field is nested, so can it not automatically apply the correct nested facet/query when such a field is selected in queries or facets?

(Alternatively/in addition as suggested by #532, you could have a checkbox to allow users to select it themselves, perhaps as an interim measure)

I'm sure there are some cases where this gets complicated, but there are also a bunch of cases where it's a straightforward changing of one block of JSON to another.

Latest update: #1084 (comment)

[shayts7]

+1 for nested object aggregation.

[matthewgaulin]

+1

[Charlesbot]

+1

[ghost]

+1

[bzz]

+1

[Kallin]

+1

[deepakas]

+1

[matogertel]

+1

[c0deaddict]

+1

[theNewFlesh]

+1

[krisb78]

+100000

[ddaumann]

+1

[Kallin]

to be clear, there is no way to do a nested filter/query/agg in kibana 4 right now, is there?

[Medi0ker]

+1

[ajrasch]

+11111

[calippo]

+1

[polloelastico]

+1

[Filirom1]

+1

[MichaelBorde]

+1

[vordimous]

+1

[tonymwinters]

+1

[KevinFarrell]

+1

[nbeede]

+2

[calvdee]

+1

[Sandy381]

+1

[tg295]

+1

[ebuildy]

as a quick wordaround, you could use Transformation to create new elasticsearch indices, see https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-overview.html, to flat your data to match "kibana format".

[fatihakafou]

+1

[gueri-fr]

+1

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[pokrovskyy]

My 5 cents here. IMO it is a hugely undervalued feature that is missing in Kibana for years. Without the ability to efficiently visualize & aggregate nested fields - I do not see much value in using nested fields at all. I built an app data collection into ElasticSearch using my best understanding of how nested fields can help build and store logical data structures and collected days of useful data which are pretty much useless now.

Great workaround suggestion by @mattkime to use scripted fields which I also came up to & tried - it kind of worked, but certain fields in my case are floats and for whatever reason Kibana kept exploding on them. Not to mention the need to configure them all. So no-go. Transformations not really useful too if you want to have live dashboards

Eventually, I ended up changing the incoming data format by flattening all the nested structures into a flat document. Yuck - but at least it's all usable now. Not very happy with this, but so far seems like it's the only reliable way.

BOTTOM LINE - read this and think twice before using nested fields in your ElasticSearch strategies, you might be having issues visualizing them later 😄

[mrPsycox]

Do you have news on this feature? Right now, nested fields are showed but are not aggregatable.

[ninoslavmiskovic]

In the near term, our focus is on other key initiatives. Still, we remain open to exploring nested field support in the future as we continue to assess the evolving needs of customers and users.

[subrt]

+1

[hugocorreia77]

+1

[Jzhnakui]

+1！！！！！！！！！！！！！
its import！！

[carlopuri]

+1
please add this feature.
our use case is to visualise nested information based on connected "levels" and "sub levels".
vega has worst performance and the development it's very uncomfortable.
last but not least thank you for the effort spent on improving kibana

[paul21]

It's been 10 years now. Happy birthday everyone!

[waynehamadi]

It's been 10 years now. Happy birthday everyone!

haha that's funny.
I am kind of ashamed I completely dropped the ball on this, I was going to build it but it became less of a priority at my job.

Honestly, we ended up just building reports on SQL, like everyone else, instead of using Kibana for that.

I understand why elastic doesn't do it. It's a pain in the butt to do, there is a bunch of edge cases that can appear.
But at the time, the pain was real.

[Hronom]

Ok we need to wait another 5 years to make AI handle this task.

[arkel-s]

Exactly this situation 😆 https://xkcd.com/2881/

[thomasneirynck]

A short update on this:

At the moment, there is no active effort in Lens to support nested fields.

You can use Vega as a work-around. On that note: #1084 (comment), when saying "Vega has the worst performance", are you (cc @carlopuri) referring to the time it takes to run the nested-aggregation, or the actual rendering time of Vega?

Nested-fields are a niche feature; if you need them for visualizations, and you can use an alternative method when indexing, please consider to do so. More info on alternative ways to model data can be found here https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html.

While ES|QL has currently no support for nested-fields, ES|QL has been discussed as a potential way to bring analytics on nested fields to Kibana (elastic/elasticsearch#107434).