PNG report doesn't create any report

[AlonaNadler]

Kibana version: master
I started trial license and tried to export png from a dashboard, notification that the report is in progress appeared but nothing happened beyond that. The report doesn't doesn't show up in the reporting page and I got an error in Kibana

[elasticmachine]

Pinging @elastic/kibana-app

[stacey-gammon]

Did not attempt this, but if it's a legit error I think this is a 7.0 blocker and a regression. Labeled as such.

[stacey-gammon]

cc @joelgriffith @tsullivan

[joelgriffith]

Looking into this now

[tsullivan]

Based on the error message, it looks like the jobtype field has text mapping instead of keyword.

EDIT: for reasoning, this might happening from deleting the .reporting index while Kibana is running, and then schedule a Report. The index might be getting auto-created with default mappings.

[joelgriffith]

Taking a look now

[tsullivan]

When you start Elasticsearch, you might be seeing this in the logs:

   ? info [o.e.c.m.MetaDataCreateIndexService] [spicy.local] [.reporting-2019.01.27] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []

This shows that the index is getting created from something indexing documents with the Bulk API, but there was no mapping.

This looks related to https://github.com/elastic/kibana/pull/27094/files#r241225441

This: https://github.com/tsullivan/kibana/blob/7e20835/x-pack/plugins/reporting/server/lib/esqueue/helpers/create_index.js#L95 should check for the kind of error that it is ignoring :D

[tsullivan]

When I allow an error to be thrown from that part of the code, I see this info in the error:

{"path":"/.reporting-2019.01.27","query":{},"body":"{\"settings\":{\"number_of_shards\":1,\"auto_expand_replicas\":\"0-1\"},\"mappings\":{\"esqueue\":{\"properties\":{\"meta\":{\"properties\":{\"objectType\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":256}}},\"layout\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":256}}}}},\"browser_type\":{\"type\":\"keyword\"},\"jobtype\":{\"type\":\"keyword\"},\"payload\":{\"type\":\"object\",\"enabled\":false},\"priority\":{\"type\":\"byte\"},\"timeout\":{\"type\":\"long\"},\"process_expiration\":{\"type\":\"date\"},\"created_by\":{\"type\":\"keyword\"},\"created_at\":{\"type\":\"date\"},\"started_at\":{\"type\":\"date\"},\"completed_at\":{\"type\":\"date\"},\"attempts\":{\"type\":\"short\"},\"max_attempts\":{\"type\":\"short\"},\"kibana_name\":{\"type\":\"keyword\"},\"kibana_id\":{\"type\":\"keyword\"},\"status\":{\"type\":\"keyword\"},\"output\":{\"type\":\"object\",\"properties\":{\"content_type\":{\"type\":\"keyword\"},\"size\":{\"type\":\"long\"},\"content\":{\"type\":\"object\",\"enabled\":false}}}}}}}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"Root mapping definition has unsupported parameters: [esqueue : {properties={kibana_name={type=keyword}, browser_type={type=keyword}, created_at={type=date}, priority={type=byte}, jobtype={type=keyword}, created_by={type=keyword}, timeout={type=long}, kibana_id={type=keyword}, output={type=object, properties={content_type={type=keyword}, size={type=long}, content={type=object, enabled=false}}}, process_expiration={type=date}, completed_at={type=date}, payload={type=object, enabled=false}, meta={properties={layout={type=text, fields={keyword={ignore_above=256, type=keyword}}}, objectType={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}, max_attempts={type=short}, started_at={type=date}, attempts={type=short}, status={type=keyword}}}]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [esqueue : {properties={kibana_name={type=keyword}, browser_type={type=keyword}, created_at={type=date}, priority={type=byte}, jobtype={type=keyword}, created_by={type=keyword}, timeout={type=long}, kibana_id={type=keyword}, output={type=object, properties={content_type={type=keyword}, size={type=long}, content={type=object, enabled=false}}}, process_expiration={type=date}, completed_at={type=date}, payload={type=object, enabled=false}, meta={properties={layout={type=text, fields={keyword={ignore_above=256, type=keyword}}}, objectType={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}, max_attempts={type=short}, started_at={type=date}, attempts={type=short}, status={type=keyword}}}]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"Root mapping definition has unsupported parameters: [esqueue : {properties={kibana_name={type=keyword}, browser_type={type=keyword}, created_at={type=date}, priority={type=byte}, jobtype={type=keyword}, created_by={type=keyword}, timeout={type=long}, kibana_id={type=keyword}, output={type=object, properties={content_type={type=keyword}, size={type=long}, content={type=object, enabled=false}}}, process_expiration={type=date}, completed_at={type=date}, payload={type=object, enabled=false}, meta={properties={layout={type=text, fields={keyword={ignore_above=256, type=keyword}}}, objectType={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}, max_attempts={type=short}, started_at={type=date}, attempts={type=short}, status={type=keyword}}}]\"}},\"status\":400}"}

[stacey-gammon]

Sounds like it might be related to the Elastisearch root type breaking change, Chris Davies fixed up the tests for reporting here: #29184

Maybe the actual code and mappings needs to change as well.

[joelgriffith]

Interesting... I've got stuff lined up to work once I'm through my AM meetings.

[tsullivan]

After merging #29184 locally, I'm seeing the Reporting index get created successfully when I put a job into the queue. I saw some changes go into ESQueue in that PR.

However, our doc type is esqueue and I would have thought it isn't going to work unless the type is _doc

[tsullivan]

Clarification: Based on https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html#_schedule_for_removal_of_mapping_types, in ES 7.x:

index APIs are PUT {index}/_doc/{id} in case of explicit ids and POST {index}/_doc

I'm not sure how this is going to work if our doc type is not _doc it's actually esqueue:

> x-pack/plugins/reporting/common/constants.ts:7:31:export const QUEUE_DOCTYPE = 'esqueue';
  x-pack/plugins/reporting/server/lib/esqueue/constants/default_settings.js:11:29:  DEFAULT_SETTING_DOCTYPE: 'esqueue',

While the bug seems fixed in master for now, I think we're not completely done

[joelgriffith]

@joel Griffith in 7.0, it's possible to use the 'typeless' APIs against an index that has a custom type. so even if the index has the type esqueue, using PUT {index}/_doc/{id} will work. a note that this will not work in 6.x, only 7.0

I think we're good with the change Chris made to make Kibana ES7-compliant, might need to revisit this in the 8.x series. I'll verify this is working locally and close this issue out if so

[joelgriffith]

Verified that PNG and CSV reports are working on latest (7.x.x). Closing!

[stacey-gammon]

Ah yea, probably this line: https://github.com/elastic/kibana/pull/29184/files#diff-857ec3bbdaea920d52c2d00c124c8b2bR91

is what fixed it.