Index-Pattern Export only exports URL fields

[t3chn0m4g3]

Kibana version: 7.12.1

Elasticsearch version: 7.12.1

Server OS version: Debian 10

Browser version: Latest Chrome

Browser OS version: Latest Windows 10

Original install method (e.g. download page, yum, from source, etc.): dpkg

Describe the bug: Index-Pattern should include all fields, not just the URL based fields

Steps to reproduce:

1. Login to Kibana
2. Goto Index Management > Saved Objects
3. Click on Export obejcts
4. Choose to export only index-pattern (result with all is the same, but easier to review if only index-pattern is chosen)
5. Click on Export All
6. Review downloaded NDJSON to only contain URL fields (if at all, if no URL is present file remains without fields).

With curl the same result:

curl -s -X POST 'http://127.0.0.1:5601/api/saved_objects/_export' -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d '{ "type": "index-pattern" }'

No logs show unexpected behavior, things worked as expected in previous versions (prior 7.12)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

I'm very surprised of the issue's description TBH. We're not performing any kind of transformation on the object's data when exporting, so I honestly don't see how some truncation could occur.

Could you please provide the generated output from an export , and then the result of accessing one of the faulty exported index pattern via

GET <kibana host>:<port>/api/saved_objects/index-pattern/<id>

[t3chn0m4g3]

Thanks for the feedback. Let me add some more info.

In Kibana I can see 1268 fields active in the index pattern. 25 of these fields have been manually formatted in Kibana as URL (i.e.):

Running your proposed query only the 25 fields formatted als URL will be exported:

curl -s -X GET 'http://127.0.0.1:5601/api/saved_objects/index-pattern/logstash-*' | jq
{
  "id": "logstash-*",
  "type": "index-pattern",
  "namespaces": [
    "default"
  ],
  "updated_at": "2021-05-17T15:57:40.799Z",
  "version": "WzM2OTM5LDMwXQ==",
  "attributes": {
    "fieldAttrs": "{\"id\":{\"count\":2}}",
    "fieldFormatMap": "{\"alert.cve_id.keyword\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://www.cvedetails.com/cve/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"alert.signature_id\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://doc.emergingthreats.net/bin/view/Main/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"dest_ip\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://www.talosintelligence.com/reputation_center/lookup?search={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"dest_port\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://www.speedguide.net/port.php?port={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"dns.rrname\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://www.senderbase.org/lookup/?search_string={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"event_type\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://duckduckgo.com/?q={{value}}&t=h_&ia=web\",\"labelTemplate\":\"{{value}}\"}},\"geoip.asn\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://mxtoolbox.com/SuperTool.aspx?action=asn%3a{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"geoip.city_name\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://en.wikipedia.org/w/index.php?search={{value}}&title=Special:Search&go=Go\",\"labelTemplate\":\"{{value}}\"}},\"geoip.country_name\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://en.wikipedia.org/w/index.php?search={{value}}&title=Special:Search&go=Go\",\"labelTemplate\":\"{{value}}\"}},\"geoip.ip\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://www.senderbase.org/lookup/?search_string={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"http.http_user_agent\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://ua.theafh.net/list.php?s={{value}}&include=yes&class=abr&do=desc\",\"labelTemplate\":\"{{value}}\"}},\"http.status\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://httpstatuses.com/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"http_user_agent\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://ua.theafh.net/list.php?s={{value}}&include=yes&class=abr&do=desc\",\"labelTemplate\":\"{{value}}\"}},\"link\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://duckduckgo.com/?q={{value}}&t=h_&ia=web\",\"labelTemplate\":\"{{value}}\"}},\"os\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://duckduckgo.com/?q={{value}}&t=h_&ia=web\",\"labelTemplate\":\"{{value}}\"}},\"proxy_connection.local_ip\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://www.senderbase.org/lookup/?search_string={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"proxy_connection.local_port\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://www.speedguide.net/port.php?port={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"proxy_connection.remote_ip\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://www.senderbase.org/lookup/?search_string={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"proxy_connection.remote_port\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://www.speedguide.net/port.php?port={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"src_ip\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://www.talosintelligence.com/reputation_center/lookup?search={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"src_ip.keyword\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://www.talosintelligence.com/reputation_center/lookup?search={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"src_port\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"http://www.speedguide.net/port.php?port={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"status\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://httpstatuses.com/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"tls.sni\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://www.ssllabs.com/ssltest/analyze.html?d={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"tls.version\":{\"id\":\"url\",\"params\":{\"parsedUrl\":{\"origin\":\"http://127.0.0.1:5601\",\"pathname\":\"/kibana/app/kibana\",\"basePath\":\"/kibana\"},\"urlTemplate\":\"https://duckduckgo.com/?q={{value}}&t=h_&ia=web\",\"labelTemplate\":\"{{value}}\"}}}",
    "fields": "[]",
    "runtimeFieldMap": "{}",
    "timeFieldName": "@timestamp",
    "title": "logstash-*"
  },
  "references": [],
  "migrationVersion": {
    "index-pattern": "7.11.0"
  },
  "coreMigrationVersion": "7.12.1"
}

Result is the same when exporting through Kibana GUI (Saved Obejcts) directly. If I remove all URL information and set the corresponding fields back to default there is no output of fields at all.

[Dosant]

I assume this might be related to #82223: since the 7.11 field list is loaded via field caps API and there is no field cache of es fields on index pattern object anymore.

If this is the case then this new behavior is expected and not a bug.

@mattkime, could you please confirm if my assumption is legit?

[t3chn0m4g3]

@Dosant Thanks. The explanation in #82223 makes sense.