Can't sort on scripted date or boolean fields #9257

Bargs · 2016-11-29T20:42:45Z

Kibana version: 5.0+
Elasticsearch version: 5.0+
Description of the problem including expected versus actual behavior:

Sorting on a scripted date or boolean field in Discover results in an error.

This is because the ES script based sorting only supports types number and string.

Painless scripted fields can be numbers, strings, dates or booleans. Dates need to be returned as millis since the epoch for visualizations to work, so we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we send them as type string, so we can add a similar condition there as well.

The text was updated successfully, but these errors were encountered:

The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: elastic#9257

The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: #9257

Backports PR #9261 **Commit 1:** Fix sorting on scripted date and boolean fields The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: #9257 * Original sha: 0d5a824 * Authored by Matthew Bargar <[email protected]> on 2016-11-29T21:05:48Z **Commit 2:** Add helpful comment * Original sha: 14dc396 * Authored by Matthew Bargar <[email protected]> on 2016-11-30T21:02:41Z

Also clean up and simplify scope destroying Fix sorting on scripted date and boolean fields (elastic#9261) The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: elastic#9257 Add docs on all available console server config options (elastic#9288) settings: do not query ES for settings in non-green status (elastic#9308) If the ui settings status is not green, that means there is at least one dependency (so elasticsearch at the moment) that is not met in order for it to function correctly, so we shouldn't attempt to determine user settings at all. This ensures that when something like the version check fails in the elasticsearch plugin, Kibana correctly behaves by not attempting requests to elasticsearch, which prevents 500 errors and allows users to see the error status on the status page. We also now periodically check for compatible elasticsearch versions so that Kibana can automatically recover if the elasticsearch node is upgraded to the appropriate version. Change loading screen background to white to make it less distracting when navigating between apps. (elastic#9313) more refactoring Fix sorting on scripted date and boolean fields (elastic#9261) The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: elastic#9257 Add docs on all available console server config options (elastic#9288) settings: do not query ES for settings in non-green status (elastic#9308) If the ui settings status is not green, that means there is at least one dependency (so elasticsearch at the moment) that is not met in order for it to function correctly, so we shouldn't attempt to determine user settings at all. This ensures that when something like the version check fails in the elasticsearch plugin, Kibana correctly behaves by not attempting requests to elasticsearch, which prevents 500 errors and allows users to see the error status on the status page. We also now periodically check for compatible elasticsearch versions so that Kibana can automatically recover if the elasticsearch node is upgraded to the appropriate version. Change loading screen background to white to make it less distracting when navigating between apps. (elastic#9313) Skip assertion when the test blips. (elastic#9251) Tests fails intermittently, and for identical reasons. When this occurs, skip the test. This only occurs in CI and cannot be reproduced locally. Additional changes: - Use async/await to improve legibility. - Move async behaviour to beforeEach and keep test stubs synchronous to improve labeling of tests. [git] ignore extra files in the root config/ directory (elastic#9296) upgrade makelogs (elastic#9295) build: remove deepModules hackery (elastic#9327) The deepModules hacks in the build system were added to support the long paths that resulted from npm2, but npm3 fundamentally addresses that problem, so deepModules is no longer necessary. In practical terms, npm3 shouldn't ever cause path lengths to become so long that they trigger path length problems on certain operating systems. more cleanup and tests Save/rename flow fix (elastic#9087) * Save/rename flow fix - Use auto generated ids instead of coupling the id to the title which creates problems. - Adjust UI to make the save flow more understandable. - Remove confirmation on overwrite since we will now be creating duplicate objects even if they have the same title. * use undefined instead of null * Change titleChanged function name * address code review comments * Add isSaving flag to avoid checkbox flicker, fix regression bug from refactor. Added tests and fixed a couple bugs Updated info message * Update doc and nav title with new name on rename don't hardcode Dashboard

* make scope isolate * Make panel scope isolate Also clean up and simplify scope destroying Fix sorting on scripted date and boolean fields (#9261) The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: #9257 Add docs on all available console server config options (#9288) settings: do not query ES for settings in non-green status (#9308) If the ui settings status is not green, that means there is at least one dependency (so elasticsearch at the moment) that is not met in order for it to function correctly, so we shouldn't attempt to determine user settings at all. This ensures that when something like the version check fails in the elasticsearch plugin, Kibana correctly behaves by not attempting requests to elasticsearch, which prevents 500 errors and allows users to see the error status on the status page. We also now periodically check for compatible elasticsearch versions so that Kibana can automatically recover if the elasticsearch node is upgraded to the appropriate version. Change loading screen background to white to make it less distracting when navigating between apps. (#9313) more refactoring Fix sorting on scripted date and boolean fields (#9261) The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: #9257 Add docs on all available console server config options (#9288) settings: do not query ES for settings in non-green status (#9308) If the ui settings status is not green, that means there is at least one dependency (so elasticsearch at the moment) that is not met in order for it to function correctly, so we shouldn't attempt to determine user settings at all. This ensures that when something like the version check fails in the elasticsearch plugin, Kibana correctly behaves by not attempting requests to elasticsearch, which prevents 500 errors and allows users to see the error status on the status page. We also now periodically check for compatible elasticsearch versions so that Kibana can automatically recover if the elasticsearch node is upgraded to the appropriate version. Change loading screen background to white to make it less distracting when navigating between apps. (#9313) Skip assertion when the test blips. (#9251) Tests fails intermittently, and for identical reasons. When this occurs, skip the test. This only occurs in CI and cannot be reproduced locally. Additional changes: - Use async/await to improve legibility. - Move async behaviour to beforeEach and keep test stubs synchronous to improve labeling of tests. [git] ignore extra files in the root config/ directory (#9296) upgrade makelogs (#9295) build: remove deepModules hackery (#9327) The deepModules hacks in the build system were added to support the long paths that resulted from npm2, but npm3 fundamentally addresses that problem, so deepModules is no longer necessary. In practical terms, npm3 shouldn't ever cause path lengths to become so long that they trigger path length problems on certain operating systems. more cleanup and tests Save/rename flow fix (#9087) * Save/rename flow fix - Use auto generated ids instead of coupling the id to the title which creates problems. - Adjust UI to make the save flow more understandable. - Remove confirmation on overwrite since we will now be creating duplicate objects even if they have the same title. * use undefined instead of null * Change titleChanged function name * address code review comments * Add isSaving flag to avoid checkbox flicker, fix regression bug from refactor. Added tests and fixed a couple bugs Updated info message * Update doc and nav title with new name on rename don't hardcode Dashboard * namespace panel factory cleanup * Fix parameter name in html * Address comments - Get rid of factory function and Panel class. - Rename panel.js to panel_state.js - Rename dashboard_panel_directive to dashboard_panel * Fix file path reference in tests. * Panel => PanelState

Backports PR #9335 **Commit 1:** make scope isolate * Original sha: c54eb3f * Authored by Stacey Gammon <[email protected]> on 2016-11-30T20:55:28Z **Commit 2:** Make panel scope isolate Also clean up and simplify scope destroying Fix sorting on scripted date and boolean fields (#9261) The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: #9257 Add docs on all available console server config options (#9288) settings: do not query ES for settings in non-green status (#9308) If the ui settings status is not green, that means there is at least one dependency (so elasticsearch at the moment) that is not met in order for it to function correctly, so we shouldn't attempt to determine user settings at all. This ensures that when something like the version check fails in the elasticsearch plugin, Kibana correctly behaves by not attempting requests to elasticsearch, which prevents 500 errors and allows users to see the error status on the status page. We also now periodically check for compatible elasticsearch versions so that Kibana can automatically recover if the elasticsearch node is upgraded to the appropriate version. Change loading screen background to white to make it less distracting when navigating between apps. (#9313) more refactoring Fix sorting on scripted date and boolean fields (#9261) The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: #9257 Add docs on all available console server config options (#9288) settings: do not query ES for settings in non-green status (#9308) If the ui settings status is not green, that means there is at least one dependency (so elasticsearch at the moment) that is not met in order for it to function correctly, so we shouldn't attempt to determine user settings at all. This ensures that when something like the version check fails in the elasticsearch plugin, Kibana correctly behaves by not attempting requests to elasticsearch, which prevents 500 errors and allows users to see the error status on the status page. We also now periodically check for compatible elasticsearch versions so that Kibana can automatically recover if the elasticsearch node is upgraded to the appropriate version. Change loading screen background to white to make it less distracting when navigating between apps. (#9313) Skip assertion when the test blips. (#9251) Tests fails intermittently, and for identical reasons. When this occurs, skip the test. This only occurs in CI and cannot be reproduced locally. Additional changes: - Use async/await to improve legibility. - Move async behaviour to beforeEach and keep test stubs synchronous to improve labeling of tests. [git] ignore extra files in the root config/ directory (#9296) upgrade makelogs (#9295) build: remove deepModules hackery (#9327) The deepModules hacks in the build system were added to support the long paths that resulted from npm2, but npm3 fundamentally addresses that problem, so deepModules is no longer necessary. In practical terms, npm3 shouldn't ever cause path lengths to become so long that they trigger path length problems on certain operating systems. more cleanup and tests Save/rename flow fix (#9087) * Save/rename flow fix - Use auto generated ids instead of coupling the id to the title which creates problems. - Adjust UI to make the save flow more understandable. - Remove confirmation on overwrite since we will now be creating duplicate objects even if they have the same title. * use undefined instead of null * Change titleChanged function name * address code review comments * Add isSaving flag to avoid checkbox flicker, fix regression bug from refactor. Added tests and fixed a couple bugs Updated info message * Update doc and nav title with new name on rename don't hardcode Dashboard * Original sha: e76f638 * Authored by Stacey Gammon <[email protected]> on 2016-12-01T18:54:50Z **Commit 3:** Merge branch 'master' of https://github.com/elastic/kibana into panel-refactor-cleanup * Original sha: 82323be * Authored by Stacey Gammon <[email protected]> on 2016-12-02T15:23:56Z **Commit 4:** namespace panel factory cleanup * Original sha: 3a09d66 * Authored by Stacey Gammon <[email protected]> on 2016-12-02T15:31:09Z **Commit 5:** Fix parameter name in html * Original sha: 93afafa * Authored by Stacey Gammon <[email protected]> on 2016-12-06T14:25:51Z **Commit 6:** Address comments - Get rid of factory function and Panel class. - Rename panel.js to panel_state.js - Rename dashboard_panel_directive to dashboard_panel * Original sha: aa8d6c8 * Authored by Stacey Gammon <[email protected]> on 2016-12-06T15:27:58Z **Commit 7:** Merge branch 'master' of https://github.com/elastic/kibana into panel-refactor-cleanup * Original sha: cfd610a * Authored by Stacey Gammon <[email protected]> on 2016-12-06T16:04:56Z **Commit 8:** Fix file path reference in tests. * Original sha: 317e76e * Authored by Stacey Gammon <[email protected]> on 2016-12-06T18:11:42Z **Commit 9:** Merge branch 'master' of https://github.com/elastic/kibana into panel-refactor-cleanup * Original sha: a6288dd * Authored by Stacey Gammon <[email protected]> on 2016-12-07T14:22:36Z **Commit 10:** Panel => PanelState * Original sha: c87f45b * Authored by Stacey Gammon <[email protected]> on 2016-12-07T14:32:08Z

Backports PR elastic#9261 **Commit 1:** Fix sorting on scripted date and boolean fields The elasticsearch API only [supports][1][2] sort scripts of type `number` and `string`. Since dates need to be returned as millis since the epoch for visualizations to work anyway, we can simply add a condition to send dates as type number in the sort API. ES will cast booleans if we tell them its a string, so we can add a similar condition there as well. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-request-sort.html#_script_based_sorting [2]: https://github.com/elastic/elasticsearch/blob/aeb97ff41298e26b107a733837dfe17f123c0c9b/core/src/main/java/org/elasticsearch/search/sort/ScriptSortBuilder.java#L359 Fixes: elastic#9257 * Original sha: 0d5a824 * Authored by Matthew Bargar <[email protected]> on 2016-11-29T21:05:48Z **Commit 2:** Add helpful comment * Original sha: 14dc396 * Authored by Matthew Bargar <[email protected]> on 2016-11-30T21:02:41Z Former-commit-id: b5ae727

Bargs added :Discovery Feature:Scripted Fields Scripted fields features bug Fixes for quality problems that affect the customer experience labels Nov 29, 2016

Bargs mentioned this issue Nov 29, 2016

Fix sorting on scripted date and boolean fields #9261

Merged

Bargs closed this as completed in #9261 Nov 30, 2016

This was referenced Nov 30, 2016

[5.1] Fix sorting on scripted date and boolean fields #9285

Merged

[5.x] Fix sorting on scripted date and boolean fields #9286

Merged

elastic-jasper mentioned this issue Dec 7, 2016

[5.x] Make panel scope isolate, and some cleanup #9407

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't sort on scripted date or boolean fields #9257

Can't sort on scripted date or boolean fields #9257

Bargs commented Nov 29, 2016

Can't sort on scripted date or boolean fields #9257

Can't sort on scripted date or boolean fields #9257

Comments

Bargs commented Nov 29, 2016