[Console] Console with better SQL support #51446

jloleysens · 2019-11-22T14:46:43Z

Summary

The idea behind this PoC is to investigate the amount of work it would take to up the SQL game in Console. This is quite rough around the edges still but could be a good way to get a conversation going around the best approach we could take for SQL in Console. Should it be a separate editor? Would we want to do the same thing with Painless (syntax highlighting supported)?

Added support for SQL highlighting and better SQL request completion.

Things added:

~~SQL highlighting (note the addition of """sql syntax)~~
SQL highlighting for only the POST _sql endpoint
Better request expansion (this is request level as opposed to just body level expansion). This won't make sense for all endpoints but for something like SQL could be a nice fit.

Resources

Proposed syntax

POST _sql?format=json
{
"query":
    """sql
    SELECT * FROM "TABLE"
    """
}

Completion with sql markers

[UPDATE] Alternative, w/o SQL markers this is the option used in the end

Given that the addition of SQL markers introduces new syntax an alternative is to use the _sql endpoint as our marker for when to add SQL syntax highlighting.

Updated w/o sql markers

Future work

It would be good to implement language markers in the future, this PoC demonstrated that it will not be too hard to do, we should just approach it as it's own unit of work. We'll need markers for SQL + Painless.

Release notes

Added improved autocomplete expansion for an entire request which is now available on the SQL endpoint in Console.

Added SQL syntax highlighting for the SQL endpoint in Console.

elasticmachine · 2019-11-22T14:46:46Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

elasticmachine · 2019-11-22T16:10:00Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: fd7cde0

cjcenizal · 2019-12-03T18:42:55Z

Love the addition of SQL syntax highlighting! I am on the fence regarding the use of Github-style language identifiers. On one hand it requires users to learn a completely new syntax feature. On the other hand it would allow us to add support for additional syntaxes in the future (e.g. Painless, XML). I need to think about this some more.

jloleysens · 2019-12-04T11:59:18Z

@cjcenizal I hear what you're saying re the introduction of the new syntax.

The latest commit uses the Ace highlighter stack to determine when it's inside of a body where query: """ should turn on SQL highlighting. I think there are good things and less good things about this approach:

Less good things

At an implementation level I think this is a less good solution for the following reasons:

We are adding highlighting behaviour to one very specific endpoint. We need to preserve regular highlighting until we reach a marker (in our case "query": """). This creates a fork point and means that rules are partially duplicated (we can use code to duplicate them but I think that would make the current rules less clear to readers).
SQL syntax highlighting is no longer modular
- If in future we want to introduce syntax validation of SQL we'll have to create a similar thing parser/worker side where the worker needs to know about the "sql" endpoint
- Do users ever want SQL syntax highlighting somewhere else? If the answer is yes, this solution puts more freight on the lexer state
We are adding to our investment in how the Ace syntax highlighter works. This just means porting over to something like Monarch (used in Monaco) in future may be more tricksy.

Good things

Users don't need to learn anything new (yey!). As you pointed out this is quite a big plus and makes this approach the path of least resistance i.t.o. of releasing this work.
We don't need to support stripping out and re-adding of syntax markers (at the moment our auto indent functionality will need to be updated to support removing and/or re-adding """xlanguage markers (this may be a non-issue if we don't care about this specific case)

The """x markers route

Similar to <script></script>, <style></style> tags in HTML we will have a very straightforward way of adding support for new highlighting to Console. Possibly also syntax validation.
Painless highlighting should get a """painless marker (we can still support the legacy behaviour here, but it is important to be consistent).
We avoid some of the tech debt I identified above.

elasticmachine · 2019-12-04T13:06:05Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 468eff2

cjcenizal · 2019-12-04T23:22:51Z

@jloleysens Great analysis, I agree with all of your points. I'm starting to lean towards the language markers approach. Could we enhance it in the future with hardcoded rules that identify where SQL, Painless, etc are expected and logic that reformats the strings with the language marker automatically applied?

The scenario I have in mind is one in which someone copies and pastes a request from a blog post or docs -- it's unlikely that this request will already be using the triple quoted strings and language markers, so it would be a nice assist to the user if they could have the appropriate syntax highlighting applied without having to reformat things manually.

jloleysens · 2019-12-05T15:01:54Z

Could we enhance it in the future with hardcoded rules that identify where SQL, Painless, etc are expected and logic that reformats the strings with the language marker automatically applied?

That sounds like a great idea for an enhancement and I think the scenario you are describing is going to happen very frequently!

I'll keep some of the fixes to the use of templates from the latest commit and mix in the logic for language markers which we can bundle with the introduction of SQL syntax highlighting.

- no flow :c

elasticmachine · 2019-12-05T17:03:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 2d585ee

This reverts commit 2d585ee.

This reverts commit f019616.

jloleysens · 2019-12-06T09:56:03Z

@elasticmachine merge upstream

elasticmachine · 2019-12-06T11:14:36Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 90abda2

jloleysens · 2019-12-06T11:51:43Z

@elasticmachine merge upstream

jloleysens · 2019-12-09T14:57:36Z

@pcsanwald Would you mind taking another look?

pcsanwald · 2019-12-09T16:16:54Z

Looks better, thank you!

I wouldn't mind if the unsupported color was a little more differentiated, but would defer to style guide (do we have one for this sort of syntax highlighting?)

One additional note: if we do add/remove SQL keywords, we'll need to keep our dialect highlighting up to date. I suppose the risk of adding/changing keywords is going to lessen over time, but I do want to be sure that we coordinate with SQL folks on the maintenance necessary going forward for adding this functionality.

jloleysens · 2019-12-09T16:51:32Z

@pcsanwald in the case of LEFT specifically it's listed under functions in the docs which is why it has that purple color: https://www.elastic.co/guide/en/elasticsearch/reference//current/sql-functions-string.html#sql-functions-string-left

I don't think it's the same as ANSII SQL's Left - might be mistaken. But getting some design feedback should definitely be a follow up PR imo.

One additional note: if we do add/remove SQL keywords, we'll need to keep our dialect highlighting up to date

I agree that we should keep a very close eye on the number of things we add to our plate that require manual checkup and updating. In this specific case I think we are fairly safe as changes to the SQL dialect are probably less frequent - @costin could you weigh in here?

costin · 2019-12-09T20:53:16Z

LEFT and RIGHT are overloaded because they can appear as keywords (LEFT/RIGHT JOIN) but also as (string) functions LEFT/RIGHT().
As we're in charge of a grammar we can definitely take responsibility of updating the list of keywords or known functions - in fact we can probably generate those automatically so the plugin can pick those definitions up.
You're right that in general, the changes are not that frequent and it's perfectly fine imo to do any updates by hand (again we can help/take care of that as well).

cjcenizal

Code looks good, just had one question about some code that looks like it might be a no-op. I tested a bit locally but I'm not very familiar with our SQL API so I'd like to see if @costin could give it a whirl and see how he feels about the syntax highlighting.

cjcenizal · 2019-12-09T23:28:05Z

...y/core_plugins/console/public/quarantined/src/autocomplete/components/url_pattern_matcher.js

@@ -54,6 +56,9 @@ export class UrlPatternMatcher {
    endpoint.methods.forEach((method) => {
      let c;
      let activeComponent = this[method].rootComponent;
+      if (endpoint.template) {
+        new FullRequestComponent(pattern + '[body]', activeComponent, endpoint.template);


I might be missing something, but it doesn't look like we're assigning this to a variable, so what is this doing? I read through the constructors in the inheritance chain and didn't see any notable side effects.

This function call results in assignment in the component hierarchy:

kibana/src/legacy/core_plugins/console/public/np_ready/lib/autocomplete/components/shared_component.js

Line 26 in 0eb4c18

parent.addComponent(this);

SharedComponent is the common ancestor to components like FullRequestComponent. It's definitely not super clear at the moment. Perhaps a readme for how the autocomplete system works could be good? Wdyt?

Ahh, I see!

Perhaps a readme for how the autocomplete system works could be good?

This would certainly clear up a lot for me, but if it's time-consuming then I'm not sure it'd be worth it. Maybe we could begin with a brief outline of the roles of each component in the component hierarchy?

...lugins/console/public/quarantined/src/sense_editor/mode/elasticsearch_sql_highlight_rules.ts

src/legacy/core_plugins/console/np_ready/public/legacy.ts

bpintea · 2019-12-10T14:29:52Z

I'll try to give it go by EoW. If there's a snapshot build of this branch, though, that would speed up the process.

…le-sql-highlighting * 'master' of github.com:elastic/kibana: (56 commits) Migrate url shortener service (elastic#50896) Re-enable datemath in from/to canvas timelion args (elastic#52159) [Logs + Metrics UI] Remove eslint exceptions (elastic#50979) [Logs + Metrics UI] Add missing headers in Logs & metrics (elastic#52405) [ML] API integration tests - initial tests for bucket span estimator (elastic#52636) [Watcher] New Platform (NP) Migration (elastic#50908) Decouple Authorization subsystem from Legacy API. (elastic#52638) [APM] Fix some warnings logged in APM tests (elastic#52487) [ui/public/utils] Delete unused base_object & find_by_param (elastic#52500) [ui/public/utils] Move items into ui/vis (elastic#52615) fix newlines in kbn-analytics build script Add top level examples folder and command to run, `--run-examples`. (elastic#52027) feat(NA): add trap for SIGINT in the git precommit hook (elastic#52662) [DOCS] Updtes description of elasticsearch.requestHeadersWhitelist (elastic#52675) [Telemetry/Pulse] Updates advanced settings text for usage data (elastic#52657) [SIEM][Detection Engine] Adds the default name space to the end of the signals index [Logs UI] Generalize ML module management (elastic#50662) Removing stateful saved object finder (elastic#52166) Shim oss telemetry (elastic#51168) [Reporting/Screenshots] Do not fail the report if request is aborted (elastic#52344) ... # Conflicts: # src/legacy/core_plugins/console/public/legacy.ts # src/legacy/core_plugins/console/public/np_ready/application/models/legacy_core_editor/mode/elasticsearch_sql_highlight_rules.ts # src/legacy/core_plugins/console/public/np_ready/lib/autocomplete/components/full_request_component.ts # src/legacy/core_plugins/console/public/quarantined/src/sense_editor/row_parser.js

astefan · 2019-12-12T00:20:25Z

It's great to see progress on the UI side for SQL, thank you for doing this.

Regarding the SQL highlighting, does it make sense to highlight other parts of a query, as well? For example, numbers to be in different color, same goes for string constants (text surrounded by single quotes ''). And colors to be a bit more differentiated? The snippet below comes from DBeaver

vs

jloleysens · 2019-12-13T12:03:11Z

@astefan Thanks for taking a look! I don't think I'm the best person for picking the final colors, but I did see there is a pallet of color overrides for ace so I just made brought the SQL highlighting (i.t.o. relative colours) more in line with the image you sent.

Perhaps the best way to proceed would be to merge this work in (if we're happy with the current tokens that are being highlighted) and get design to do another pass where colours are tweaked.

elasticmachine · 2019-12-13T14:59:47Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 3bb08f3

History

💚 Build #15370 succeeded 3bb08f3
💚 Build #14793 succeeded 34c9684
💚 Build #14229 succeeded 316aa27

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

bpintea · 2019-12-13T15:39:32Z

@jloleysens thanks again for this effort, it looks great!

I'm leaving a few comments to potentially be picked up by next iteration. (Feel free to ignore any or all, I'm sure some of these have already been considered.)

I'm wondering if we could have autocompletion for the other request params as well, besides query?
Currently, the double-quoted items (fields/tables) are coloured as the single-quoted items (constants). It might be good to use different colouring. I.e. .. WHERE "field"='value' ..
The COUNT function is recognised and coloured, but not any other function. It'd be great to consider the others too, if possible. An example with cast:
In my (irrelevant?) browser (Opera), after clicking on the offer for completion for the query parameter, another offer is being made, which would lead to an invalid request.

cjcenizal · 2019-12-13T17:49:37Z

In my (irrelevant?) browser (Opera)

Thanks for putting a smile on my face this morning @bpintea. 😄

jloleysens · 2019-12-14T12:24:00Z

Thanks for the feedback @bpintea! The highlighting you mentioned (cast and single vs double quote) was addressed in one of the last commits before merging :). We can look into the other params! Our definitions may be outdated! The completion behavior you noted is a tricksy one. The suggestions pop up as the cursor moves around but the expansions are “dumb” in the sense that they are just snippets. We can neaten this up for sure - it may be the case for all completions at the moment. Will have to take a look!

…

On 13 Dec 2019, at 18:49, CJ Cenizal ***@***.***> wrote: In my (irrelevant?) browser (Opera) Thanks for putting a smile on my face this morning @bpintea. 😄 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

* First iteration of PoC for SQL highlighting and better completion * Alternate implementation (no """sql markers) * xLang markers * - Remove 'start-sql' - no flow :c * Revert "- Remove 'start-sql'" This reverts commit 2d585ee. * Revert "xLang markers" This reverts commit f019616. * Revert "xLang markers" and add comment This reverts commit f019616. * Add elasticsearch sql highlight rules * Add links to sources of data * Update colors * Redo built in functions # Conflicts: # src/legacy/core_plugins/console/public/np_ready/application/models/legacy_core_editor/mode/input_highlight_rules.js # src/legacy/core_plugins/console/public/np_ready/application/models/legacy_core_editor/mode/x_json_highlight_rules.js

…52893 * '7.x' of github.com:elastic/kibana: [Console] Fix OSS build (elastic#53885) (elastic#54094) [Console] Console with better SQL support (elastic#51446) (elastic#54091) Fix suggested value for time_zone in range query (elastic#53841) (elastic#54092) [APM] Show errors on the timeline instead of under the transaction (elastic#53756) (elastic#54109) use NP deprecations in uiSettings (elastic#53755) (elastic#54009) adding message to transaction and span metadata (elastic#54017) (elastic#54090) # Conflicts: # x-pack/legacy/plugins/console_extensions/spec/overrides/sql.query.json

First iteration of PoC for SQL highlighting and better completion

fd7cde0

jloleysens added release_note:enhancement discuss Feature:Console Dev Tools Console Feature Feature:Dev Tools v8.0.0 Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more labels Nov 22, 2019

jloleysens requested review from cjcenizal and jethr0null November 22, 2019 14:46

Alternate implementation (no """sql markers)

468eff2

xLang markers

f019616

jloleysens marked this pull request as ready for review December 5, 2019 15:25

- Remove 'start-sql'

2d585ee

- no flow :c

jloleysens added 3 commits December 6, 2019 10:48

Revert "- Remove 'start-sql'"

06a626e

This reverts commit 2d585ee.

Revert "xLang markers"

b7cd1bb

This reverts commit f019616.

Revert "xLang markers" and add comment

ef41dc9

This reverts commit f019616.

jloleysens changed the title ~~[Console][PoC] Console with better SQL support~~ [Console] Console with better SQL support Dec 6, 2019

jloleysens removed the discuss label Dec 6, 2019

Merge branch 'master' into feature/console-sql-highlighting

90abda2

Merge branch 'master' into feature/console-sql-highlighting

316aa27

cjcenizal approved these changes Dec 9, 2019

View reviewed changes

jloleysens added 2 commits December 11, 2019 15:09

Add links to sources of data

34c9684

Update colors

39f738b

Redo built in functions

3bb08f3

jloleysens mentioned this pull request Dec 13, 2019

Review SQL syntax highlighting in Console #52985

Open

jloleysens merged commit 96fb2d1 into elastic:master Dec 13, 2019

jloleysens deleted the feature/console-sql-highlighting branch December 13, 2019 13:39

jloleysens removed the request for review from jethr0null December 13, 2019 13:44

jloleysens mentioned this pull request Jan 7, 2020

[7.x] [Console] Console with better SQL support (#51446) #54091

Merged

jloleysens added the v7.6.0 label Jan 17, 2020

This was referenced Apr 22, 2021

Autocomplete in dev tools console for SQL endpoints #20570

Closed

SQL interface in dev tools #20580

Closed

yuliacech mentioned this pull request Apr 3, 2023

[Canvas][Data source] Use a text editor input for the ES SQL data source #40907

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Console] Console with better SQL support #51446

[Console] Console with better SQL support #51446

jloleysens commented Nov 22, 2019 •

edited

Loading

elasticmachine commented Nov 22, 2019

elasticmachine commented Nov 22, 2019

cjcenizal commented Dec 3, 2019

jloleysens commented Dec 4, 2019

elasticmachine commented Dec 4, 2019

cjcenizal commented Dec 4, 2019

jloleysens commented Dec 5, 2019

elasticmachine commented Dec 5, 2019

jloleysens commented Dec 6, 2019

elasticmachine commented Dec 6, 2019

jloleysens commented Dec 6, 2019

jloleysens commented Dec 9, 2019

pcsanwald commented Dec 9, 2019

jloleysens commented Dec 9, 2019 •

edited

Loading

costin commented Dec 9, 2019

cjcenizal left a comment

cjcenizal Dec 9, 2019

jloleysens Dec 11, 2019

cjcenizal Dec 11, 2019

bpintea commented Dec 10, 2019

astefan commented Dec 12, 2019

jloleysens commented Dec 13, 2019 •

edited

Loading

elasticmachine commented Dec 13, 2019

bpintea commented Dec 13, 2019

cjcenizal commented Dec 13, 2019

jloleysens commented Dec 14, 2019 via email

[Console] Console with better SQL support #51446

[Console] Console with better SQL support #51446

Conversation

jloleysens commented Nov 22, 2019 • edited Loading

Summary

Things added:

Resources

Proposed syntax

[UPDATE] Alternative, w/o SQL markers this is the option used in the end

Future work

Release notes

elasticmachine commented Nov 22, 2019

elasticmachine commented Nov 22, 2019

💔 Build Failed

cjcenizal commented Dec 3, 2019

jloleysens commented Dec 4, 2019

Less good things

Good things

The """x markers route

elasticmachine commented Dec 4, 2019

💚 Build Succeeded

cjcenizal commented Dec 4, 2019

jloleysens commented Dec 5, 2019

elasticmachine commented Dec 5, 2019

💚 Build Succeeded

jloleysens commented Dec 6, 2019

elasticmachine commented Dec 6, 2019

💔 Build Failed

jloleysens commented Dec 6, 2019

jloleysens commented Dec 9, 2019

pcsanwald commented Dec 9, 2019

jloleysens commented Dec 9, 2019 • edited Loading

costin commented Dec 9, 2019

cjcenizal left a comment

Choose a reason for hiding this comment

cjcenizal Dec 9, 2019

Choose a reason for hiding this comment

jloleysens Dec 11, 2019

Choose a reason for hiding this comment

cjcenizal Dec 11, 2019

Choose a reason for hiding this comment

bpintea commented Dec 10, 2019

astefan commented Dec 12, 2019

jloleysens commented Dec 13, 2019 • edited Loading

elasticmachine commented Dec 13, 2019

💚 Build Succeeded

History

bpintea commented Dec 13, 2019

cjcenizal commented Dec 13, 2019

jloleysens commented Dec 14, 2019 via email

jloleysens commented Nov 22, 2019 •

edited

Loading

jloleysens commented Dec 9, 2019 •

edited

Loading

jloleysens commented Dec 13, 2019 •

edited

Loading