[Canvas] Integrate with search sessions

[poffdeluxe]

Integrate Canvas's datasource functions with search sessions and pass search session ids to editor apps (Lens, Visualize) when trying to edit an embeddable from a Canvas workpad.

Potential gains to migrating to search sessions:

1. Quick loading of workpads if we can load up previous search session
2. Caching of requests (in the future) and not performing unnecessary slow queries
3. Nearly instantaneous loading of editor apps if user elects to edit an embeddable from an editor app

Background

In #94754, we migrated to the new search service provided by the data plugin to handle data retrieval. As a part of this work, we implemented a basic synchronous search strategy (search strategies just tell the search service how it should request the data from the Elasticsearch backend) for SQL. The esdocs, escount functions were migrated to use data.search (and the default ES search strategy) rather than attempt use the Elasticsearch client directly.

Search sessions are an additional feature of the search service that allows for saving queries or run them in the background.
For an excellent primer on Search Sessions, check out the Kibana dev docs: https://docs.elastic.dev/kibana-dev-docs/tutorials/data/search-and-sessions

The work on integrating search sessions into Canvas was originally blocked by SQL not having async support but that's been added through this PR: elastic/elasticsearch#73991

The Dashboard application is also going to be helpful on this project since Dashboard already handles the creation and handling of search sessions. We can copy a lot of patterns from dashboard in terms of when to create/modify search sessions and how to handle passing search sessions to editor applications (that way the editor app can re-use the search session when loading up the query which causes the visualization to be immediately available to the user with very little wait time since the query doesn't have to be re-run by the editor)

Milestones

• Convert existing synchronous SQL Search Strategy to asynchronous¹
  • EQL has an async search strategy already and the SQL endpoint is nearly the same API. We can use it as a template and example
• Convert datasource functions (essql, esdocs, escount) to be asynchronous and handle in Canvas¹
  • This might be tricky -- search sessions generally allow for pushing queries to the backend if they take awhile so we'll have to figure out how that will work with expressions. This might be an optional feature so it might be better to skip that and add an additional follow-on milestone for that if implementation seems difficult and is optional. Will need to consult with app services/product about this.
  • This step might make it difficult to put this entire feature behind a Labs project depending on how much has to be re-worked
• Implement logic to restore workpad state from search service locator ¹
• Update filter behavior to modify search session if necessary¹
• Tests¹
• Pass search session to editor apps when selecting the "Edit" action on embeddables²
• Move essql, esdocs, and escount expression functions and the SQL search strategy to live in a shared plugin (perhaps data plugin?) so other apps (like Lens) can use them when needed³

Deployment

Each milestone here is marked with a number to denote when items can be broken into separate PRs.

1. Initial PR for first version with basic search session support (This work is required before follow-on work can continue)
2. Follow-on PR for passing search sessions to external applications
3. Follow-on PR to slightly refactor and move where the Canvas datasource functions live so they can be re-used by other apps going forward

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[poffdeluxe]

Still experimenting with this but it seems like the biggest lift at the moment will be to create the search strategy for ES SQL

[poffdeluxe]

Related to #88790

[poffdeluxe]

Will close #75068

[lizozom]

I would recommend splitting this into three tasks:

• Implementing the SQL search strategy
• Migrating all searches to use data.search
• Integrating search sessions (this task)

Tasks 1 and 2 should be pretty straight forward and quick and the team have allocated time in 7.13 to help you with them. 🙏

[lizozom]

I can update that @bpintea is looking into async SQL support.

[poffdeluxe]

I've updated this issue to breakdown work a bit more and noted where this could be split into multiple PRs.

[clintandrewhall]

I'm pinging @ppisljar and @elastic/kibana-app-services because I would be interested to explore writing new data-fetching expressions in the Expressions plugin for everyone to use, (or at least placing them in an as-yet-defined package). Committing to new functions outside of Canvas also means we can move to them in Canvas and deprecate the old easily.

[poffdeluxe]

I added a milestone line item to move the datasource functions to a shared plugin per our discussion in the expressions working group meeting this morning

[ThomThomson]

Closing this for the time being because we are not currently prioritizing Canvas tech debt or integrating Canvas more tightly with the rest of Kibana. We can reopen this one if our priorities change.