Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Logs] Scaffolding for Log Explorer within Discover #135913

Draft
wants to merge 108 commits into
base: main
Choose a base branch
from

Conversation

Kerry350
Copy link
Contributor

@Kerry350 Kerry350 commented Jul 7, 2022

Summary

Ticket: #134764 (comment)

This adds a new /log-explorer route within Discover to serve as a "base" for the Log Explorer POC. There isn't really any new code here, it's mostly copied and pasted from the existing Discover code and then slightly altered. We can adjust further and remove certain functionality (that we don't need) as we go. We'll also be scoping down types, props, and state to those we need. There will also be a state machine to back the more complex state.

This is a preliminary draft PR to mainly collaborate on code, but we can also have discussions here pinpointed against certain lines.

I've left some initial comments which highlight things that I found interesting, confusing, or useful putting in the scaffolding.

@@ -43,6 +44,9 @@ export const discoverRouter = (services: DiscoverServices, history: History, isD
<Route path="/view/:id">
<DiscoverMainRoute isDev={isDev} />
</Route>
<Route path="/log-explorer">
<DiscoverLogExplorerRoute isDev={isDev} />
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: Discover uses hash based routing.

The saved search ID lands as a query parameter, not part of the path.

</DiscoverTourProvider>
)}
<div className="dscDiscoverGrid">
<DataGridMemoized
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be replaced with the new DataGrid functionality from the EUI POC.

* State related logic
*/
const {
data$,
Copy link
Contributor Author

@Kerry350 Kerry350 Jul 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

useDiscoverState calls through to useSavedSearch this is where the bulk of the data querying heavy lifting happens. This is where we'll want to refactor the querying logic. The data$ observable in standard Discover would contain the documents, chart data, and total hits for populating the count.

id: id || 'new',
});

const loadDefaultOrCurrentIndexPattern = useCallback(
Copy link
Contributor Author

@Kerry350 Kerry350 Jul 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't really need to change this, as it works fine, and we want the exact same functionality. But, this is a bit of a loaded function, it does do what it says, but it also performs a lot of other logic (such as various no data states, determining whether there are data views, whether there is ES data etc).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing is the logic to detect the no data state. this can stay as it's similar in other places. other logic is about the saved search and data view(s). this should be refactored (It's a leftover how it worked in the Angular world)

return;
}

const { appStateContainer } = getState({ history, uiSettings: config });
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this function a little surprising. It sets up the state container, state storage, and calls syncState, but it seems that this function gets called in multiple places within Discover. We might want to look at only calling this once, via a provider / context.

Okay, that might actually just be because of the legacy Discover.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it's definitely a leftover :). there should just be one location the appStateContainer is created


const { appStateContainer } = getState({ history, uiSettings: config });
const { index } = appStateContainer.getState();
const ip = await loadIndexPattern(index || '', data.dataViews, config);
Copy link
Contributor Author

@Kerry350 Kerry350 Jul 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

loadIndexPattern doesn't return an index pattern directly, it returns IndexPatternData which contains the list, but also some extra properties like stateVal and stateValFound, which are used in other parts of code to determine certain things (like showing a toast via resolveIndexPattern).

hasData: {
...data.dataViews.hasData,

// We've already called this, so we can optimize the analytics services to
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general Discover seems to have more granular (and probably much more meaningful) analytics. We probably want to bear this in mind for functionality we add.

weltenwort and others added 22 commits July 12, 2022 11:04
weltenwort and others added 22 commits August 22, 2022 10:46
- Still some areas WIP but merge conflicts have been fixed here
@weltenwort
Copy link
Member

/oblt-deploy

@kibana-ci
Copy link
Collaborator

kibana-ci commented Aug 26, 2022

💔 Build Failed

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #3 / Discover grid cell rendering collect object fields and renders them as json in details
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering collect object fields and renders them like _source
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering collect object fields and renders them like _source with fallback for unmapped
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering does not collect subfields when the the column is unmapped but part of fields response
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering limits amount of rendered items
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders _source column correctly
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders _source column correctly when isDetails is set to true
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders a functional close button when CodeEditor is rendered
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders bytes column correctly
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders bytes column correctly using _source when details is true
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders bytes column correctly using fields when details is true
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders correctly when invalid column is given
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders correctly when invalid row is given
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders fields-based column correctly
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders fields-based column correctly when isDetails is set to true
  • [job] [logs] Jest Tests #3 / Discover grid cell rendering renders unmapped fields correctly
  • [job] [logs] Jest Tests #3 / discover responsive sidebar should allow adding filters
  • [job] [logs] Jest Tests #3 / discover responsive sidebar should allow deselecting fields
  • [job] [logs] Jest Tests #3 / discover responsive sidebar should allow selecting fields
  • [job] [logs] Jest Tests #3 / discover responsive sidebar should have Selected Fields and Available Fields with Popular Fields sections

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
discover 527 640 +113

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
data 2431 2435 +4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
discover 462.3KB 631.6KB +169.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
data 429.6KB 429.7KB +94.0B
Unknown metric groups

API count

id before after diff
data 3117 3121 +4

ESLint disabled in files

id before after diff
apm 14 13 -1
observability 6 5 -1
total -2

ESLint disabled line counts

id before after diff
apm 79 77 -2
discover 37 41 +4
enterpriseSearch 13 11 -2
observability 44 43 -1
synthetics 59 53 -6
ux 10 9 -1
total -8

Total ESLint disabled count

id before after diff
apm 93 90 -3
discover 38 42 +4
enterpriseSearch 13 11 -2
observability 50 48 -2
synthetics 65 59 -6
ux 13 12 -1
total -10

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Copy link
Member

@kertal kertal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a awesome & HUGE PR in every way 👍 , started to dig into it, test it and onboard on xstate, just wanted to report a first issue I've found while scrolling. What I did was scrolling up and down, so I don't think I did something forbidden:

Uncaught Error: Expected both chunks to have status "loaded", but found "loaded" and "empty".
    at load_before_service.ts:113:1
    at utils.js:413:1
    at Array.reduce (<anonymous>)
    at Object.updateContext (utils.js:401:1)
    at Object.resolveActions (actions.js:442:1)
    at StateNode.resolveTransition (StateNode.js:890:1)
    at StateNode.transition (StateNode.js:829:1)
    at interpreter.js:901:1
    at Object.provide (serviceScope.js:12:1)
    at Interpreter._nextState (interpreter.js:900:1)

BTW will the jumpy behavior of the scrollbar when loading improve when iterating on the grid? And I guess showing a loading indication would be something that's next, right?

@weltenwort
Copy link
Member

a first issue I've found while scrolling

hm, there might be some edge cases we didn't handle correctly in the prototype. especially the empty and loading failed states are not fleshed out. we tried to get the "happy" path working within the time restrictions.

BTW will the jumpy behavior of the scrollbar when loading improve when iterating on the grid?

Yes, when I write it again I would not initialize an empty chunk when loading starts, but only when it finishes. That way there should be less jumping.

And I guess showing a loading indication would be something that's next, right?

Indeed, we didn't include that in the prototype, but the state can be easily used to render something. We were imagining an indicator that hovers over the grid at the top or bottom to work around the grid's row rendering limitations.

} = useDiscoverStateContext();

// Inspector
// TODO: Fix this and move to independent hook
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here we go!
#140466
good call, thx!

sortCriteria: [
// TODO: don't hard-code this
[dataView.timeFieldName!, 'asc'],
['_doc', 'asc'], // _shard_doc is only available when used inside a PIT
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants