[Logs UI] HTTP API for log entries

[afgomez]

Summary

Part of #51047.

This PR adds (will add) two endpoints:

• /log_entries/entries for the log entries themselves. They will be used in the log stream view.
• /log_entries/highlights for the log entry highlights. They will be used to highlight messages in the log stream view.

The API is not used yet, since its interface differs from the GraphQL implementation. The API will be used once we change the UI to allow date ranges.

Examples

Get entries from the top (oldest)

The API has only two mandatory parameters: startDate and endDate. Both take an epoch in milliseconds.

// Request
{
  startDate: 100000001000, 
  endDate: 100000002000
}

// Response
{
  data: {
    entries: [
		{
			id: "...",
			cursor: { time: 100000001000, tiebreaker: 12345 }
			columns: [...]
		},
		...
	],
    topCursor: { time: 100000001000, tiebreaker: 12345 }, // Cursor for the top of the page,
    bottomCursor: { time: 100000001200, tiebreaker: 67890 }, // Cursor for the bottom of the page,
  }
}

By default the request gives the first page of entries, in ascending order, starting at startDate.

To get the following page: pass the after parameter with the value of the bottomCursor

// Request
{
  startDate: 100000001000, 
  endDate: 100000002000,
  after: { time: 100000001200, tiebreaker: 67890 } 
}

// Response
{
  data: {
    entries: [
		{
			id: "...",
			cursor: { time: 100000001201, tiebreaker: 67891 }
			columns: [...]
		},
		...
	],
    topCursor: { time: 100000001201, tiebreaker: 67891 },
    bottomCursor: { time: 100000001400, tiebreaker: 78901 },
  }
}

Get entries from the bottom (newest)

To get the last entries in the range, the user needs to specify the parameter before with the special value "last".

// Request
{
  startDate: 100000001000, 
  endDate: 100000002000,
  before: "last"
}

// Response
{
  data: {
    entries: [
		{
			id: "...",
			cursor: { time: 100000001800, tiebreaker: 88888 }
			columns: [...]
		},
		...
	],
    topCursor: { time: 100000001800, tiebreaker: 88888 },
    bottomCursor: { time: 100000002000, tiebreaker: 99999 },
  }
}

To get the entries before, pass the value of the topCursor in the before parameter.

// Request
{
  startDate: 100000001000, 
  endDate: 100000002000,
  before: { time: 100000001800, tiebreaker: 88888 }
}

// Response
{
  data: {
    entries: [
		{
			id: "...",
			cursor: { time: 100000001800, tiebreaker: 88888 }
			columns: [...]
		},
		...
	],
    topCursor: { time: 100000001600, tiebreaker: 66666 },
    bottomCursor: { time: 100000001799, tiebreaker: 88887 },
  }
}

Get entries around a specific cursor

Sometimes is interesting to get entries with a specific log line in the center. To do so the user must pass a cursor as a center parameter. If the user doesn't know the value of the tiebreaker for a specific line, they must use 0. The response will then contain logs around the timestamp.

// Request
{
  startDate: 100000001000, 
  endDate: 100000002000,
  center: { time: 100000001500, tiebreaker: 0 }
}

// Response
{
  data: {
    entries: [
		{
			id: "...",
			cursor: { time: 100000001450, tiebreaker: 44444 }
			columns: [...]
		},
		...
	],
    topCursor: { time: 100000001450, tiebreaker: 44444 },
    bottomCursor: { time: 100000001550, tiebreaker: 55555 },
  }
}

The user then can get the pages before or after using the topCursor and the bottomCursor of the response.

Add a query

All options support a query parameter to filter. The value must be the parsed query from the searchbar

// Request
{
  startDate: 100000001000, 
  endDate: 100000002000,
  query: '{"bool":{"should":[{"match_phrase":{"event.dataset":"nginx.access"}}],"minimum_should_match":1}}'
}

Get highlights

The /log_entries/highlights endpoint takes the same parameters as the /log_entries/entries endpoint, plus an array of highlight terms. The response is an array with the entries for each highlight term.

// Request
{
  startDate: 100000001000, 
  endDate: 100000002000,
  highlightTerms: ['admin'],
}

// Response
{
  // Each element here corresponds to each term in the request
  data: [{
	// The `entries` contain only elements with the highlight term
    entries: [
		{
			id: "...",
			cursor: { time: 100000001000, tiebreaker: 12345 }
			columns: [...]
		},
		...
	],
    topCursor: { time: 100000001000, tiebreaker: 12345 },
    bottomCursor: { time: 100000001600, tiebreaker: 66666 },
  }]
}

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

• This was checked for cross-browser compatibility, including a check against IE11
• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• [ ] This was checked for keyboard-only and screenreader accessibility

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: b17b7c2614f7c124237dd5a7227ea8399ffe1f3d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: a4f3b12e2871a45f9a4c9e36c8d5fbfd1c7aabd5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[Zacqary]

Hard to understand at a glance that hit._source is an object.path.like.this. Anything you can do with type definitions to make this more obvious? Or maybe just add a comment

[afgomez]

I hear you, but I don't want to dedicate much time to this function. It will go away in a separate PR.

Right now the API is handled in three files

• lib/adapters/kibana_log_entries_adapter, which connects with Elasticsearch.
• lib/domains/log_entries_domain, which connects the adapter with the route files
• The route file.

This function in the adapter takes the ES response and transforms it onto a LogEntryDocument, an intermediate format for the domain that then gets transformed again in the route.

I had a chat with @weltenwort and @Kerry350 a couple of weeks ago about how this code was organised, and the conclusion was to merge the domain and the adapter files into one. Once we do that we don't need an intermediate format anymore and this function will go away.

---

I will take your comment into account when I join the two files in one. I agree with you that it's not clear straight away what is in _source. I guess there's some documentation somewhere of how filebeat stores the log metadata in ES. We could just add a comment with a link to it.

[Zacqary]

With the above comment in mind, this whole function might be easier to follow if you destructure the hit at the beginning

const { _id: gid, _source: fieldName, sort: [time, tiebreaker] }

[afgomez]

See previous comment :)

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 7f62441247d448cd2f8357e6e59fea5a1641d2d0

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 911dac2307214a19ceb92be423a79f43e373befd

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: d04f15e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 5829456

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: c454479

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[Zacqary]

Looks good to me; once CI is fixed should be good to merge

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 451be62

History

• 💔 Build #17780 failed f860c5e
• 💔 Build #17600 failed 1971e5c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream