[Fleet] Agent logs to show link to discover for security projects #167976

juliaElastic · 2023-10-04T11:27:25Z

Summary

Show Open in Discover button instead of Open in Logs in security serverless projects.

To verify:

Start es and kibana locally in serverless mode

yarn es serverless --kill
yarn serverless-security

#  Login with username elastic_serverless or system_indices_superuser and password changeme

Start fleet-server locally with the fleet-server-dev service token:

# add to fleet-server.yml
output:
  elasticsearch:
    hosts: "http://localhost:9200"
    service_token: AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL2ZsZWV0LXNlcnZlci1kZXY6VVo1TWd6MnFTX3FVTWliWGNXNzlwQQ

# build fleet-server
SNAPSHOT=true DEV=true make release-darwin/amd64
# start fleet-server
./build/binaries/fleet-server-8.11.0-SNAPSHOT-darwin-x86_64/fleet-server -c fleet-server.yml

enroll an agent (or create a fake .fleet-agents doc)
go to Agent details and check that instead of Open in Logs there is a button Open in Discover

I have used the capabilities list from serverless.{type}.yml which is set to security/observability in the corresponding project types, but for some reason the capabilities list is not populated. @nchaulet any idea why those configs are not picked up?
Getting this on the UI:

agent_logs.tsx:87 {
  "internal": {
    "fleetServerStandalone": true,
    "disableProxies": true,
    "activeAgentsSoftLimit": 25000,
    "onlyAllowAgentUpgradeToKnownVersions": true
  },
  "agents": {
    "enabled": true
  },
  "developer": {},
  "enableExperimental": []
}

Agent logs view in security project:

Button navigates to Discover with these filters:
Data is populated with a real agent

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

apmmachine · 2023-10-04T11:27:42Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

...c/applications/fleet/sections/agents/agent_details_page/components/agent_logs/agent_logs.tsx

joshdover · 2023-10-04T13:35:09Z

I'd really like to find a way to retain the Logs view in Fleet. There has been work to try to maintain this but it seems that something got broken along the way:

@tonyghiani could you help us figure out how we can get this working again in Fleet on Security projects?

tonyghiani · 2023-10-04T13:44:16Z

I'd really like to find a way to retain the Logs view in Fleet.
@tonyghiani could you help us figure out how we can get this working again in Fleet on Security projects?

Hey @joshdover, happy to help, could you just expand a bit more on what seems not to work anymore, please?

The tickets you mentioned were about decoupling the LogStream component and logic to be reused in Fleet and other consumers, has it stopped working?

joshdover · 2023-10-04T13:45:20Z

@tonyghiani Please see the bug that this PR closes: #167304. We're seeing 500s on requests to /api/infra/log_views/default

juliaElastic · 2023-10-04T14:05:19Z

@tonyghiani Please see the bug that this PR closes: #167304. We're seeing 500s on requests to /api/infra/log_views/default

Yes we were seeing this error:

GET /api/infra/log_views/default 500

{
    "statusCode": 500,
    "error": "Internal Server Error",
    "message": "A fallback LogView handler is not registered. Register one in the setup method of your server plugin."
}

tonyghiani · 2023-10-04T14:13:10Z

@juliaElastic and I will continue the conversation offline to debug the issue. I understand what the issue root is, we just need to validate it.
cc @joshdover

juliaElastic · 2023-10-04T14:24:03Z

Created a security project in qa to debug, and the logging view is working now, not sure when this was fixed but it seems there is no issue:

https://jb-test-security-ffef54.kb.eu-west-1.aws.qa.elastic.cloud/app/fleet/agents/1d0afeb8-7081-4d16-a485-553557fa2574/logs

Tested locally with kibana serverless-security, and it works there too.

Only the Open in Logs UI button doesn't work, so we still want to replace that to Open in Discover, right? @joshdover

joshdover · 2023-10-04T16:44:16Z

Only the Open in Logs UI button doesn't work, so we still want to replace that to Open in Discover, right? @joshdover

Yes, SGTM

tonyghiani · 2023-10-05T08:29:39Z

@juliaElastic @joshdover should this link go to the new Log Explorer experience based on Discover instead of navigating to normal Discover? If on the security serverless project the observabilityLogExplorer app is enabled (which I believe is the case), we should point the user there, what do you think?

juliaElastic · 2023-10-05T08:58:12Z

@juliaElastic @joshdover should this link go to the new Log Explorer experience based on Discover instead of navigating to normal Discover? If on the security serverless project the observabilityLogExplorer app is enabled (which I believe is the case), we should point the user there, what do you think?

Thanks for the suggestion, though Log Explorer is not available in security projects.

elasticmachine · 2023-10-05T13:30:45Z

Pinging @elastic/fleet (Team:Fleet)

kpollich

LGTM - thank you for the tests. 🚀

nchaulet

Sorry I did not see that you were trying to use that capabilities settings from the client, you will have to whitelist it so it can be use here

juliaElastic · 2023-10-06T08:36:03Z

Sorry I did not see that you were trying to use that capabilities settings from the client, you will have to whitelist it so it can be use here

This was it, thanks! It works locally after adding here.

juliaElastic · 2023-10-06T09:06:43Z

Updated to show Open in Discover for observability project too, see reasoning here.

nchaulet · 2023-10-06T12:48:45Z

...c/applications/fleet/sections/agents/agent_details_page/components/agent_logs/agent_logs.tsx

@@ -119,6 +120,8 @@ export const AgentLogsUI: React.FunctionComponent<AgentLogsProps> = memo(
  ({ agent, agentPolicy, state }) => {
    const { data, application, http } = useStartServices();
    const { update: updateState } = AgentLogsUrlStateHelper.useTransitions();
+    const config = useConfig();
+    const isLogsUIAvailable = !config.internal?.registry?.capabilities;


Should we use the cloud plugin isServerlessEnabled method instead of that condition it could be more robust, wdyt?

updated, I tested by setting xpack.cloud.serverless.project_id: "1234" in serverless.yml, as the flag is based on that

kibana-ci · 2023-10-06T14:54:31Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: ab5c748

Failed CI Steps

Defend Workflows Cypress Tests on Serverless #1

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.2MB	1.2MB	+525.0B

History

💛 Build #165882 was flaky beed748
💛 Build #165818 was flaky 1559d15
💚 Build #165617 succeeded 142d9c4
💚 Build #165331 succeeded 44f5c63
💛 Build #165234 was flaky d6bc04b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

nchaulet

LGTM 🚀

@nchaulet

…astic#167976) ## Summary Closes elastic#167304 Show `Open in Discover` button instead of `Open in Logs` in security serverless projects. To verify: - Start es and kibana locally in serverless mode ``` yarn es serverless --kill yarn serverless-security # Login with username elastic_serverless or system_indices_superuser and password changeme ``` - Start fleet-server locally with the fleet-server-dev service token: ``` # add to fleet-server.yml output: elasticsearch: hosts: "http://localhost:9200" service_token: AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL2ZsZWV0LXNlcnZlci1kZXY6VVo1TWd6MnFTX3FVTWliWGNXNzlwQQ # build fleet-server SNAPSHOT=true DEV=true make release-darwin/amd64 # start fleet-server ./build/binaries/fleet-server-8.11.0-SNAPSHOT-darwin-x86_64/fleet-server -c fleet-server.yml ``` - enroll an agent (or create a fake `.fleet-agents` doc) - go to Agent details and check that instead of `Open in Logs` there is a button `Open in Discover` I have used the capabilities list from `serverless.{type}.yml` which is set to security/observability in the corresponding project types, but for some reason the capabilities list is not populated. @nchaulet any idea why those configs are not picked up? Getting this on the UI: ``` agent_logs.tsx:87 { "internal": { "fleetServerStandalone": true, "disableProxies": true, "activeAgentsSoftLimit": 25000, "onlyAllowAgentUpgradeToKnownVersions": true }, "agents": { "enabled": true }, "developer": {}, "enableExperimental": [] } ``` Agent logs view in security project: <img width="1346" alt="image" src="https://github.com/elastic/kibana/assets/90178898/a900c8a3-51d7-4e1a-8848-426352b39246"> Button navigates to Discover with these filters: Data is populated with a real agent <img width="1775" alt="image" src="https://github.com/elastic/kibana/assets/90178898/a51b3d19-04eb-49a9-9018-308737757b1d"> ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

…astic#171525) Fixes elastic#168349 ## Summary Fix links to Logs view to point to Discover in Serverless. As the Logs view UI is not available in serverless, the "Open in logs" buttons should point to Discover instead. Rather than hardcode the url in each of the places where is needed, I extracted a small component that builds the two urls and allows switching in an easier way. If in the future on of the two links will go away, it will be easier to find those occurrences. ### Testing Test for serverless following [these instructions](elastic#167976) **Error logs in agent activity flyout** - Enroll an agent and try to cause some error - for instance upgrading an agent that is not upgradeable - Click on "Agent Activity" and find the error and a button besides it - On stateful the button says "Open in Logs" ![Screenshot 2023-11-20 at 13 07 08](https://github.com/elastic/kibana/assets/16084106/704cf0e2-c7ee-4751-9e7f-7dcd263a5aa4) - On serverless is "Open in discover" ![Screenshot 2023-11-20 at 13 08 02](https://github.com/elastic/kibana/assets/16084106/3902f09e-93dc-48d3-867e-1f80d977f437) - Check that both show the same logs: ![Screenshot 2023-11-16 at 11 49 24](https://github.com/elastic/kibana/assets/16084106/d863d99f-0c70-45e5-9316-a37645464c34) ![Screenshot 2023-11-16 at 11 48 54](https://github.com/elastic/kibana/assets/16084106/7cbd0a5f-3b31-4c4d-a4b7-4eb7390983c8) **Agent logs** (Same test as above) - Enroll an agent - Click on the agent and go to the "Logs" tab - On stateful the button says "Open in Logs" ![Screenshot 2023-11-20 at 13 04 41](https://github.com/elastic/kibana/assets/16084106/6a43a062-37db-47ea-819f-acd170439395) - On serverless is "Open in discover" ![Screenshot 2023-11-20 at 13 04 11](https://github.com/elastic/kibana/assets/16084106/e15fdc8b-8780-4ac6-afc6-bff3d3a96be5) - Check that both show the same logs **Custom Logs UI** There is also a link to logs on custom logs UI but I just linked to discover for that one: https://github.com/elastic/kibana/pull/171525/files#diff-e337aa916d60d0d1033e3298c8c9c33c6a6fcd87a8ded971a4a87f5ccfc0981fR20-R22 --------- Co-authored-by: Kibana Machine <[email protected]>

show link to discover

d6bc04b

juliaElastic added the release_note:skip Skip the PR/issue when compiling release notes label Oct 4, 2023

juliaElastic self-assigned this Oct 4, 2023

juliaElastic commented Oct 4, 2023

View reviewed changes

...c/applications/fleet/sections/agents/agent_details_page/components/agent_logs/agent_logs.tsx Outdated Show resolved Hide resolved

juliaElastic mentioned this pull request Oct 4, 2023

[Fleet] Agent logs stream view broken on security serverless projects #167304

Closed

juliaElastic added 2 commits October 4, 2023 16:39

keep logstream

7ebe933

remove extra char

44f5c63

tonyghiani mentioned this pull request Oct 5, 2023

[Logs Explorer] Explorer is accessible via global search in the Security projects #166854

Closed

added tests

f79f98b

juliaElastic marked this pull request as ready for review October 5, 2023 13:26

juliaElastic requested a review from a team as a code owner October 5, 2023 13:26

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Oct 5, 2023

Merge branch 'main' into log-view-serverless

142d9c4

kpollich approved these changes Oct 5, 2023

View reviewed changes

nchaulet requested changes Oct 5, 2023

View reviewed changes

expose registry config to browser

1559d15

juliaElastic requested a review from nchaulet October 6, 2023 08:35

enable open in discover in observability project too

beed748

nchaulet reviewed Oct 6, 2023

View reviewed changes

use isServerlessEnabled

ab5c748

juliaElastic requested a review from nchaulet October 6, 2023 14:33

nchaulet approved these changes Oct 6, 2023

View reviewed changes

juliaElastic merged commit da9b65e into elastic:main Oct 9, 2023

kibanamachine added v8.12.0 backport:skip This commit does not require backporting labels Oct 9, 2023

This was referenced Nov 16, 2023

[Fleet] Add links to Discover in serverless when Logs UI not available #168349

Closed

[Fleet] Fix links to Logs view to point to Discover in Serverless #171525

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Agent logs to show link to discover for security projects #167976

[Fleet] Agent logs to show link to discover for security projects #167976

juliaElastic commented Oct 4, 2023 •

edited

Loading

apmmachine commented Oct 4, 2023

joshdover commented Oct 4, 2023

tonyghiani commented Oct 4, 2023

joshdover commented Oct 4, 2023 •

edited

Loading

juliaElastic commented Oct 4, 2023

tonyghiani commented Oct 4, 2023

juliaElastic commented Oct 4, 2023 •

edited

Loading

joshdover commented Oct 4, 2023

tonyghiani commented Oct 5, 2023

juliaElastic commented Oct 5, 2023

elasticmachine commented Oct 5, 2023

kpollich left a comment

nchaulet left a comment •

edited

Loading

juliaElastic commented Oct 6, 2023

juliaElastic commented Oct 6, 2023

nchaulet Oct 6, 2023

juliaElastic Oct 6, 2023

kibana-ci commented Oct 6, 2023

nchaulet left a comment

[Fleet] Agent logs to show link to discover for security projects #167976

[Fleet] Agent logs to show link to discover for security projects #167976

Conversation

juliaElastic commented Oct 4, 2023 • edited Loading

Summary

Checklist

apmmachine commented Oct 4, 2023

🤖 GitHub comments

joshdover commented Oct 4, 2023

tonyghiani commented Oct 4, 2023

joshdover commented Oct 4, 2023 • edited Loading

juliaElastic commented Oct 4, 2023

tonyghiani commented Oct 4, 2023

juliaElastic commented Oct 4, 2023 • edited Loading

joshdover commented Oct 4, 2023

tonyghiani commented Oct 5, 2023

juliaElastic commented Oct 5, 2023

elasticmachine commented Oct 5, 2023

kpollich left a comment

Choose a reason for hiding this comment

nchaulet left a comment • edited Loading

Choose a reason for hiding this comment

juliaElastic commented Oct 6, 2023

juliaElastic commented Oct 6, 2023

nchaulet Oct 6, 2023

Choose a reason for hiding this comment

juliaElastic Oct 6, 2023

Choose a reason for hiding this comment

kibana-ci commented Oct 6, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

History

nchaulet left a comment

Choose a reason for hiding this comment

juliaElastic commented Oct 4, 2023 •

edited

Loading

joshdover commented Oct 4, 2023 •

edited

Loading

juliaElastic commented Oct 4, 2023 •

edited

Loading

nchaulet left a comment •

edited

Loading