[Logs Onboarding] Adds install shipper step for custom logs #157802

ogupte · 2023-05-15T21:01:35Z

Closes #154937

This PR includes the steps to install standalone elastic agent + reporting the status from bash script back to kibana.

…154937) (WIP)

apmmachine · 2023-05-15T21:01:49Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…-fix'

…fix'

x-pack/plugins/observability_onboarding/public/assets/standalone-agent-setup.sh

- add API to get latest onboarding state

…-ref HEAD~1..HEAD --fix'

…ic-agent-154937' into logs-onboaring-yaml-generator

[Logs onboarding] Generating yaml configuration in server

…ndalone-elastic-agent-154937

...rvability_onboarding/server/routes/custom_logs/find_latest_observability_onboarding_state.ts

…fix'

…te --fix'

… mapping

…tup script

- cleans up some routes

…157802)" This reverts commit 12401b2.

jbudz · 2023-05-23T20:15:05Z

This was reverted - https://buildkite.com/elastic/kibana-on-merge/builds/30746#01884a27-a5a7-4d65-90f4-d8da9ccd781a/2582-2845

sorenlouv · 2023-05-23T22:47:53Z

To add a little context as to why this was reverted: The changes in this PR is causing the CI to fail with the error " FATAL Error: Missing version for public endpoint GET /api/observability_onboarding/custom_logs/step/{name}".

Versions in public endpoints are required due to #158233.

dgieselaar · 2023-05-24T03:41:13Z

@yngrdyn @ogupte why are these endpoints public?

yngrdyn · 2023-05-24T06:19:12Z

@yngrdyn @ogupte why are these endpoints public?

We needed them to be accessible from outside, there is a script that will be in charge of downloading and installing elastic agent (right now is very basic, the idea is to extend it), this script also needs to report back to kibana the progress in order to achieve the last step of onboarding the flow.

ogupte · 2023-05-24T12:05:45Z

Why did CI pass prior to merging if it failed after merging to main? Maybe the API check was added just before we merged this?

dgieselaar · 2023-05-24T12:06:39Z

@ogupte yes, I merged it yesterday somewhere in the afternoon I think

Closes #154937 This PR is an extension of [#157802](#157802) which was reverted because public Api endpoint versioning problems. ![Screenshot 2023-05-15 at 5 02 29 PM](https://github.com/elastic/kibana/assets/1967266/62484fb3-e02f-410d-aa7a-86bcc4dc0b03) --------- Co-authored-by: Oliver Gupte <[email protected]> Co-authored-by: kibanamachine <[email protected]>

ruflin · 2023-05-25T07:55:32Z

x-pack/plugins/observability_onboarding/public/assets/standalone_agent_setup.sh

+
+echo "Downloading Elastic Agent"
+# https://www.elastic.co/guide/en/fleet/8.7/install-standalone-elastic-agent.html
+curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.7.1-linux-x86_64.tar.gz


Ideally the agent version becomes an input param so users always get the version compatible with their Stack version.

ruflin · 2023-05-25T07:57:13Z

x-pack/plugins/observability_onboarding/public/assets/standalone_agent_setup.sh

+API_ENDPOINT=$2
+
+updateStepProgress() {
+  echo "  GET $API_ENDPOINT/step/$1?status=$2"


This means Kibana has to be accessible for this to work. This is ok but we should mention it in our docs or when users hit erros, make sure we provide good error messages that Kibana is not reachable for example.

ruflin · 2023-05-25T07:57:37Z

x-pack/plugins/observability_onboarding/public/assets/standalone_agent_setup.sh

+API_KEY_ENCODED=$1
+API_ENDPOINT=$2
+
+updateStepProgress() {


I like the simplicity of it. It means we can add / remove steps when needed.

…157802) Closes elastic#154937 This PR includes the steps to install standalone elastic agent + reporting the status from bash script back to kibana. ![Screenshot 2023-05-15 at 5 02 29 PM](https://github.com/elastic/kibana/assets/1967266/62484fb3-e02f-410d-aa7a-86bcc4dc0b03) --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Yngrid Coello <[email protected]> Co-authored-by: Yngrid Coello <[email protected]>

…lastic#157802)" This reverts commit 12401b2.

[Logs Onboarding] Adds install shipper step for custom logs (elastic#…

d406d25

…154937) (WIP)

This was referenced May 15, 2023

[Logs onboarding] Install shipper for standalone Elastic Agent ogupte/kibana#6

Closed

[Logs onboarding] Custom logs flow #155821

Closed

[Design] Custom logs onboarding flow #154925

Closed

kibanamachine and others added 3 commits May 15, 2023 21:38

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

aeb5aaf

…-fix'

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

6571f21

…fix'

Adds new saved object type observability-onboarding-state

e622f11

yngrdyn reviewed May 16, 2023

View reviewed changes

x-pack/plugins/observability_onboarding/public/assets/standalone-agent-setup.sh Outdated Show resolved Hide resolved

yngrdyn and others added 11 commits May 18, 2023 16:36

Generating yaml configuration in server

f394807

Cleans up script and fixes progress updates to be saved in saved objects

6a36ead

set saved objects ids based on apikeyId

dd472c8

- fixed premature API call from wizard when rendered for transition

b3bc425

- add API to get latest onboarding state

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

aba20b5

…-ref HEAD~1..HEAD --fix'

Merge remote-tracking branch 'origin/logs-onboarding-standalone-elast…

741ad36

…ic-agent-154937' into logs-onboaring-yaml-generator

Using internal savedObjects client

2dce880

Merge pull request #7 from ogupte/logs-onboaring-yaml-generator

28bc0fa

[Logs onboarding] Generating yaml configuration in server

Merge remote-tracking branch 'origin/master' into logs-onboarding-sta…

385cc2e

…ndalone-elastic-agent-154937

Merge remote-tracking branch 'origin/master' into logs-onboarding-sta…

0834913

…ndalone-elastic-agent-154937

Fixing build

58c70ac

yngrdyn force-pushed the logs-onboarding-standalone-elastic-agent-154937 branch from dcf63de to 58c70ac Compare May 22, 2023 14:33

yngrdyn reviewed May 22, 2023

View reviewed changes

...rvability_onboarding/server/routes/custom_logs/find_latest_observability_onboarding_state.ts Show resolved Hide resolved

kibanamachine and others added 6 commits May 22, 2023 14:39

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

8c8a83a

…fix'

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

7eb32a8

…te --fix'

remove apiKeyId mapping and move progess up one level in saved object…

e559f5e

… mapping

updated saved object mapping for observability-onboarding-state

678f9bd

uses the static asset path provided for plugins for the downloaded se…

f7d0b44

…tup script

- adds API that checks for existence of log events for given dataset

e7b8d0d

- cleans up some routes

yngrdyn force-pushed the logs-onboarding-standalone-elastic-agent-154937 branch from cf54f74 to ee7b7b3 Compare May 23, 2023 09:21

Fixing build

db0fc5f

yngrdyn merged commit 12401b2 into elastic:main May 23, 2023

kibanamachine added v8.9.0 backport:skip This commit does not require backporting labels May 23, 2023

jbudz added a commit that referenced this pull request May 23, 2023

Revert "[Logs Onboarding] Adds install shipper step for custom logs (#…

46dc7ff

…157802)" This reverts commit 12401b2.

jbudz added the reverted label May 23, 2023

yngrdyn mentioned this pull request May 24, 2023

[Logs Onboarding] Adds install shipper step for custom logs #158386

Merged

ruflin reviewed May 25, 2023

View reviewed changes

delanni pushed a commit to delanni/kibana that referenced this pull request May 25, 2023

Revert "[Logs Onboarding] Adds install shipper step for custom logs (e…

7b280d8

…lastic#157802)" This reverts commit 12401b2.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Logs Onboarding] Adds install shipper step for custom logs #157802

[Logs Onboarding] Adds install shipper step for custom logs #157802

ogupte commented May 15, 2023 •

edited by yngrdyn

Loading

apmmachine commented May 15, 2023

jbudz commented May 23, 2023

sorenlouv commented May 23, 2023 •

edited

Loading

dgieselaar commented May 24, 2023

yngrdyn commented May 24, 2023

ogupte commented May 24, 2023

dgieselaar commented May 24, 2023

ruflin May 25, 2023

ruflin May 25, 2023

ruflin May 25, 2023

[Logs Onboarding] Adds install shipper step for custom logs #157802

[Logs Onboarding] Adds install shipper step for custom logs #157802

Conversation

ogupte commented May 15, 2023 • edited by yngrdyn Loading

apmmachine commented May 15, 2023

🤖 GitHub comments

jbudz commented May 23, 2023

sorenlouv commented May 23, 2023 • edited Loading

dgieselaar commented May 24, 2023

yngrdyn commented May 24, 2023

ogupte commented May 24, 2023

dgieselaar commented May 24, 2023

ruflin May 25, 2023

Choose a reason for hiding this comment

ruflin May 25, 2023

Choose a reason for hiding this comment

ruflin May 25, 2023

Choose a reason for hiding this comment

ogupte commented May 15, 2023 •

edited by yngrdyn

Loading

sorenlouv commented May 23, 2023 •

edited

Loading