Fix elastic-agent cloud image run issues #1630

fearful-symmetry · 2022-10-27T22:34:08Z

What does this PR do?

We're seeing issues with various docker builds of elastic-agent failing with an "input not supported" error. It appears that when elastic-agent starts up in container mode, it'll set the paths incorrectly under V2, so we aren't reading the specfiles from the component directory.

This makes a few changes to the container setup to get the paths working under the "flat" V2 structure. I'm not 100% sure what else is happening under "container mode" so I want someone with a little more experience here to make sure I'm not breaking anything. I tested this natively and inside a container, and it seems to work.

In the future, we might want to modify the specfile load.go so we return an specific error if we can't find any specfiles

Checklist

My code follows the style guidelines of this project
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2022-10-27T22:37:50Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-10-28T20:41:01.289+0000
Duration: 17 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4461
Skipped	13
Total	4474

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages.
run integration tests : Run the Elastic Agent Integration tests.
run end-to-end tests : Generate the packages and run the E2E Tests.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2022-10-27T22:54:23Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	98.305% (`58/59`)	❕
Files	70.256% (`137/195`)	❕
Classes	70.161% (`261/372`)	❕
Methods	54.318% (`780/1436`)	❕
Lines	39.634% (`8287/20909`)	❕
Conditionals	100.0% (`0/0`)	💚

cmacknz · 2022-10-28T15:06:39Z

The biggest thing we need to make sure we don't break is using custom a STATE_PATH, which is often used to mount that directory on a persistent or host volume when the agent runs on Kubernetes so that the directory contents persist across restarts and upgrades. It should hopefully be easy to test this by just changing the state path environment variable.

cmacknz · 2022-10-28T17:46:35Z

internal/pkg/agent/cmd/container.go

@@ -789,10 +789,9 @@ func setPaths(statePath, configPath, logsPath string, writePaths bool) error {
 	}
 	originalInstall := paths.Install()
 	originalTop := paths.Top()
-	paths.SetTop(topPath)
+	paths.SetTop(statePath) // change this?


I think this is correct from my reading of the comments here:

elastic-agent/internal/pkg/agent/application/paths/common.go

Lines 59 to 71 in fc3eba3

// Top returns the top directory for Elastic Agent, all the versioned

// home directories live under this top-level/data/elastic-agent-${hash}

func Top() string {

return topPath

}

// SetTop overrides the Top path.

//

// Used by the container subcommand to adjust the overall top path allowing state can be maintained between container

// restarts.

func SetTop(path string) {

topPath = path

}

I also suspect you want to change the topPath variable here so that the permissions are set correctly:

elastic-agent/internal/pkg/agent/cmd/container.go

Lines 769 to 775 in fc3eba3

topPath := filepath.Join(statePath, "data")

configPath = envWithDefault(configPath, "CONFIG_PATH")

if configPath == "" {

configPath = statePath

}

// ensure that the directory and sub-directory data exists

if err := os.MkdirAll(topPath, 0755); err != nil {

cmacknz · 2022-10-28T17:56:52Z

internal/pkg/agent/cmd/container.go

 	paths.SetConfig(configPath)
-	// when custom top path is provided the home directory is not versioned
-	paths.SetVersionHome(false)


I'm curious to know what this does (I expect we need Michal or Blake to answer this). It's a bit hard to tell from the code.

As far as I can tell in the container I built, the home path is versioned by default. I get /usr/share/elastic-agent/data/elastic-agent-fc3eba. I don't know that this would change if the STATE_PATH were to change either so removing it seems right but it would be good to confirm.

Yah, this just adds the elastic-agent-fc3eba directory, which is still there in docker.

fearful-symmetry · 2022-10-28T18:21:37Z

@cmacknz , yah, I suspect there's deeper problems with how the state_path is being set, as elastic-agent in its current state (and in this PR) uses the state path for the entire run directory. I assume this code made more sense before the "flattened" V2 directory structure, but right now I'm not sure we have the mechanism to separate out the state info from the rest of the pre-installed elastic-agent assets.

fearful-symmetry · 2022-10-28T20:46:57Z

Alright, added the changelog and fixed up a few things. @cmacknz right now, it seems like elastic-agent just kind of mixes statefiles in the flat directory along with everything else. If the intent is to have STATE_PATH be a separate directory that users can mount just for tracking state, we may want to open an issue to track that.

michalpristas · 2022-10-29T14:00:28Z

internal/pkg/agent/cmd/container.go

@@ -40,9 +40,9 @@ import (
 const (
 	requestRetrySleepEnv     = "KIBANA_REQUEST_RETRY_SLEEP"
 	maxRequestRetriesEnv     = "KIBANA_REQUEST_RETRY_COUNT"
-	defaultRequestRetrySleep = "1s"                             // sleep 1 sec between retries for HTTP requests
-	defaultMaxRequestRetries = "30"                             // maximum number of retries for HTTP requests
-	defaultStateDirectory    = "/usr/share/elastic-agent/state" // directory that will hold the state data


this is here probably to share state across runs. have mounted volume.
this PR fixes the problem but i don't believe it's proper fix and will introduce another set of issues.

i believe proper fix would be to fix not just data/downloads but also data/components
will prepare PR on monday and test it as well

cmacknz · 2022-10-31T14:16:35Z

@michalpristas should this be closed in favour of #1653?

michalpristas · 2022-10-31T15:34:32Z

@cmacknz yes please

michalpristas · 2022-10-31T15:35:07Z

closing in favor of #1653

first pass at trying to fix cloud image issues

f00d42d

fearful-symmetry added bug Something isn't working v8.6.0 labels Oct 27, 2022

fearful-symmetry requested a review from michalpristas October 27, 2022 22:34

fearful-symmetry self-assigned this Oct 27, 2022

fearful-symmetry requested a review from a team as a code owner October 27, 2022 22:34

fearful-symmetry requested review from michel-laterman and removed request for a team October 27, 2022 22:34

remove commented-out code

2ff79ca

fearful-symmetry mentioned this pull request Oct 27, 2022

Agent v2 elastic/cloudbeat#458

Merged

cmacknz requested a review from blakerouse October 28, 2022 13:17

cmacknz added the skip-changelog label Oct 28, 2022

cmacknz reviewed Oct 28, 2022

View reviewed changes

add changelog, change mkdir call

5f015f4

michalpristas requested changes Oct 29, 2022

View reviewed changes

michalpristas closed this Oct 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix elastic-agent cloud image run issues #1630

Fix elastic-agent cloud image run issues #1630

fearful-symmetry commented Oct 27, 2022

elasticmachine commented Oct 27, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Oct 27, 2022 •

edited

Loading

cmacknz commented Oct 28, 2022

cmacknz Oct 28, 2022

cmacknz Oct 28, 2022

fearful-symmetry Oct 28, 2022

fearful-symmetry commented Oct 28, 2022

fearful-symmetry commented Oct 28, 2022

michalpristas Oct 29, 2022 •

edited

Loading

cmacknz commented Oct 31, 2022

michalpristas commented Oct 31, 2022

michalpristas commented Oct 31, 2022

	// Top returns the top directory for Elastic Agent, all the versioned
	// home directories live under this top-level/data/elastic-agent-${hash}
	func Top() string {
	return topPath
	}

	// SetTop overrides the Top path.
	//
	// Used by the container subcommand to adjust the overall top path allowing state can be maintained between container
	// restarts.
	func SetTop(path string) {
	topPath = path
	}

	topPath := filepath.Join(statePath, "data")
	configPath = envWithDefault(configPath, "CONFIG_PATH")
	if configPath == "" {
	configPath = statePath
	}
	// ensure that the directory and sub-directory data exists
	if err := os.MkdirAll(topPath, 0755); err != nil {

Fix elastic-agent cloud image run issues #1630

Fix elastic-agent cloud image run issues #1630

Conversation

fearful-symmetry commented Oct 27, 2022

What does this PR do?

Checklist

elasticmachine commented Oct 27, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Oct 27, 2022 • edited Loading

🌐 Coverage report

cmacknz commented Oct 28, 2022

cmacknz Oct 28, 2022

Choose a reason for hiding this comment

cmacknz Oct 28, 2022

Choose a reason for hiding this comment

fearful-symmetry Oct 28, 2022

Choose a reason for hiding this comment

fearful-symmetry commented Oct 28, 2022

fearful-symmetry commented Oct 28, 2022

michalpristas Oct 29, 2022 • edited Loading

Choose a reason for hiding this comment

cmacknz commented Oct 31, 2022

michalpristas commented Oct 31, 2022

michalpristas commented Oct 31, 2022

elasticmachine commented Oct 27, 2022 •

edited

Loading

elasticmachine commented Oct 27, 2022 •

edited

Loading

michalpristas Oct 29, 2022 •

edited

Loading