Elastic Agent install fails on Win 2012 Server as: filebeat[xyz] operation-install: exit status 1

[EricDavisX]

The new Alpha Elastic Agent is supported on Win 2012 I believe, I tried running it and it enrolled ok, but when running it, it throws an error, which is bubbled up into the Fleet Ingest Manager UI as:

Application: filebeat[f8419071-5c7d-4c54-8a0e-5cf4c9a9eec5]: operation-install: exit status 1 

I'm testing on BC3 of 7.8 on Elastic Cloud Staging, and using a hand compiled 8.0 Agent from 5/27.
Agent OS Version: Windows Server 2012 R2 Standard Build 9600 (2013)

• I'm afraid I only have a copy of this in vSphere, behind the Endgame VPN at the moment. We can work on how to get the team access or buy another image to test with for public use.

Attached are 2 screenshots and the cmd prompt logs from the machine itself


error-logs-win-2012.txt

specific error lines in the console output:
2020-05-27T15:15:45-04:00 ERROR reporter.go:47 2020-05-27T15:15:45-04:00: type:
'ERROR': sub_type: 'CONFIG' message: Application: metricbeat[f8419071-5c7d-4c54
-8a0e-5cf4c9a9eec5]: operation-install: exit status 1
2020-05-27T15:15:45-04:00 DEBUG action_dispatcher.go:93 Failed to dispatch actio
n 'action_id: e770d274-7d40-4fe5-b7e2-37fe3b7f8035, type: CONFIG_CHANGE', error:
operator: failed to execute step sc-run, error: operation-install: exit status
1: operation-install: exit status 1
operator: failed to execute step sc-run, error: operation-install: exit
status 1: operation-install: exit status 1
operation-install: exit status 1
exit status 1
2020-05-27T15:15:45-04:00 ERROR fleet_gateway.go:163 failed to dispatch actio
ns, error: operator: failed to execute step sc-run, error: operation-install: ex
it status 1: operation-install: exit status 1
operator: failed to execute step sc-run, error: operation-install: exit
status 1: operation-install: exit status 1
operation-install: exit status 1
exit status 1

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[EricDavisX]

contact me for RDP / connectivity info to the host. not that I have much of an answer to help provide it.

FYI the enroll call, if it helps, was:
./elastic-agent enroll https://4a8960f848b24467969a0e03b81ff856.eastus2.staging.azure.foundit.no:443 WnFVYlUzSUJsTkZDblUwN29oOGc6LVFKNzk4eTBTdzZUWWNIZHBES3hDQQ==

• I can leave this server up if anyone wants to get into ES / Kibana, let me know and I can pass ES creds

[EricDavisX]

FYI - the same vSphere sourced Windows images are working for other Windows Agent tests I've run, to rule out the general environment connectivity overall.

[EricDavisX]

@nayyara-samuel @crowens @bag0bits (hope you don't mind I selected your names for initial questions) Do we have any hope of getting a few specific folks access to the Endgame VPN, so they could test on our wonderful vSphere server? Or is it more feasible, faster to get them VMWare Fusion licenses and try to download / acquire the templates used to have them test locally @ruflin I think we had chatted briefly on this.

[ph]

@EricDavisX would be nice to give access for @michalpristas .

[EricDavisX]

we've also pinged Nick Fritz for access to see how Endpoint team may be able to help

[EricDavisX]

@kamalpreetpahwa-qasource if you could also test this out with a clean Win 2012 server Endgame side vSphere image I would be appreciative. I think maybe my vm is in a slightly bad state (maybe). If it works for you I think we can close this. We can try fleet and stand-alone, separately, please. and report back? Thank you.

[blakerouse]

I got a felling this might have been fixed by #19248. Because Elastic Agent used to run the powershell script to install metricbeat and filebeat as a service, even though they were not ran as a service by Elastic Agent. That PR removed that as it was noticed with installing Endpoint.

[EricDavisX]

we can wait a little bit and test it together then with Agent + Endpoint, as well as with Stand-alone Endpoint deploys @kamalpreetpahwa-qasource . thanks Blake for the info.

[ghost]

Sure @EricDavisX , let me update when we can test this issue.

[ghost]

Hi @EricDavisX

We have validated this issue on clean Windows server 2012 machine with 8.0 snapshot and issue is still occurring. Please find below the details and observations.

Kibana 8.0 cloud environment

Agent 8.0-snapshot

Host: Window server 2012 r2 standard

Observation:

1. Enrolled agent using Fleet>Token String:
   We observed that error "File beat Application status changed to Failed: exit status 1" appears while running the agent. Please refer below screenshot for error.

Screenshot 1:

Screenshot 2:

Screenshot 3:

For Stand alone agent, we are facing configuration issue, for which we have already reported ticket #19550.

Moreover, we are facing a new issue for "Integrations" tab on 8.0 Snapshot build. We have reported this issue.elastic/kibana#70565

Please let us know if anything else need to verify.

Thanks,

[michalpristas]

@blakerouse were you testing on win 10 or server?

[blakerouse]

Yes I am testing on Windows 10.

I was only suggesting it could be fixed, but seems we need to take a deeper look at 2012 R2.

[EricDavisX]

I tested again today with Agent build today and I see Metricbeat is reported as failed this time (not filebeat). maybe interesting. I can offer to do a zoom call or something to drive to check out / debug more when we want to make time, now that we're in better shape.

Application: metricbeat--7.9.0-SNAPSHOT[1a55cf6b-98c4-4137-8bed-ba8015791f15]: State changed to FAILED: exit status 1

[michalpristas]

zoom would be nice, let's sync to agree on some time this week

[EricDavisX]

fyi - I reviewed this with Michal and here are the logs we got at the time, note I changed the file name to be accepted in git attachment.

elastic-agent-json.log
elastic-agent-json1.log
elastic-agent-json2.log
elastic-agent-json3.log
elastic-agent-json4.log
elastic-agent-json5.log

[michalpristas]

reproduced locally. the issue is that windows 8/server 2012 use older version of powershell which does not contain functions we use to extract archive.
will prepare a fix soon

[michalpristas]

Fix here: #20088

[EricDavisX]

will test in BC5 when it is built

[michalpristas]

@EricDavisX can we close this one?

[EricDavisX]

yes! I thought I had. : / 👍

[ghost]

Bug Conversion:

Created 01 new Test-Case for this Ticket

• https://elastic.testrail.io/index.php?/cases/view/27068