[Agent] Installed Windows 7 Agent shows no logs in Kibana (data is in Discover though) and shows host log error: agent events contains events with different agent id from currently authorized agent

[EricDavisX]

This is a 7.12 BC3 stack deploy to cloud with the BC3 Agent on a Win7 VM

Agent shows as 'healthy' but no logs show in the Agent details, presumably due to the mis-match somewhere of the Agent ID. With recent changes on Install in Beats (and none on Kibana side that seem to relate) I assume it is a Beats Agent bug.

The first error in the log ends up being repeated, over an over, stacking up the # of actions that are not dispatched, the first was:

{"log.level":"error","@timestamp":"2021-03-03T14:03:54.788-0500","log.origin":{"file.name":"application/fleet_gateway.go","file.line":185},"message":"failed to dispatch actions, error: acknowledge 1 actions '[action_id: 59c47640-7c44-11eb-b1f6-1755d2a3f765, type: POLICY_CHANGE]' for elastic-agent 'cbad3b28-720c-4bdf-8d58-988047ae540e' failed: Status code: 400, Kibana returned an error: Bad Request, message: agent events contains events with different agent id from currently authorized agent","ecs.version":"1.6.0"}

Full Logs zip attached along with state.yml (changed to .txt for git inclusion)
state.yml.txt
7.12-bc3-win-7-logs.zip

The Agent does seem to be sending data to ES and functioning, perhaps with just some default set policy.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[EricDavisX]

@ph let us add this to 7.12 triage set. We are now a few urgent issues deep, can we get Michal some assistance in reviewing?

[EricDavisX]

Team, I got some second opinions from Sec Engg prod team - they have a BC3 server with many working hosts that do NOT have this problem. BUT indeed, the Win 7 x64 host they setup seems to be in the same state as mine, with the bug. So, it is real I guess, just *somehow specific to win 7. I've updated the short desc. I don't know if that should be a blocker for 7.12, but I'd still like to know what is up before moving on... and maybe have more opinions on other Win 7 host deployments.

UPDATE: The team Win 7 x86 box seems ok. This is really curious

[EricDavisX]

Update 2: my original test case (call it: host1) is now showing logs and has Endpoint deployed. I can't explain it... I waited many minutes after changing the policy to see if it would have an impact before reporting this.

The policy change must have eventually worked?

The 2nd host we tested with Win 7 in the same state (host2) has logs that I captured here:

Win-7-v7.12.0-bc3-alternate-host-logs.zip

[michalpristas]

could you grab another portion of logs from machine 1.
i'm wondering if there was a restart in between or how it got fixed by itself.
if you can get action_store/state.yml and all the yml files basically to see config it would be also great

[blakerouse]

@michalpristas Does agent write the state.yml file even in standalone mode?

[EricDavisX]

I grabbed the whole logs folder already - it is posted above in description: 7.12-bc3-win-7-logs.zip

[michalpristas]

@blakerouse i think it should not, there's not a real reason to do so. i'm thinking (seeing how windows fs behaves) that maybe something with fleet.yml file which even in standalone serves as a agent_id storage.

some race between standalone running service and enroll performed during install.

edit: either with some fs flush or race as nromally understood where service is a slow starter and performs write to fleet.yml after enroll. but in this case it should just reload.unless it hits the spot when standalone reads file figures out it is empty, generates ID and saves it. but in between enroll generated ID

[michalpristas]

found the issue very weird

[dikshachauhan-qasource]

Hi @EricDavisX

We have revalidated windows 7 x64 machine on 7.12 BC3 build and below are our Observations:

Only with system integration in Agent policy:

• Agent is in healthy state.
• No Activity logs present
• logs on discover tab are visible.
  

Endpoint security and system integration in Agent policy:

• Agent is in healthy state.
• Activity logs present
• logs on discover tab are visible.

Build details:

BUILD 39134
COMMIT 08417cbd6c15e4c866651a7dcdfeded58845206d
Artifact link: https://staging.elastic.co/7.12.0-96914cb5/summary-7.12.0.html

Agent logs with only system integration:
elastic-agent-json.log

[michalpristas]

@dikshachauhan-qasource this is what i see locally as well