Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] refactored bulk update tags retry #147594

Merged
merged 13 commits into from
Dec 20, 2022
Merged

[Fleet] refactored bulk update tags retry #147594

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
0f92d07
refactored bulk update tags retry
juliaElastic Dec 15, 2022
9bfc39f
Merge branch 'main' into fix/update-tags
juliaElastic Dec 15, 2022
c098438
fixed test
juliaElastic Dec 15, 2022
ed60158
fix action results on retry
juliaElastic Dec 16, 2022
2c3660d
fix and test
juliaElastic Dec 16, 2022
3db4cb6
use doc count in /action_status for udpate tags
juliaElastic Dec 16, 2022
0edb432
write error results on last retry with version conflicts
juliaElastic Dec 16, 2022
4c25056
removed hosted agents error from udpate tags without kuery too
juliaElastic Dec 16, 2022
1e12eee
fixed types
juliaElastic Dec 16, 2022
b003d9b
fixed integration test
juliaElastic Dec 19, 2022
c5eb223
Merge branch 'main' into fix/update-tags
kibanamachine Dec 19, 2022
53e670b
simplified logic to use retry for all kuery actions
juliaElastic Dec 19, 2022
7fe37bb
fix review comments
juliaElastic Dec 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 13 additions & 28 deletions x-pack/plugins/fleet/server/services/agents/update_agent_tags.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,7 @@ import type { ElasticsearchClient, SavedObjectsClientContract } from '@kbn/core/
import type { Agent } from '../../types';
import { AgentReassignmentError } from '../../errors';

import { SO_SEARCH_LIMIT } from '../../constants';

import { getAgentDocuments, getAgentsByKuery } from './crud';
import { getAgentDocuments } from './crud';
import type { GetAgentsOptions } from '.';
import { searchHitToAgent } from './helpers';
import { UpdateAgentTagsActionRunner, updateTagsBatch } from './update_agent_tags_action_runner';
Expand All @@ -30,7 +28,7 @@ export async function updateAgentTags(
tagsToRemove: string[]
): Promise<{ actionId: string }> {
const outgoingErrors: Record<Agent['id'], Error> = {};
let givenAgents: Agent[] = [];
const givenAgents: Agent[] = [];

if ('agentIds' in options) {
const givenAgentsResults = await getAgentDocuments(esClient, options.agentIds);
Expand All @@ -44,30 +42,17 @@ export async function updateAgentTags(
}
}
} else if ('kuery' in options) {
const batchSize = options.batchSize ?? SO_SEARCH_LIMIT;
const res = await getAgentsByKuery(esClient, {
kuery: options.kuery,
showInactive: options.showInactive ?? false,
page: 1,
perPage: batchSize,
});
if (res.total <= batchSize) {
givenAgents = res.agents;
} else {
return await new UpdateAgentTagsActionRunner(
esClient,
soClient,
{
...options,
batchSize,
total: res.total,
kuery: options.kuery,
tagsToAdd,
tagsToRemove,
},
{ pitId: '' }
).runActionAsyncWithRetry();
}
return await new UpdateAgentTagsActionRunner(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simplified the logic to use retry for all update tags kuery actions
as reported here, the version conflict happened even with less than 10k agents, which didn't retry before

Could reproduce in an ECE instance by adding a tag on 5k horde agents and getting this response from bulk API:

{"statusCode":500,"error":"Internal Server Error","message":"version conflict of 1865 agents"}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested on pr cloud deployment that 5000 agents update tags is done successfully (async)
image

esClient,
soClient,
{
...options,
kuery: options.kuery,
tagsToAdd,
tagsToRemove,
},
{ pitId: '' }
).runActionAsyncWithRetry();
}

return await updateTagsBatch(soClient, esClient, givenAgents, outgoingErrors, {
Expand Down
25 changes: 2 additions & 23 deletions x-pack/test/fleet_api_integration/apis/agents/update_agent_tags.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,25 +68,6 @@ export default function (providerContext: FtrProviderContext) {
expect(agent2data.body.item.tags).to.eql(['existingTag']);
});

it('should allow to update tags of multiple agents by kuery', async () => {
await supertest
.post(`/api/fleet/agents/bulk_update_agent_tags`)
.set('kbn-xsrf', 'xxx')
.send({
agents: 'active: true',
tagsToAdd: ['newTag'],
tagsToRemove: ['existingTag'],
})
.expect(200);

const { body } = await supertest.get(`/api/fleet/agents`).set('kbn-xsrf', 'xxx');
expect(body.total).to.eql(4);
body.items.forEach((agent: any) => {
expect(agent.tags.includes('newTag')).to.be(true);
expect(agent.tags.includes('existingTag')).to.be(false);
});
});

async function pollResult(
actionId: string,
nbAgentsAck: number,
Expand Down Expand Up @@ -115,15 +96,14 @@ export default function (providerContext: FtrProviderContext) {
});
}

it('should bulk update tags of multiple agents by kuery in batches - add', async () => {
it('should bulk update tags of multiple agents by kuery - add', async () => {
const { body: actionBody } = await supertest
.post(`/api/fleet/agents/bulk_update_agent_tags`)
.set('kbn-xsrf', 'xxx')
.send({
agents: 'active: true',
tagsToAdd: ['newTag'],
tagsToRemove: [],
batchSize: 3,
})
.expect(200);

Expand All @@ -139,15 +119,14 @@ export default function (providerContext: FtrProviderContext) {
await pollResult(actionId, 4, verifyActionResult);
});

it('should bulk update tags of multiple agents by kuery in batches - remove', async () => {
it('should bulk update tags of multiple agents by kuery - remove', async () => {
const { body: actionBody } = await supertest
.post(`/api/fleet/agents/bulk_update_agent_tags`)
.set('kbn-xsrf', 'xxx')
.send({
agents: 'active: true',
tagsToAdd: [],
tagsToRemove: ['existingTag'],
batchSize: 3,
})
.expect(200);

Expand Down