Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] lost update while concurrent synthetics private location API calls #190801

Closed
biscout42 opened this issue Aug 20, 2024 · 2 comments · Fixed by #195874
Closed

[BUG] lost update while concurrent synthetics private location API calls #190801

biscout42 opened this issue Aug 20, 2024 · 2 comments · Fixed by #195874
Assignees
@shahzad31
Labels
bug Fixes for quality problems that affect the customer experience Team:obs-ux-management Observability Management User Experience Team

Comments

@biscout42
Copy link

Kibana version:
8.14.3 , 8.15.0

Elasticsearch version:
8.14.3 , 8.15.0

Original install method (e.g. download page, yum, from source, etc.):
Docker compose

Describe the bug:
Two concurrent private location create requests leads to first of them is lost.

In case of concurrent tests which create private locations, one of the locations is lost. Here is an example of kibana debug log:

[2024-08-20T11:17:06.931+00:00][DEBUG][elasticsearch.query.data] 201 - 220.0B
PUT /.kibana_8.15.0/_doc/synthetics-privates-locations%3Asynthetics-privates-locations-singleton?refresh=wait_for&require_alias=true
{"synthetics-privates-locations":{"locations":[{"label":"monitor-pll-kuooipy79h3a3gnw2wydly","id":"fbb70664-2851-473f-880a-42ca4f8269a1","agentPolicyId":"fbb70664-2851-473f-880a-42ca4f8269a1","isServiceManaged":false}]},"type":"synthetics-privates-locations","references":[],"managed":false,"coreMigrationVersion":"8.8.0","typeMigrationVersion":"10.1.0","updated_at":"2024-08-20T11:17:06.635Z","created_at":"2024-08-20T11:17:06.635Z"}
[2024-08-20T11:17:06.934+00:00][DEBUG][http.server.response] POST /api/synthetics/private_locations 200 319ms - 210.0B
[2024-08-20T11:17:06.934+00:00][DEBUG][elasticsearch.query.data] 200 - 220.0B
PUT /.kibana_8.15.0/_doc/synthetics-privates-locations%3Asynthetics-privates-locations-singleton?refresh=wait_for&require_alias=true
{"synthetics-privates-locations":{"locations":[{"label":"pl-test-label","id":"0e6c28d6-148a-47cb-b6fd-def77d3a795c","agentPolicyId":"0e6c28d6-148a-47cb-b6fd-def77d3a795c","tags":["a","b"],"isServiceManaged":false,"geo":{"lat":42.42,"lon":-42.42}}]},"type":"synthetics-privates-locations","references":[],"managed":false,"coreMigrationVersion":"8.8.0","typeMigrationVersion":"10.1.0","updated_at":"2024-08-20T11:17:06.634Z","created_at":"2024-08-20T11:17:06.634Z"}
[2024-08-20T11:17:06.935+00:00][DEBUG][http.server.response] POST /api/synthetics/private_locations 200 321ms - 239.0B

After that follow-up request for private location monitor-pll-kuooipy79h3a3gnw2wydly fails. In this example it happened with an attempt to create a monitor for this private location (see error below), and I have another tests, where follow-up request GET /api/synthetics/private_locations/locationId fails with 404.
Error example:

{
 "statusCode": 400,
 "error": "Bad Request",
 "message": "Invalid locations specified. Private Location(s) 'monitor-pll-kuooipy79h3a3gnw2wydly' not found. Available private locations are 'pl-test-label'"
}

API client sees correct 200 reply after creating monitor-pll-kuooipy79h3a3gnw2wydly private location in the example above (via POST, with all its details in the body)

Steps to reproduce:

  1. Run several POST /api/synthetics/private_locations requests concurrently.
  2. Run GET /api/synthetics/private_locations/location_id for each of created locations
  3. One of GET requests fails.

Expected behavior:
All private locations exist in Kibana in case of concurrent POST requests.

Provide logs and/or server output (if relevant):
see above

Any additional context:
The root cause could be in our private location API implementation, which uses a single saved object to keep track of private locations.
@biscout42 biscout42 added the bug Fixes for quality problems that affect the customer experience label Aug 20, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Aug 20, 2024
@biscout42 biscout42 mentioned this issue Aug 20, 2024
Merged
biscout42 added a commit to elastic/terraform-provider-elasticstack that referenced this issue Aug 21, 2024
@biscout42
fix: move all resources in one namespace for tcp monitor acc tests (#717
1920946 
)

* fix: move all resources in one namespace for tcp monitor acc tests
* add mutex in kibana API client and move tests to same package to run test in single golang process

due-to elastic/kibana#190801
@mholttech mholttech mentioned this issue Aug 28, 2024
Merged
@jughosta jughosta added the Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team label Sep 2, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Sep 2, 2024
@smith
Copy link
Contributor

smith commented Sep 3, 2024

@jughosta Synthetics is owned by @elastic/obs-ux-management-team. I'll add the team label.

@smith smith added Team:obs-ux-management Observability Management User Experience Team and removed Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels Sep 3, 2024
@shahzad31 shahzad31 self-assigned this Oct 11, 2024
@shahzad31 shahzad31 mentioned this issue Oct 17, 2024
Merged
shahzad31 added a commit that referenced this issue Nov 7, 2024
@shahzad31 @kibanamachine
[Synthetics] Handle private locations simultaneous edits !! (#195874)
96c9b5b 
## Summary

Fixes #190801 !!

Handle private locations simultaneous edits !! 

Registered a new saved object to handle private locations properly.
Instead of using a singleton, now each private location will be
represented by it's own saved object.

### Existing private locations
When we are doing any write operation, we migrate them to new kind of
saved object and remove the legacy saved object type.


### Testing

- Create multiple private locations on main 
- Switch to branch and create few more locations
- Make sure all private locations are editable, deleteable and have been
migrated to new saved object types in the course

---------

Co-authored-by: kibanamachine <[email protected]>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 7, 2024
@shahzad31
[Synthetics] Handle private locations simultaneous edits !! (elastic#…
164b9f3 
…195874)

## Summary

Fixes elastic#190801 !!

Handle private locations simultaneous edits !!

Registered a new saved object to handle private locations properly.
Instead of using a singleton, now each private location will be
represented by it's own saved object.

### Existing private locations
When we are doing any write operation, we migrate them to new kind of
saved object and remove the legacy saved object type.

### Testing

- Create multiple private locations on main
- Switch to branch and create few more locations
- Make sure all private locations are editable, deleteable and have been
migrated to new saved object types in the course

---------

Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit 96c9b5b)
kibanamachine added a commit that referenced this issue Nov 7, 2024
@kibanamachine @shahzad31
[8.x] [Synthetics] Handle private locations simultaneous edits !! (#1…
afb795d 
…95874) (#199387)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Synthetics] Handle private locations simultaneous edits !!
(#195874)](#195874)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT
[{"author":{"name":"Shahzad","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-07T21:31:09Z","message":"[Synthetics]
Handle private locations simultaneous edits !! (#195874)\n\n##
Summary\r\n\r\nFixes #190801
!!\r\n\r\nHandle private locations simultaneous edits !!
\r\n\r\nRegistered a new saved object to handle private locations
properly.\r\nInstead of using a singleton, now each private location
will be\r\nrepresented by it's own saved object.\r\n\r\n### Existing
private locations\r\nWhen we are doing any write operation, we migrate
them to new kind of\r\nsaved object and remove the legacy saved object
type.\r\n\r\n\r\n### Testing\r\n\r\n- Create multiple private locations
on main \r\n- Switch to branch and create few more locations\r\n- Make
sure all private locations are editable, deleteable and have
been\r\nmigrated to new saved object types in the
course\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"96c9b5b5d0a82c6e4b0a9b86fc1ab4dd9fa4d707","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-management"],"title":"[Synthetics]
Handle private locations simultaneous edits
!!","number":195874,"url":"https://github.com/elastic/kibana/pull/195874","mergeCommit":{"message":"[Synthetics]
Handle private locations simultaneous edits !! (#195874)\n\n##
Summary\r\n\r\nFixes #190801
!!\r\n\r\nHandle private locations simultaneous edits !!
\r\n\r\nRegistered a new saved object to handle private locations
properly.\r\nInstead of using a singleton, now each private location
will be\r\nrepresented by it's own saved object.\r\n\r\n### Existing
private locations\r\nWhen we are doing any write operation, we migrate
them to new kind of\r\nsaved object and remove the legacy saved object
type.\r\n\r\n\r\n### Testing\r\n\r\n- Create multiple private locations
on main \r\n- Switch to branch and create few more locations\r\n- Make
sure all private locations are editable, deleteable and have
been\r\nmigrated to new saved object types in the
course\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"96c9b5b5d0a82c6e4b0a9b86fc1ab4dd9fa4d707"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195874","number":195874,"mergeCommit":{"message":"[Synthetics]
Handle private locations simultaneous edits !! (#195874)\n\n##
Summary\r\n\r\nFixes #190801
!!\r\n\r\nHandle private locations simultaneous edits !!
\r\n\r\nRegistered a new saved object to handle private locations
properly.\r\nInstead of using a singleton, now each private location
will be\r\nrepresented by it's own saved object.\r\n\r\n### Existing
private locations\r\nWhen we are doing any write operation, we migrate
them to new kind of\r\nsaved object and remove the legacy saved object
type.\r\n\r\n\r\n### Testing\r\n\r\n- Create multiple private locations
on main \r\n- Switch to branch and create few more locations\r\n- Make
sure all private locations are editable, deleteable and have
been\r\nmigrated to new saved object types in the
course\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"96c9b5b5d0a82c6e4b0a9b86fc1ab4dd9fa4d707"}}]}]
BACKPORT-->

Co-authored-by: Shahzad <[email protected]>
mbondyra pushed a commit to mbondyra/kibana that referenced this issue Nov 8, 2024
@shahzad31 @kibanamachine @mbondyra
[Synthetics] Handle private locations simultaneous edits !! (elastic#…
33afe5f 
…195874)

## Summary

Fixes elastic#190801 !!

Handle private locations simultaneous edits !! 

Registered a new saved object to handle private locations properly.
Instead of using a singleton, now each private location will be
represented by it's own saved object.

### Existing private locations
When we are doing any write operation, we migrate them to new kind of
saved object and remove the legacy saved object type.


### Testing

- Create multiple private locations on main 
- Switch to branch and create few more locations
- Make sure all private locations are editable, deleteable and have been
migrated to new saved object types in the course

---------

Co-authored-by: kibanamachine <[email protected]>
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this issue Nov 8, 2024
@shahzad31 @kibanamachine @CAWilson94
[Synthetics] Handle private locations simultaneous edits !! (elastic#…
8488119 
…195874)

## Summary

Fixes elastic#190801 !!

Handle private locations simultaneous edits !! 

Registered a new saved object to handle private locations properly.
Instead of using a singleton, now each private location will be
represented by it's own saved object.

### Existing private locations
When we are doing any write operation, we migrate them to new kind of
saved object and remove the legacy saved object type.


### Testing

- Create multiple private locations on main 
- Switch to branch and create few more locations
- Make sure all private locations are editable, deleteable and have been
migrated to new saved object types in the course

---------

Co-authored-by: kibanamachine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shahzad31 shahzad31

Labels
bug Fixes for quality problems that affect the customer experience Team:obs-ux-management Observability Management User Experience Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Synthetics] Handle private locations simultaneous edits !! shahzad31/kibana
5 participants
@smith @jughosta @shahzad31 @elasticmachine @biscout42