Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prevent double job status update; periodic job count miscalculation #9768

Merged
merged 7 commits into from
Jan 22, 2021

Conversation

drewbailey
Copy link
Contributor

@drewbailey drewbailey commented Jan 11, 2021

#8435 introduced atomic eval insertion when registering/de-registering jobs.

This change removes a now obsolete guard which checked if the index was
equal to the job.CreateIndex, which would empty the status.

Now that the job regisration eval insertion is atomic with the registration
this check is no longer necessary to set the job statuses correctly

By preventing the double job status calculation, we also need to ensure that job summary is properly calculated. This ensures the job summary is calculated correctly, even if the job status does not change.

fixes #8692

@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 11, 2021 17:01 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad January 11, 2021 17:01 Inactive
@drewbailey drewbailey requested review from tgross and notnoop and removed request for tgross January 11, 2021 17:02
@drewbailey drewbailey force-pushed the b/prevent-double-job-status-update branch from 2e5b57a to 1c37af8 Compare January 11, 2021 17:46
@vercel vercel bot temporarily deployed to Preview – nomad January 11, 2021 17:46 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 11, 2021 17:46 Inactive
e2e/periodic/input/simple.nomad Show resolved Hide resolved
require.NotEmpty(t, childID)

testutil.WaitForResult(func() (bool, error) {
status, err := e2eutil.JobInspectTemplate(jobID, `{{with index . 1}}{{printf "%s" .Status}}{{end}}`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really slick 👍

nomad/fsm_test.go Outdated Show resolved Hide resolved
nomad/fsm_test.go Outdated Show resolved Hide resolved
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 11, 2021 18:01 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad January 11, 2021 18:01 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad January 11, 2021 18:04 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 11, 2021 18:04 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad January 11, 2021 18:29 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 11, 2021 18:29 Inactive
@drewbailey drewbailey changed the title B/prevent double job status update prevent double job status update Jan 11, 2021
Copy link
Contributor

@notnoop notnoop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for digging into this!

@vercel vercel bot temporarily deployed to Preview – nomad January 12, 2021 16:02 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 12, 2021 16:02 Inactive
@drewbailey drewbailey force-pushed the b/prevent-double-job-status-update branch from 93a65ef to 74030f7 Compare January 15, 2021 14:11
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 15, 2021 14:11 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad January 15, 2021 14:11 Inactive
@drewbailey drewbailey requested review from notnoop and tgross January 15, 2021 14:12
@drewbailey
Copy link
Contributor Author

After the initial review, there were multiple state_store specs failing.

Removing the if index == job.CreateIndex check to prevent re-inserting the job with the same status caused the job summary not to get updated properly. Now we still potentially update the job summary even if there is no new job status.

This fixed most the failing tests, but TestStateStore_UpdateJobStability was using a modified reference instead of a copy when inserting the job a second time.

@drewbailey drewbailey force-pushed the b/prevent-double-job-status-update branch from 74030f7 to a4405a9 Compare January 19, 2021 16:31
@vercel vercel bot temporarily deployed to Preview – nomad January 19, 2021 16:31 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 19, 2021 18:00 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad January 19, 2021 18:00 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 21, 2021 13:58 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad January 21, 2021 13:58 Inactive
@drewbailey drewbailey requested a review from shoenig January 21, 2021 14:47
})

t.Run("no listeners", func(t *testing.T) {
result := gatewayBindAddresses(&structs.ConsulIngressConfigEntry{Listeners: nil})
require.Nil(t, result)
require.Empty(t, result)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shoenig requested re-review, this commit seems to deviate from how a bit of the consul gateway empty values are handled

nomad/structs/services.go Outdated Show resolved Hide resolved
#8435 introduced atomic eval
insertion iwth job (de-)registration. This change removes a now obsolete
guard which checked if the index was equal to the job.CreateIndex, which
would empty the status. Now that the job regisration eval insetion is
atomic with the registration this check is no longer necessary to set
the job statuses correctly.
fix nil assertions for empty map

rm unnecessary guard
@drewbailey drewbailey force-pushed the b/prevent-double-job-status-update branch from b1c5848 to f35eee3 Compare January 22, 2021 13:51
@vercel vercel bot temporarily deployed to Preview – nomad-storybook-and-ui January 22, 2021 13:51 Inactive
@vercel vercel bot temporarily deployed to Preview – nomad January 22, 2021 13:51 Inactive
@drewbailey drewbailey merged commit 3cb1132 into master Jan 22, 2021
@drewbailey drewbailey deleted the b/prevent-double-job-status-update branch January 22, 2021 14:18
@drewbailey drewbailey changed the title prevent double job status update prevent double job status update; periodic job count miscalculation Jan 22, 2021
backspace pushed a commit that referenced this pull request Jan 22, 2021
* Prevent Job Statuses from being calculated twice

#8435 introduced atomic eval
insertion iwth job (de-)registration. This change removes a now obsolete
guard which checked if the index was equal to the job.CreateIndex, which
would empty the status. Now that the job regisration eval insetion is
atomic with the registration this check is no longer necessary to set
the job statuses correctly.

* test to ensure only single job event for job register

* periodic e2e

* separate job update summary step

* fix updatejobstability to use copy instead of modified reference of job

* update envoygatewaybindaddresses copy to prevent job diff on null vs empty

* set ConsulGatewayBindAddress to empty map instead of nil

fix nil assertions for empty map

rm unnecessary guard
@tgross tgross added this to the 1.0.4 milestone Feb 5, 2021
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 29, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Periodic jobs duplicate pending on children job summary
4 participants