fix: set max slots and checkpoint gc policy should comply with config policies

[amandavialva01]

Ticket

CM-590

Description

This PR fixes a couple of issues:

• det e set max-slots to comply with invariant config and constraints
• det e set gc-policy to comply with invariant configs

Test Plan

Checklist

• Changes have been manually QA'd
• New features have been approved by the corresponding PM
• User-facing API changes have the "User-facing API Change" label
• Release notes have been added as a separate file under docs/release-notes/
  See Release Note for details.
• Licenses have been included for new code which was copied and/or modified from any external code

[codecov]

Codecov Report

Attention: Patch coverage is 88.88889% with 7 lines in your changes missing coverage. Please review.

Project coverage is 54.77%. Comparing base (782f7a0) to head (ae4a0c6).
Report is 21 commits behind head on main.

Files with missing lines	Patch %	Lines
...ternal/configpolicy/postgres_task_config_policy.go	86.36%	3 Missing ⚠️
master/internal/api_experiment.go	92.59%	2 Missing ⚠️
master/internal/configpolicy/utils.go	80.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10140      +/-   ##
==========================================
+ Coverage   54.71%   54.77%   +0.06%     
==========================================
  Files        1266     1266              
  Lines      159970   159988      +18     
  Branches     3662     3662              
==========================================
+ Hits        87525    87635     +110     
+ Misses      72312    72220      -92     
  Partials      133      133              

Flag	Coverage Δ
backend	46.19% <88.88%> (+0.18%)	⬆️
harness	72.56% <ø> (ø)
web	54.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
master/internal/experiment.go	35.93% <100.00%> (+2.15%)	⬆️
master/internal/api_experiment.go	61.23% <92.59%> (+4.31%)	⬆️
master/internal/configpolicy/utils.go	73.20% <80.00%> (-0.85%)	⬇️
...ternal/configpolicy/postgres_task_config_policy.go	90.12% <86.36%> (-1.02%)	⬇️

... and 2 files with indirect coverage changes

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	ae4a0c6
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/6723830747dbe90008f4eaeb

[stoksc]

patch experiment really sketches me out. e.SetGroupWeight(*newResources.Weight) and all call e.db.SaveExperimentConfig(e.ID, e.activeConfig) and so does the endpoint itself. i feel like there are definitely calls where i could make the exp in memory state and db state inconsistent..?

[amandavialva01]

patch experiment really sketches me out. e.SetGroupWeight(*newResources.Weight) and all call e.db.SaveExperimentConfig(e.ID, e.activeConfig) and so does the endpoint itself. i feel like there are definitely calls where i could make the exp in memory state and db state inconsistent..?

hmm yea good point.
Is the solution then to just add that func into setMaxSlots? instead of moving it in the patch handler? its the only of the three (max-slots, priority, weight) that doesn't have this call

[amandavialva01]

I wanna note that technically, users can still override the name and description of an experiment with det e set name and det e set description, but I think that should be allowed, @kkunapuli wdyt?

[kkunapuli]

I wanna note that technically, users can still override the name and description of an experiment with det e set name and det e set description, but I think that should be allowed, @kkunapuli wdyt?

Yeah, I agree.