fix: fix flaky generic task pause test

[azhou-determined]

Ticket

Description

test is flaky, race between task going from PAUSING -> PAUSED and test check. so retry a few times.

Test Plan

Checklist

• Changes have been manually QA'd
• New features have been approved by the corresponding PM
• User-facing API changes have the "User-facing API Change" label
• Release notes have been added as a separate file under docs/release-notes/
  See Release Note for details.
• Licenses have been included for new code which was copied and/or modified from any external code

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Anda.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[netlify]

✅ Deploy Preview for determined-ui ready!

Name	Link
🔨 Latest commit	914fce4
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/66ed8626e6994400085ecc3f
😎 Deploy Preview	https://deploy-preview-9962--determined-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Anda.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 51.50%. Comparing base (cedfcfe) to head (914fce4).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9962      +/-   ##
==========================================
- Coverage   54.50%   51.50%   -3.01%     
==========================================
  Files        1255      710     -545     
  Lines      156729   102293   -54436     
  Branches     3601     3600       -1     
==========================================
- Hits        85423    52684   -32739     
+ Misses      71173    49476   -21697     
  Partials      133      133              

Flag	Coverage Δ
backend	43.80% <ø> (-1.37%)	⬇️
harness	44.25% <ø> (-28.50%)	⬇️
web	54.27% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 700 files with indirect coverage changes

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Anda.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[AmanuelAaron]

nit: We could use task.wait_for_task_state(), but this solution is fine

[azhou-determined]

oh didn't see this, great! thanks!

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Anda.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Anda.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails