fix(api): lock instrument cache during calibration

[fsinapi]

Overview

If a desktop app is on the same network as a robot, it will poll the GET /instruments endpoint to get the current attached instruments. This process involves loading any saved calibration offsets in the hardware controller. If the robot is currently calibrating an axis, this is a very bad thing! The calibration offsets are cleared at the beginning of calibration and must remain cleared for the entirety of calibration.

This PR adds a simple asyncio lock to disable the cache_instruments function. The calibration functions acquire the lock for their entire duration, thus preventing the robot from accidentally applying unwanted offsets.

Test Plan

Push this to a robot and ping the GET /instruments endpoint both during calibration & before/after calibration, while monitoring the logs with journalctl --follow | grep "opentrons-api\["

• During calibration, an entry should pop up saying "Instrument cache is locked, not refreshing" and calibration should complete fine
• Outside of calibration, the logs should say "Updating instrument model cache"

The endpoint should always return valid data as well.

Changelog

• Added a context manager to lock the ot3api cache_instruments function
• Acquire aforementioned context manager during any calibration functions

Review requests

• This worked fine on my testing through the ODD, do other workflows for calibration exist that may present an issue?

Risk assessment

[Laura-Danielle]

nice solution :)

[codecov]

Codecov Report

Merging #12577 (64ec956) into internal-release_0.5.0 (3c3003a) will not change coverage.
The diff coverage is n/a.

@@                   Coverage Diff                   @@
##           internal-release_0.5.0   #12577   +/-   ##
=======================================================
  Coverage                   73.59%   73.59%           
=======================================================
  Files                        2267     2267           
  Lines                       62352    62352           
  Branches                     6596     6596           
=======================================================
  Hits                        45886    45886           
  Misses                      14890    14890           
  Partials                     1576     1576           

Flag	Coverage Δ
notify-server	89.13% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
api/src/opentrons/hardware_control/ot3api.py	79.37% <ø> (ø)

[sfoster1]

Looks good, though I'm a little surprised this works without thread manager bridging