Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(protocol-engine): rework command models for HTTP API schema #7993

Merged
merged 12 commits into from
Jun 26, 2021

Conversation

mcous
Copy link
Contributor

@mcous mcous commented Jun 23, 2021

Overview

This PR is precursor work to #7870 and #7871.

Since the Protocol Engine is the primary processor and state-tracker of protocol commands, we want the models that it uses to be broadly compatible with the HTTP API. This PR reworks all the Protocol Engine's commands to (mostly) match the Command model from existing sessions.

This is a massive PR because it turns out, there was a lot of not-single-responsibility stuff about commands in the Protocol Engine. This PR, by necessity, attempts to sequester some of those responsibilities to prevent this sort of hellish churn in the future.

Changelog

  • refactor(protocol-engine): rework command models for HTTP API schema

Review requests

This is awful and I'm sorry. When in doubt, fire-drill me in Slack for an in-person call to resolve stuff.

Important smoke tests

I've run the following sessions to completion on my robot on commit b011d29:

  • Test that all the sessions other than liveProtocol are working with the enableProtocolEngine feature flag disabled
    • calibrationCheck
    • tipLengthCalibration
    • deckCalibration
    • pipetteOffsetCalibration
    • protocol
  • Test that JSON protocols are still able to run on the engine via HTTP with the enableProtocolEngine feature flag enabled
    • Fundamentally works, with the caveats below

Unrelated issues found during testing

Not going to address any of these in this PR, but noting them to make sure we don't lose them:

  • Protocol engine does not support drop tip in fixed trash from JSON protocol
    • Fix: check for fixedTrash quirk rather than looking for the hardcoded FIXED_TRASH_ID
    • Accidentally included this fix in this PR. I regret nothing
  • Protocol engine does not home pipettes before usage
    • Fix: not sure, but seems like a hardware controller job?
    • Figure out if the engine is using the HC incorrectly

Risk assessment

Medium. This PR gets close to existing sessions endpoints by necessity. It (intentionally) breaks the non-production liveProtocol sessions in favor of the upcoming feature-flagged basic session. Also note that liveProtocol sessions are not the same thing as the protocol sessions documented in the http-api-beta. These protocol-file based sessions should continue to function.

Mitigation steps:

  • Integration tests for all production sessions are still passing
  • I've made sure that I can still run every session on my robot while running this branch

@mcous mcous added api Affects the `api` project robot server Affects the `robot-server` project protocol-engine Ticket related to the Protocol Engine project and associated HTTP APIs labels Jun 23, 2021
@mcous mcous requested review from a team as code owners June 23, 2021 14:08
@codecov
Copy link

codecov bot commented Jun 23, 2021

Codecov Report

Merging #7993 (1d40a59) into edge (a90e66d) will increase coverage by 0.07%.
The diff coverage is 80.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             edge    #7993      +/-   ##
==========================================
+ Coverage   85.88%   85.95%   +0.07%     
==========================================
  Files         348      351       +3     
  Lines       21254    21384     +130     
==========================================
+ Hits        18253    18380     +127     
- Misses       3001     3004       +3     
Impacted Files Coverage Δ
robot-server/robot_server/sessions/router.py 100.00% <ø> (ø)
...on/session_types/live_protocol/command_executor.py 88.88% <50.00%> (-11.12%) ⬇️
...ice/session/session_types/live_protocol/session.py 92.59% <100.00%> (-3.71%) ⬇️
robot-server/robot_server/sessions/engine_store.py 97.36% <100.00%> (ø)
...ns/protocol_engine/resources/resource_providers.py 88.88% <0.00%> (-11.12%) ⬇️
opentrons/protocol_engine/errors/__init__.py 92.30% <0.00%> (-7.70%) ⬇️
opentrons/protocol_engine/commands/command.py 94.44% <0.00%> (-1.48%) ⬇️
opentrons/protocol_engine/protocol_engine.py 97.36% <0.00%> (-0.68%) ⬇️
.../protocols/runner/json_proto/command_translator.py 70.93% <0.00%> (-0.34%) ⬇️
... and 27 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe4d6db...1d40a59. Read the comment docs.

@mcous mcous added the robot-svcs Falls under the purview of the Robot Services squad (formerly CPX, Core Platform Experience). label Jun 23, 2021
@mcous mcous requested review from sanni-t and SyntaxColoring June 23, 2021 16:18
# return to top if labware is fixed trash
if labware_id == FIXED_TRASH_ID:
# return to top if labware is fixed trash=
is_fixed_trash = self._labware.get_labware_has_quirk(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh oops, this is the fix I said I wasn't going to include. Accidentally committed; I think I wanna leave it in because I'm feeling lazy and it's a very small fix

Copy link
Member

@sanni-t sanni-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still making my way through the PR but posting these so I don't lose them

Copy link
Contributor

@SyntaxColoring SyntaxColoring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review. What I've read so far looks good! Just some small questions and comments. I'm up to api/src/opentrons/protocol_engine/execution/command_executor.py.

I only skimmed the drop_tip.py, load_labware.py, etc. files because their changes seemed rote.

Comment on lines +61 to +63
_ImplementationCls: Type[
AddLabwareDefinitionImplementation
] = AddLabwareDefinitionImplementation
Copy link
Contributor

@SyntaxColoring SyntaxColoring Jun 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we wanted to avoid a magically-named _ImplementationCls attribute, we could do something like this:

@command_mapper.map_to_implementation(AddLabwareDefinitionImplementation)
class AddLabwareDefinition(...)
    # ...
    # _ImplementationCls automatically added by class decorator.

This would make the link to the command_mapper module explicit.

I'm totally fine with it staying as-is for the sake of simplicity, though.

Copy link
Contributor Author

@mcous mcous Jun 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's a good idea to keep in the back pocket. For now, something like this would introduce a circular dependency between protocol_engine.commands and protocol_engine.execution that would need untangling.

What I like about this, though, is that I think the AbstractCommandImplementation base class should be coming in from protocol_engine.execution, rather than its current home of protocol_engine.commands. So I think it's worthwhile to untangle this and figure out a more explicit connection

api/src/opentrons/protocol_engine/clients/transports.py Outdated Show resolved Hide resolved
api/src/opentrons/protocol_engine/commands/command.py Outdated Show resolved Hide resolved
api/src/opentrons/protocol_engine/commands/command.py Outdated Show resolved Hide resolved
api/src/opentrons/protocol_engine/commands/command.py Outdated Show resolved Hide resolved
api/src/opentrons/protocol_engine/__init__.py Show resolved Hide resolved
Copy link
Contributor

@SyntaxColoring SyntaxColoring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another partial review. Still looks good so far. I'm just hitting the tests now.

api/src/opentrons/protocol_engine/state/commands.py Outdated Show resolved Hide resolved
api/src/opentrons/protocol_engine/state/geometry.py Outdated Show resolved Hide resolved
api/src/opentrons/protocol_engine/state/labware.py Outdated Show resolved Hide resolved
Comment on lines +5 to +11
from opentrons.protocol_engine import (
commands as pe_commands,
DeckSlotLocation,
PipetteName,
WellLocation,
WellOrigin,
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice. 👍

Copy link
Member

@sanni-t sanni-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

continuing..

Copy link
Contributor

@SyntaxColoring SyntaxColoring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remainder of review. LGTM 👍 , modulo the above comments.

Copy link
Member

@sanni-t sanni-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just the one question.
🎮



@pytest.fixture
def mock_state_store() -> MagicMock:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the reason for moving some of these frequently used fixtures out of conftest and adding them to individual test files?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This particular (MagicMock) fixture was only used in one test, so I moved it there.

In general, though, I'm finding conftest fixtures kind of hard to follow a lot of the time, and some tests become a lot more readable even if the fixtures are duplicated

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, conftest feels too implicit a lot of the time, for me. Especially when there are multiple levels of conftests.

I often think about turning fixtures into just regular functions that we import from some test_utils module. That would solve the implicitness problem.

@mcous mcous merged commit 3f74c4a into edge Jun 26, 2021
@mcous mcous deleted the engine_simplify-command-models branch June 26, 2021 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api Affects the `api` project protocol-engine Ticket related to the Protocol Engine project and associated HTTP APIs robot server Affects the `robot-server` project robot-svcs Falls under the purview of the Robot Services squad (formerly CPX, Core Platform Experience).
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants