Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix module loading bug in instance tracker class #1902

Merged
merged 3 commits into from
Oct 20, 2023
Merged

Conversation

cosmicBboy
Copy link
Contributor

@cosmicBboy cosmicBboy commented Oct 19, 2023

TL;DR

This PR fixes an issue reported here where pyflyte and other executables were trying to be loaded by the tracker.py module when finding trackable instances.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • [ ] Smoke tested
  • [ ] Unit tests added (will be added subsequently)
  • [ ] Code documentation added
  • [ ] Any pending items have an associated Issue

Complete description

The eager workflow PR introduced a bug that caused breakages in @dynamic workflows and agents. The underlying reason is that eager workflows need to be able to load task/dynamic/workflow functions as InstanceTrackingMeta instances so that it can execute them via FlyteRemote. The issue is that, before 1579, InstanceTrackingMeta did not support loading entities defined in the __main__ module.

This PR handles a module loading error that was being raised because the tracker module was trying to load the pyflyte and other pyflyte-* executables.

Tracking Issue

flyteorg/flyte#4245

Follow-up issue

#1892

Signed-off-by: Niels Bantilan <[email protected]>
@codecov
Copy link

codecov bot commented Oct 19, 2023

Codecov Report

Attention: 18 lines in your changes are missing coverage. Please review.

Comparison is base (85b8d29) 94.97% compared to head (de87a3c) 55.08%.
Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #1902       +/-   ##
===========================================
- Coverage   94.97%   55.08%   -39.90%     
===========================================
  Files         135      301      +166     
  Lines        6115    22447    +16332     
  Branches        0     3371     +3371     
===========================================
+ Hits         5808    12364     +6556     
- Misses        307     9918     +9611     
- Partials        0      165      +165     
Files Coverage Δ
flytekit/core/tracker.py 47.05% <5.26%> (ø)

... and 167 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@cosmicBboy cosmicBboy changed the title [wip] modify instance tracker class Fix module loading bug in instance tracker class Oct 19, 2023
Signed-off-by: Niels Bantilan <[email protected]>
Copy link
Member

@pingsutw pingsutw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Let's update the PR description before merging it

@pingsutw pingsutw mentioned this pull request Oct 20, 2023
8 tasks
@eapolinario eapolinario merged commit dc9d26b into master Oct 20, 2023
68 of 70 checks passed
@cosmicBboy cosmicBboy deleted the fix-tracker branch October 27, 2023 15:40
ringohoffman pushed a commit to ringohoffman/flytekit that referenced this pull request Nov 24, 2023
* modify instance tracker class

Signed-off-by: Niels Bantilan <[email protected]>

* handle error in _get_module_from_main

Signed-off-by: Niels Bantilan <[email protected]>

* add more comments

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants