Roman/dry ingest pipeline step #3203
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rbiseck3
force-pushed
the
roman/dry-ingest-pipeline-step
branch
from
cedc9fb to
e2d3009
Compare
| @@ -61,7 +61,7 @@ def _wrap_mp(self, input_kwargs: dict) -> Any: | |||
|
|
|||
| def _set_log_level(self, log_level: int): | |||
| # Set the log level for each spawned process when using multiprocessing pool | |||
| logger.setLevel(log_level) | |||
| make_default_logger(log_level) | |||
There was a problem hiding this comment.
This causes logs to be created for every process I think. In other words, I see 2 of the same log for most things.
Though I think it is worth it if it may surface errors better.
There was a problem hiding this comment.
Fixed this in the latest commit
rbiseck3
force-pushed
the
roman/dry-ingest-pipeline-step
branch
2 times, most recently
from
90bf62c to
0ecebcc
Compare
rbiseck3
force-pushed
the
roman/dry-ingest-pipeline-step
branch
from
84506b7 to
f0ed4d7
Compare
potter-potter
approved these changes
Jun 14, 2024
rbiseck3
force-pushed
the
roman/dry-ingest-pipeline-step
branch
from
4cf96ac to
85e374b
Compare
Suspect IssuesThis pull request was deployed and Sentry observed the following issues:
Did you find this useful? React with a 👍 or 👎 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The main goal of this was to reduce the duplicate code that was being written for each ingest pipeline step to support async and not async functionality.
Additional bug fixes found and fixed:
requires_dependencieswrapper wasn't wrapping async functions correctly. This was fixed so thatasyncio.iscoroutinefunction()gets trigger correctly.