Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve error message in ModularPipelineError raised within _validate_datasets_exist() #3953

Open
yury-fedotov opened this issue Jun 14, 2024 · 0 comments
Labels
Issue: Feature Request New feature or improvement to existing feature

Comments

@yury-fedotov
Copy link
Contributor

yury-fedotov commented Jun 14, 2024

Description

A nice feature of modular pipelines is that if you make a mistake in mapping inputs/outputs to what nodes need, it raises a ModularPipelineError with a helpful message containing a set of catalog items you missed:

Failed to map datasets and/or parameters onto the nodes provided: <what you didn't map>...
Did you mean <those> instead?

However I think this error message can be even more useful if we detail the mismatch more. For example, to distinguish this:

  • Am I not supplying something that one of the nodes needs to pipeline wrapper?
  • Or am I supplying something to the pipeline wrapper that none of the nodes need?

As far as I understand, those 2 scenarios would now lead to the same error message. However it seems possible to make this distinction based on inputs that go to _validate_datasets_exist() function that's responsible for raising this exception.

Context

I think it might improve developer experience while building modular pipelines.

Possible Implementation

Instead of inferring a single list of mismatches:

non_existent = (inputs | outputs | parameters) - existing

Infer it as two separate sets (that sum to non_existent) called something like redundant_inputs and missing_inputs. And configure the error message reflect the difference.

@yury-fedotov yury-fedotov added the Issue: Feature Request New feature or improvement to existing feature label Jun 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue: Feature Request New feature or improvement to existing feature
Projects
Status: No status
Development

No branches or pull requests

1 participant