Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[alpha] Addition of simple pipeline abstraction #11

Merged
merged 2 commits into from
Mar 27, 2023

Conversation

NISH1001
Copy link
Collaborator

@NISH1001 NISH1001 commented Mar 23, 2023

Major Changes

  • evalem.pipelines module is added. Now a new type/component evalem.pipelines._base.Pipeline is introduced.
  • evalem.pipelines.defaults.SimpleEvaluationPipeline is implemented which takes in a single model, input and list of evaluators to run the model through the pipeline. We can invoke .run(..) or the object itself is callable.

Minor Changes

  • tests/ test suite is refactored to make use of conftest.py format configuration for pytest

Usage

from evalem.pipelines import SimpleEvaluationPipeline
from evalem.models import TextClassificationHFPipelineWrapper
from evalem.evaluators import TextClassificationEvaluator

# can switch to any implemented wrapper
model = TextClassificationHFPipelineWrapper()

# can switch to other evaluator implementation
evaluator = TextClassificationEvaluator()

# initialize
eval_pipe = SimpleEvaluationPipeline(model=model, evaluators=evaluator)

results = pipe(inputs, references)

# or
results = pipe.run(inputs, references)

Note: The pipeline is stateless. In future we can have another implementation that can take in the mappings of input datasets to corresponding models and produce some results for all the metrics.

Now we can parametrize data, models and evaluators.
predictions = self.model(inputs, **kwargs)
return list(
map(
lambda e: e(predictions=predictions, references=references),
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might have to pass kwargs to each evaluator just in case to avoid any bug?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you mean like a catch all kwargs that any evaluator might need, within the pipeline? if that's the case, it might be too overloaded imo. then again if we are not expecting too many evaluator specific kwargs, i think it's fine

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ya. For now, it's a bit tricky because some of the interfaces seem to lack kwargs. So, passing kwargs here explicitly might have runtime error. Gonna let it be for now.

model = request.getfixturevalue(model)
predictions = model(data.get("inputs", []))
print(predictions)
assert len(predictions) == len(data.get("references", []))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if testing reference and prediction lengths is a valid test but i might be wrong

Copy link
Collaborator

@muthukumaranR muthukumaranR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not 100% sure about tests and their coverage - for now i think it looks good but leave it to @xhagrg @code-geek . rest(not much) is lgtm

@NISH1001 NISH1001 merged commit 9365268 into main Mar 27, 2023
@NISH1001 NISH1001 deleted the feature/pipeline-abstraction branch March 27, 2023 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants