Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/471/simplify model signature #510

Merged
merged 2 commits into from
Jul 24, 2023
Merged

Feature/471/simplify model signature #510

merged 2 commits into from
Jul 24, 2023

Conversation

blythed
Copy link
Collaborator

@blythed blythed commented Jul 24, 2023 •
edited
Loading

Description

  • Simplify descendants of Model by adding more options to base class.
    • All affected
      • transformers
      • torch
      • sklearn
    • Simplification allowed removal of
      • sentence_transformers
      • vanilla
    • Should enable removal of langchain
  • Removed ModelEnsemble
    • It really muddies the waters on how to train models
    • It will make more sense to add a .fit method to VectorIndex

Related Issue(s)

#471

Checklist

  • Change or addition is covered by unit and/or integration tests. If bugfix, there should be at least one test that fails pre-PR and passes after.
  • Classes and functions substantially affected by this PR have sphinx format docstrings added or updated.
  • If your changes introduce modifications to functionality, behavior, or usage of the project, please ensure that the documentation is updated accordingly.

Additional Notes

@blythed blythed requested a review from thejumpman2323 July 24, 2023 10:00
thejumpman2323
thejumpman2323 suggested changes Jul 24, 2023
View reviewed changes
Copy link
Contributor

@thejumpman2323 thejumpman2323 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we update the notebook for transformers as well? sentiment analysis?

README.md
- 🤸 Single developer setup for **lightweight** use-cases.
- 📈 **Scalable** setup for enterprise use-cases.

<ul>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to revert these?

superduperdb/core/vector_index.py Outdated
def _validate(
self,
db: Datalayer,
db: 'superduperdb.datalayer.base.datalayer.Datalayer', # type: ignore[name-defined]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datalayer

superduperdb/models/transformers/wrapper.py Show resolved Hide resolved
@blythed blythed requested a review from thejumpman2323 July 24, 2023 17:00
thejumpman2323
thejumpman2323 reviewed Jul 24, 2023
View reviewed changes
superduperdb/models/transformers/wrapper.py
args=self.training_arguments,
train_dataset=train_data,
eval_dataset=valid_data,
data_collator=self.collate_fn.artifact,
custom_saver=lambda: db.replace(self, upsert=True),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow neat!!!

@thejumpman2323
Copy link
Contributor

LGTM!

just check the commit names you could squash the last two.
Thanks

@blythed blythed merged commit fc906bd into superduper-io:main Jul 24, 2023
@blythed blythed deleted the feature/471/simplify-model-signature branch July 24, 2023 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thejumpman2323 thejumpman2323 thejumpman2323 requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@blythed @thejumpman2323