Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding BERTScore metric #424

Merged
merged 72 commits into from
Aug 8, 2021
Merged

Conversation

gagan3012
Copy link
Contributor

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

This PR adds the BERTScore metric from its implementation here: https://github.com/Tiiiger/bert_score
Fixes #365 (issue).

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@pep8speaks
Copy link

pep8speaks commented Aug 3, 2021

Hello @gagan3012! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-08-08 09:58:15 UTC

@codecov
Copy link

codecov bot commented Aug 3, 2021

Codecov Report

Merging #424 (8801e64) into master (525642d) will decrease coverage by 21.30%.
The diff coverage is 71.73%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master     #424       +/-   ##
===========================================
- Coverage   96.03%   74.73%   -21.31%     
===========================================
  Files         127      129        +2     
  Lines        4134     4179       +45     
===========================================
- Hits         3970     3123      -847     
- Misses        164     1056      +892     
Flag Coverage Δ
Linux 74.73% <71.73%> (-0.04%) ⬇️
Windows 74.73% <71.73%> (-0.04%) ⬇️
cpu 74.73% <71.73%> (-21.23%) ⬇️
gpu ?
macOS 74.73% <71.73%> (-21.23%) ⬇️
pytest 74.73% <71.73%> (-21.31%) ⬇️
python3.6 ?
python3.8 ?
python3.9 ?
torch1.3.1 ?
torch1.4.0 ?
torch1.9.0 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
torchmetrics/functional/text/bert.py 31.25% <31.25%> (ø)
torchmetrics/text/bert.py 92.30% <92.30%> (ø)
torchmetrics/__init__.py 100.00% <100.00%> (ø)
torchmetrics/functional/__init__.py 100.00% <100.00%> (ø)
torchmetrics/text/__init__.py 100.00% <100.00%> (ø)
torchmetrics/utilities/imports.py 82.92% <100.00%> (+0.42%) ⬆️
torchmetrics/utilities/distributed.py 17.24% <0.00%> (-81.04%) ⬇️
...ics/functional/classification/calibration_error.py 16.32% <0.00%> (-77.56%) ⬇️
torchmetrics/image/kid.py 20.22% <0.00%> (-76.41%) ⬇️
torchmetrics/image/inception.py 31.70% <0.00%> (-63.42%) ⬇️
... and 84 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 525642d...8801e64. Read the comment docs.

@Borda Borda added this to the v0.5 milestone Aug 3, 2021
torchmetrics/functional/text/bertscore.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/bertscore.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/bertscore.py Outdated Show resolved Hide resolved
torchmetrics/functional/text/bertscore.py Outdated Show resolved Hide resolved
@Borda Borda requested review from SkafteNicki and Borda August 6, 2021 16:44
Copy link
Member

@Borda Borda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

later we shall come with our own BERT implementation as the bert_score require heavy Transformers dependency

@Borda Borda mentioned this pull request Aug 8, 2021
@Borda Borda enabled auto-merge (squash) August 8, 2021 09:39
@Borda Borda merged commit 25e261a into Lightning-AI:master Aug 8, 2021
@Borda Borda added this to the v0.5 milestone Aug 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BERTScore Implementation
5 participants