Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom baseline #74

Merged
merged 3 commits into from
Aug 24, 2020
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update cli option name & plotting
  • Loading branch information
felixgwu committed Aug 17, 2020
commit fab449efe1cccf0348e789ad18a8f9dcd02fa246
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@

Automatic Evaluation Metric described in the paper [BERTScore: Evaluating Text Generation with BERT](https://arxiv.org/abs/1904.09675) (ICLR 2020).
#### News:
- The option `--rescale-with-baseline` is changed to `--rescale_with_baseline` so that it is consistent with other options.
- Updated to version 0.3.5
- Being compatible with Huggingface's transformers >=v3.0.0 and minor fixes ([#58](https://github.com/Tiiiger/bert_score/pull/58), [#66](https://github.com/Tiiiger/bert_score/pull/66), [#68](https://github.com/Tiiiger/bert_score/pull/68))
- Several improvements related to efficency ([#67](https://github.com/Tiiiger/bert_score/pull/67), [#69](https://github.com/Tiiiger/bert_score/pull/69))
@@ -116,7 +117,7 @@ where "roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0)" is the hash code
Starting from version 0.3.0, we support rescaling the scores with baseline scores

```sh
bert-score -r example/refs.txt -c example/hyps.txt --lang en --rescale-with-baseline
bert-score -r example/refs.txt -c example/hyps.txt --lang en --rescale_with_baseline
```
You will get:

6 changes: 4 additions & 2 deletions bert_score/score.py
Original file line number Diff line number Diff line change
@@ -175,7 +175,8 @@ def score(


def plot_example(
candidate, reference, model_type=None, num_layers=None, lang=None, rescale_with_baseline=False, fname=""
candidate, reference, model_type=None, num_layers=None, lang=None, rescale_with_baseline=False,
baseline_path=None, fname="",
):
"""
BERTScore metric.
@@ -239,7 +240,8 @@ def plot_example(
sim = sim[1:-1, 1:-1]

if rescale_with_baseline:
baseline_path = os.path.join(os.path.dirname(__file__), f"rescale_baseline/{lang}/{model_type}.tsv")
if baseline_path is None:
baseline_path = os.path.join(os.path.dirname(__file__), f"rescale_baseline/{lang}/{model_type}.tsv")
if os.path.isfile(baseline_path):
baselines = torch.from_numpy(pd.read_csv(baseline_path).iloc[num_layers].to_numpy())[1:].float()
sim = (sim - baselines[2].item()) / (1 - baselines[2].item())
2 changes: 1 addition & 1 deletion bert_score_cli/score.py
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ def main():
parser.add_argument("--nthreads", type=int, default=4, help="number of cpu workers (default: 4)")
parser.add_argument("--idf", action="store_true", help="BERT Score with IDF scaling")
parser.add_argument(
"--rescale-with-baseline", action="store_true", help="Rescaling the numerical score with precomputed baselines"
"--rescale_with_baseline", action="store_true", help="Rescaling the numerical score with precomputed baselines"
)
parser.add_argument("--baseline_path", default=None, type=str, help="path of custom baseline csv file")
parser.add_argument("-s", "--seg_level", action="store_true", help="show individual score of each pair")
4 changes: 3 additions & 1 deletion bert_score_cli/visualize.py
Original file line number Diff line number Diff line change
@@ -20,8 +20,9 @@ def main():
parser.add_argument("-c", "--cand", type=str, required=True, help="candidate sentence")
parser.add_argument("-f", "--file", type=str, default="visualize.png", help="name of file to save output matrix in")
parser.add_argument(
"--rescale-with-baseline", action="store_true", help="Rescaling the numerical score with precomputed baselines"
"--rescale_with_baseline", action="store_true", help="Rescaling the numerical score with precomputed baselines"
)
parser.add_argument("--baseline_path", default=None, type=str, help="path of custom baseline csv file")

args = parser.parse_args()

@@ -33,6 +34,7 @@ def main():
num_layers=args.num_layers,
fname=args.file,
rescale_with_baseline=args.rescale_with_baseline,
baseline_path=args.baseline_path,
)


2 changes: 1 addition & 1 deletion journal/rescale_baseline.md
Original file line number Diff line number Diff line change
@@ -48,7 +48,7 @@ out = bert_score.score(
and for the command-line version:
```bash
bert-score -r example/refs.txt -c example/hyps.txt \
--lang en --rescale-with-baseline
--lang en --rescale_with_baseline
```