Skip to content

Latest commit

 

History

History
19 lines (15 loc) · 693 Bytes

Readme.md

File metadata and controls

19 lines (15 loc) · 693 Bytes
Raw

This directory contains the code to reproduce the human vs system-generated questions experiment (Figure 3). Included are:

  • input/{candidates,prompts,questions}.jsonl: The manually written questions collected as part of the experiments/answer-selection experiment, copied here for easier access.

Required data:

  • data/tac2008/{summaries,metrics}.jsonl

Required environments:

  • QAEVAL_ENV
  • TRANSFORMERS_ENV

To reproduce the results, run

sh experiments/question-generation/run.sh

from the root of the repository.

The plots for QAEval-EM and QAEval-F1 and the different correlation coefficients will be written to the output/plots/. Our paper used the F1/Spearman plot.