Skip to content

Latest commit

 

History

History
113 lines (81 loc) · 7.54 KB

README.md

File metadata and controls

113 lines (81 loc) · 7.54 KB
Raw

KLUE: Korean Language Understanding Evaluation

The KLUE is introduced to make advances in Korean NLP. Korean pre-trained language models (PLMs) have appeared to solve Korean NLP problems since PLMs have brought significant performance gains in NLP problems in other languages. Despite the proliferation of Korean language models, however, none of the proper evaluation datasets has been opened yet. The lack of such benchmark dataset limits the fair comparison between the models and further progress on model architectures.

Along with the benchmark tasks and data, we provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. We furthermore release the PLMs, KLUE-BERT and KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby facilitate future research.

See our paper for more details.

Design Principles

In designing the Korean Language Understanding Evaluation (KLUE) benchmark, we aim to make KLUE;

  1. cover diverse tasks and corpora
  2. accessible to everyone without any restriction
  3. include accurate and unambiguous annotations
  4. mitigate AI ethical issues.

Benchmark Datasets

KLUE benchmark is composed of 8 tasks:

  • Topic Classification (TC)
  • Sentence Textual Similarity (STS)
  • Natural Language Inference (NLI)
  • Named Entity Recognition (NER)
  • Relation Extraction (RE)
  • (Part-Of-Speech) + Dependency Parsing (DP)
  • Machine Reading Comprehension (MRC)
  • Dialogue State Tracking (DST)

See wiki for dataset description.

NOTE: In the paper, we describe more in detail how our 4 principles have guided creating KLUE from task selection, corpus selection, annotation protocols, determining evaluation metrics to baseline construction.

KLUE-PLMs

We have trained 2 models: KLUE-BERT and KLUE-RoBERTa.

Model Embedding size Hidden size # Layers # Heads
KLUE-BERT-base 768 768 12 12
KLUE-RoBERTa-small 768 768 6 12
KLUE-RoBERTa-base 768 768 12 12
KLUE-RoBERTa-large 1024 1024 24 16

NOTE: All the pretrained models are uploaded in Huggingface Model Hub. Check https://huggingface.co/klue.

Baseline Scores

Evaluation results of our PLMs and other baselines on KLUE benchmark. Bold shows the best performance across the models, and Italic indicates the best performance among BASE models.

Model TC STS NLI NER RE DP MRC DST
F1 Pearsons' r F1 ACC entity F1 char F1 F1 AUPRC UAS LAS EM ROUGE JGA Slot F1
mBERT-base 81.55 84.66 76.00 73.20 76.50 89.23 57.88 53.82 90.30 86.66 44.66 55.92 35.46 88.63
XLM-R-base 83.52 89.16 82.01 77.33 80.37 92.12 57.46 54.98 89.20 87.69 27.48 53.93 39.82 89.61
XLM-R-large 86.06 92.97 85.86 85.93 82.27 93.22 58.39 61.15 92.71 88.70 35.99 66.77 41.20 89.80
KR-BERT-base 84.58 88.61 81.07 77.17 74.58 90.13 62.74 60.94 89.92 87.48 48.28 58.54 45.33 90.70
koELECTRA-base 84.59 92.46 84.84 85.63 86.11 92.56 62.85 58.94 92.90 87.77 59.82 66.05 41.58 89.60
KLUE-BERT-base 85.73 90.85 82.84 81.63 83.97 91.39 66.44 66.17 89.96 88.05 62.32 68.51 46.64 91.61
KLUE-RoBERTa-small 84.98 91.54 85.16 79.33 83.65 91.14 60.89 58.96 90.04 88.14 57.32 62.70 46.62 91.44
KLUE-RoBERTa-base 85.07 92.50 85.40 84.83 84.60 91.44 67.65 68.55 93.04 88.32 68.67 73.98 47.49 91.64
KLUE-RoBERTa-large 85.69 93.35 86.63 89.17 85.00 91.86 71.13 72.98 93.48 88.36 75.58 80.59 50.22 92.23

Leaderboard

https://klue-benchmark.com

Submission Guideline

See https://aistages-prod-server-public.s3.amazonaws.com/app/Competitions/000065/data/klue_code.tar.gz

Members

Researchers

Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Youngsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Ryu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park

Advisors

Alice Oh, Jung-Woo Ha, Kyunghyun Cho

Sponsors

  • Platinum: Upstage, NAVER Clova, Google
  • Gold: Kakao Enterprise
  • Silver: Scatter Lab, Selectstar
  • Bronze: Riiid, DeepNatural, KAIST
  • Data Providers: Acrofan, KED

Organizers

  • Host: Upstage
  • Co-organizers: Naver AI Labs, NYU, KAIST
  • Research Collaborators: Kakao Enterprise, Scatter Lab, Riiid, Seoul National Univ., Yonsei Univ., Sogang Univ., Kyunghee Univ., Hanbat National Univ.

Reference

@misc{park2021klue,
      title={KLUE: Korean Language Understanding Evaluation},
      author={Sungjoon Park and Jihyung Moon and Sungdong Kim and Won Ik Cho and Jiyoon Han and Jangwon Park and Chisung Song and Junseong Kim and Yongsook Song and Taehwan Oh and Joohong Lee and Juhyun Oh and Sungwon Lyu and Younghoon Jeong and Inkwon Lee and Sangwoo Seo and Dongjun Lee and Hyunwoo Kim and Myeonghwa Lee and Seongbo Jang and Seungwon Do and Sunkyoung Kim and Kyungtae Lim and Jongwon Lee and Kyumin Park and Jamin Shin and Seonghyun Kim and Lucy Park and Alice Oh and Jungwoo Ha and Kyunghyun Cho},
      year={2021},
      eprint={2105.09680},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Creative Commons License