Skip to content

Latest commit

 

History

History
77 lines (66 loc) · 2.49 KB

README.md

File metadata and controls

77 lines (66 loc) · 2.49 KB

Analyzing analytical methods

This repository contains the instructions and code to help you reproduce the results in the following paper:

Grzegorz Chrupała, Bertrand Higy and Afra Alishahi (2020). Analyzing analytical methods: The case of phonology in neural models of spoken language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://arxiv.org/abs/2004.07070

Prerequisites

For all experiments:

pip install -r requirements.txt

RNN-VGS experiments

  • Create the input data for analyses:
python -c 'import prepare; prepare.prepare_rnn_vgs()'
  • Run analyses:
python -c 'import analyze; analyze.analyze_rnn_vgs()'
  • Plot main results (Figure 2)
python -c 'import analyze; analyze.plot_rnn_vgs()'
  • Plot Figure 4:
python -c 'import analyze; analyze.plot_r2_partial()'
  • Plot Figure 5
python -c 'import analyze; analyze.plot_pooled_feature_std()

RNN-ASR experiments

  • Create the input data for analyses:
python -c 'import prepare; prepare.prepare_rnn_asr()'
  • Run analyses:
python -c 'import analyze; analyze.analyze_rnn_asr()'
  • Plot main results (Figure 3)
python -c 'import analyze; analyze.plot_rnn_asr()'

Transformer-ASR experiments

The transformer-ASR model is a transformer model trained with ESPnet on Librispeech. The forked version of the code we used to extract the activations is available from https://github.com/bhigy/espnet/tree/phoneme-repr. The README at the root of the repository provides instructions for installation, while details about activations extraction can be found under egs/librispeech/asr1/README.md. Alternativaly, the activations are also provided as part of the data you downloaded in the prerequisites, under data/activations/transformer-asr.

  • Copy the activations you extracted or the ones we provide under data/out/transformer-asr.
  • Create the input data for analyses:
python -c 'import prepare; prepare.prepare_transformer_asr()'
  • Run analyses:
python -c 'import analyze; analyze.analyze_transformer_asr()'
  • Plot main results (Figure 1)
python -c 'import analyze; analyze.plot_transformer_asr()'