Phone based LF-MMI training

[csukuangfj]

Phone based LF-MMI training is easier than wordpiece based LF-MMI training,
so I would like to get a working version of MMI training based on phone first.

[danpovey]

Cool!

…

On Mon, Aug 23, 2021 at 12:59 PM Fangjun Kuang ***@***.***> wrote: Phone based LF-MMI training is easier than wordpiece based LF-MMI training, so I would like to get a working version of MMI training based on phone first. ------------------------------ You can view, comment on, or merge this pull request online at: #19 Commit Summary - Add phone based LF-MMI training. - Fix style issues. File Changes - *M* .flake8 <https://github.com/k2-fsa/icefall/pull/19/files#diff-6951dbb399883798a226c1fb496fdb4183b1ab48865e75eddecf6ceb6cf46442> (6) - *A* egs/librispeech/ASR/conformer_mmi_phone/__init__.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-48d0a50579efcae2f1ded145cac99b4c9f4c4074d0407c284419f24a3b38867e> (0) - *A* egs/librispeech/ASR/conformer_mmi_phone/asr_datamodule.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-07f485ccffa7805fd4e90ff6910f1dcaeecf261dc1404a5d44f58c92dc43b234> (1) - *A* egs/librispeech/ASR/conformer_mmi_phone/conformer.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-7abd308c291ace689eb985735ac303fede33c9b155a92b25c62b6e27d06471b7> (933) - *A* egs/librispeech/ASR/conformer_mmi_phone/decode.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-18dcf06d7df3fb129e73fcc832f384dfc6decf53753b60041b458288af96a08d> (548) - *A* egs/librispeech/ASR/conformer_mmi_phone/embedding.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-070e9a3465805fe9c1d8138e03c3bddf27868a6d7a4ea519ecd27075bfcaf7c1> (221) - *A* egs/librispeech/ASR/conformer_mmi_phone/pretrained.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-756ec7bc9500e2db14ee8cd8e0ea8be10cd52b900dcd8582d8678a34c2c6fc1a> (350) - *A* egs/librispeech/ASR/conformer_mmi_phone/subsampling.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-43d9e1fb3ccce2b5587c5969815ecbc54088055110e5a232388c954cc47fe355> (144) - *A* egs/librispeech/ASR/conformer_mmi_phone/test_subsampling.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-847d96e1d8a2c603dabb15bf5600216cfd8c67f3e30d32f4646dc423cb013747> (33) - *A* egs/librispeech/ASR/conformer_mmi_phone/test_transformer.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-c4f429ab028756487c357472bba3d22b7e4852fef04a329fbfa210158add40b3> (89) - *A* egs/librispeech/ASR/conformer_mmi_phone/train.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-8c3e71b1cc311d6f1cb452379c568809c01119a5acf3a9dbcadb259defc839a4> (839) - *A* egs/librispeech/ASR/conformer_mmi_phone/transformer.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-4eb6febe83d78f58ba9f05ffbb801c37529806b041f74893822509a2006c13e7> (1006) - *A* egs/librispeech/ASR/local/add_silence_to_transcript.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-ddcedf6fc50db3f07956e2f8778eccbcb7b55da32323602ecccad031e402da4b> (110) - *A* egs/librispeech/ASR/local/convert_transcript_to_corpus.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-c3ddfa386abb124ae7d804c037ee4710c988a8120de5b8d3183d952fe08790b5> (103) - *M* egs/librispeech/ASR/local/prepare_lang.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-260b810232f85030f8e58acb14fbaf7504a8375c2ecf5f2fa2eafae41ba2c1a8> (41) - *M* egs/librispeech/ASR/prepare.sh <https://github.com/k2-fsa/icefall/pull/19/files#diff-e8583288bc01a92697d12648ba2bebe584ccdf85d81a25f45b11769284b575cc> (54) - *A* icefall/mmi.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-5eeaa51281eacc00cf201b3ba98fdac8f58a0fa97c7a1361e4b34ac66ab2e09f> (222) - *A* icefall/mmi_graph_compiler.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-5fa0daf235576c3e14d22f5e102f9a1d303d331df8405f345a601b3a84c5818a> (184) - *A* icefall/shared/make_kn_lm.py <https://github.com/k2-fsa/icefall/pull/19/files#diff-3ce9de57d973f1028ffe0fd4fd1a4ddbe1d7ee8cf35faaa50e308a932fafa520> (377) Patch Links: - https://github.com/k2-fsa/icefall/pull/19.patch - https://github.com/k2-fsa/icefall/pull/19.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#19>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFLO5Y65R6WKLIGHRL5GLT6HIU3ANCNFSM5CTYV6IQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[csukuangfj]

Here are the results I have so far with this pull request:

HLG decoding (1best, no LM rescoring)

( with model averaging from epoch-43.pt to epoch-49.pt)

[test-clean-no_rescore] %WER 3.69% [1941 / 52576, 263 ins, 137 del, 1541 sub ]
[test-other-no_rescore] %WER 7.35% [3849 / 52343, 522 ins, 269 del, 3058 sub ]

HLG decoding (1best) + 4-gram LM rescoring

( with model averaging from epoch-42.pt to epoch-49.pt)

[test-clean-lm_scale_1.1] %WER 3.33% [1753 / 52576, 309 ins, 92 del, 1352 sub ]
[test-other-lm_scale_1.2] %WER 6.77% [3542 / 52343, 601 ins, 207 del, 2734 sub ]

---

The plans for the following days are:

(1) Training with attention decoder.

Unlike training with BPE units, where a word has only one pronunciation, there may have multiple pronunciations
for a word in phone-based units. My plan is to choose only the first pronunciation if there is more than one.

(2) Instead of training a TDNN-LSTM model as a force alignment model, integrate the changes from lhotse
lhotse-speech/lhotse#379 to use the alignment information contained in the supervision.

(3) Replace phone-based MMI training with BPE based MMI training

[danpovey]

OK, that's great.
I'm hoping that once we incorporate the alignment information, we'll find that the BPE-based LF-MMI training starts to converge. Fingers crossed!

[csukuangfj]

Now it supports using attention-decoder along with MMI training.

Tensorboard log of the below command is available at

https://tensorboard.dev/experiment/Wd049TyrRdyvOkcOiD32FQ/#scalars&_smoothingWeight=0

export CUDA_VISIBLE_DEVICES="0,1,2,3"

./conformer_mmi_phone/train.py \
  --full-libri 1 \
  --max-duration 200 \
  --bucketing-sampler 1 \
  --concatenate-cuts 0 \
  --world-size 4 \
  --bucketing-sampler 1

[danpovey]

Wow, great progress!