Week 7

Quick recap: I’m learning to build an simple ASR system with Kaldi using VoxForge data as Kaldi has a couple of pre-built recipes available. This week I will try building and installing different language modeling toolkits.

Building language models

One of the language model toolkits I’m interested in is the MIT Language Modeling Toolkit (MITLMT), I think there are a lot of useful features I want to learn. Though before I explore MITLMT, the SRI Language Modeling Toolkit (SRILM) is already packaged with Kakdi and works with the VoxForge data, so I will actually go through the installation process of SRILM first.

Notes about SRILM

SRILM is a toolkit for building and applying statistical language models (LMs), primarily for use in speech recognition, statistical tagging and segmentation, and machine translation. I’m not too familiar with N-gram language modeling, but this is the main type of language model that SRI supports.

$ local/voxforge_prepare_lm.sh
=== Building a language model ...
--- Preparing a corpus from test and train transcripts ...
/Users/erickoduniyi/kaldi-trunk/egs/voxforge/s5/local/voxforge_prepare_lm.sh: line 22: data/local/tmp/test_utt.txt: No such file or directory
/Users/erickoduniyi/kaldi-trunk/egs/voxforge/s5/local/voxforge_prepare_lm.sh: line 21: data/local/test_trans.txt: No such file or directory
/Users/erickoduniyi/kaldi-trunk/egs/voxforge/s5/local/voxforge_prepare_lm.sh: line 27: data/local/train_trans.txt: No such file or directory
/Users/erickoduniyi/kaldi-trunk/egs/voxforge/s5/local/voxforge_prepare_lm.sh: line 29: data/local/tmp/corpus.txt: No such file or directory
You appear to not have SRILM tools installed, either on your path,
or installed in /Users/erickoduniyi/kaldi-trunk/egs/voxforge/s5/../../../tools/srilm/bin/i686-m64. See tools/install_srilm.sh for installation
instructions.

I actually need to run run.sh:

$ run.sh
=== Starting VoxForge subset selection(accent: ((American)|(British)|(Australia)|(Zealand))) ...

*** VoxForge subset selection finished!
=== Starting to map anonymous users to unique IDs ...
--- Mapping the "anonymous" speakers to unique IDs ...
*** Finished mapping anonymous users!
=== Starting initial VoxForge data preparation ...
--- Making test/train data split ...
    2003 data/local/tmp/speakers_all.txt
    1983 data/local/tmp/speakers_train.txt
      20 data/local/tmp/speakers_test.txt
    2003 total
--- Preparing test_wav.scp, test_trans.txt and test.utt2spk ...
--- Preparing test.spk2utt ...
--- Preparing train_wav.scp, train_trans.txt and train.utt2spk ...
Illegal or empty gender (u) for "anonymous0107-20080905-tvm" - assuming m(ale) ...
Illegal or empty gender (p) for "anonymous0354-20091027-coy" - assuming m(ale) .

Again, this has to deal with the nature of the Voxforge data… I can’t tell if this is going to be problem…

*** Initial VoxForge data preparation finished!
=== Building a language model ...
--- Preparing a corpus from test and train transcripts ...
You appear to not have SRILM tools installed, either on your path,
or installed in /Users/erickoduniyi/kaldi-trunk/egs/voxforge/s5/../../../tools/srilm/bin/i686-m64. See tools/install_srilm.sh for installation
instructions.

Again, I actually still need to install SRILM before I can actually run voxforge_prepare_lm.sh :

$ tools/install_srilm.sh
This script cannot install SRILM in a completely automatic
way because you need to put your address in a download form.
Please download SRILM from http://www.speech.sri.com/projects/srilm/download.html

There were a set of steps that I needed to follow for installing SRILM, and the things that depend on SRILM (and a lot of other language modeling toolkits). This process actually took a lot longer than expected….

$ local/voxforge_prepare_lm.sh
=== Building a language model ...
--- Preparing a corpus from test and train transcripts ...
*** Finished building the LM model!

Okay, once that was finished This is a good place to stop, I will go back and try and install the MITLM, but using the SRILM should be fine.

Best,
EO