Acoustic Model Discussions

Flat
problem in training stage using sohinx
User: sarvesh
Date: 5/22/2009 1:51 am
Views: 12287
Rating: 6

i am posting the log file here

 

MODULE: 00 verify training files (2009-05-22 16:30)

O.S. is case sensitive ("A" != "a").

Phones will be treated as case sensitive.

Phase 1: DICT - Checking to see if the dict and filler dict agrees with the phonelist file.

Found 241 words using 54 phones

passed

Phase 2: DICT - Checking to make sure there are not duplicate entries in the dictionary

passed

Phase 3: CTL - Check general format; utterance length (must be positive); files exist

passed

Phase 4: CTL - Checking number of lines in the transcript should match lines in control file

passed

Phase 5: CTL - Determine amount of training data, see if n_tied_states seems reasonable.

Total Hours Training: 0.0920141025641023

This is a small amount of data, no comment at this time

WARNING

Phase 6: TRANSCRIPT - Checking that all the words in the transcript are in the dictionary

Words in dictionary: 238

Words in filler dictionary: 3

passed

Phase 7: TRANSCRIPT - Checking that all the phones in the transcript are in the phonelist, and all phones in the phonelist appear at least once

passed


MODULE: 01 Train LDA transformation (2009-05-22 16:30)

Skipped (set $CFG_LDA_MLLT = 'yes' to enable)


MODULE: 02 Train MLLT transformation (2009-05-22 16:30)

Skipped (set $CFG_LDA_MLLT = 'yes' to enable)


MODULE: 05 Vector Quantization (2009-05-22 16:30)

Skipped for continuous models


MODULE: 10 Training Context Independent models for forced alignment and VTLN (2009-05-22 16:30)

Skipped: $ST::CFG_FORCEDALIGN set to 'no' in sphinx_train.cfg

Skipped: $ST::CFG_VTLN set to 'no' in sphinx_train.cfg


MODULE: 11 Force-aligning transcripts (2009-05-22 16:30)

Skipped: $ST::CFG_FORCEDALIGN set to 'no' in sphinx_train.cfg


MODULE: 12 Force-aligning data for VTLN (2009-05-22 16:30)

Skipped: $ST::CFG_VTLN set to 'no' in sphinx_train.cfg


MODULE: 20 Training Context Independent models (2009-05-22 16:30)

Phase 1: Cleaning up directories:

accumulator... logs... qmanager... models... completed

Phase 2: Flat initialize

mk_mdef_gen Log File

completed

mk_flat Log File

completed

init_gau Log File

FATAL_ERROR: "corpus.c", line 1647: Failed to get the files after 100 retries of getting MFCC(about 300 seconds)

This step had 101 ERROR messages and 0 WARNING messages. Please check the log file for details.

FAILED

 

 

so please reply me to resolve the issue involved.....

--- (Edited on 5/22/2009 1:51 am [GMT-0500] by sarvesh) ---

Re: problem in training stage using sohinx
User: nsh
Date: 5/23/2009 6:19 pm
Views: 152
Rating: 5

It fails to find the feature files in the place you pointed. Most probably you forgot to extract them or made a typo in a file name.

Check the log in logdir to get more detail

--- (Edited on 5/23/2009 6:19 pm [GMT-0500] by nsh) ---

Re: problem in training stage using sohinx
User: sarvesh
Date: 5/24/2009 3:46 am
Views: 166
Rating: 6

the log files are as

/usr/src/tutorial/san/bin/init_gau \
 -ctlfn /usr/src/tutorial/san/etc/san_train.fileids \
 -part 1 \
 -npart 1 \
 -cepdir /usr/src/tutorial/san/feat \
 -cepext mfc \
 -accumdir /usr/src/tutorial/san/bwaccumdir/san_buff_1 \
 -agc none \
 -cmn current \
 -varnorm no \
 -feat 1s_c_d_dd \
 -ceplen 13

[Switch]  [Default] [Value]
-help     no        no    
-example  no        no    
-moddeffn                 
-ts2cbfn                  
-accumdir           /usr/src/tutorial/san/bwaccumdir/san_buff_1
-meanfn                   
-fullvar  no        no    
-ctlfn              /usr/src/tutorial/san/etc/san_train.fileids
-nskip                    
-runlen                   
-part               1     
-npart              1     
-lsnfn                    
-dictfn                   
-fdictfn                  
-segdir                   
-segext   v8_seg    v8_seg
-scaleseg no        no    
-cepdir             /usr/src/tutorial/san/feat
-cepext   mfc       mfc   
-silcomp  none      none  
-cmn      current   current
-varnorm  no        no    
-agc      max       none  
-feat     1s_c_d_dd 1s_c_d_dd
-svspec                   
-ceplen   13        13    
-cepwin   0         0     
-ldafn                    
-ldadim   29        29    
INFO: corpus.c(1343): Will process all remaining utts starting at 0
INFO: init_gau.c(146): Computing 1x1x1 mean estimates
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
ERROR: "corpus.c", line 1643: MFCC read of muk1
 failed.  Retrying after sleep...
stat_retry(/usr/src/tutorial/san/feat/train/muk1
.mfc) failed
.

.

.

..

FATAL_ERROR: "corpus.c", line 1647: Failed to get the files after 100 retries of getting MFCC(about 300 seconds)
Fri May 22 16:35:22 2009

 

 

and the sphinxtrain config file is

 

# Configuration script for sphinx trainer                  -*-mode:Perl-*-

$CFG_VERBOSE = 1;        # Determines how much goes to the screen.

# These are filled in at configuration time
$CFG_DB_NAME = "san";
$CFG_BASE_DIR = "/usr/src/tutorial/san";
$CFG_SPHINXTRAIN_DIR = "/usr/src/tutorial/SphinxTrain";

# Directory containing SphinxTrain binaries
$CFG_BIN_DIR = "$CFG_BASE_DIR/bin";
$CFG_GIF_DIR = "$CFG_BASE_DIR/gifs";
$CFG_SCRIPT_DIR = "$CFG_BASE_DIR/scripts_pl";

# Experiment name, will be used to name model files and log files
$CFG_EXPTNAME = "$CFG_DB_NAME";

# Audio waveform and feature file information
$CFG_WAVFILES_DIR = "$CFG_BASE_DIR/wav";
$CFG_WAVFILE_EXTENSION = 'wav';
$CFG_WAVFILE_TYPE = 'mswav'; # one of nist, mswav, raw
$CFG_FEATFILES_DIR = "$CFG_BASE_DIR/feat";
$CFG_FEATFILE_EXTENSION = 'mfc';
$CFG_VECTOR_LENGTH = 13;

$CFG_MIN_ITERATIONS = 1;  # BW Iterate at least this many times
$CFG_MAX_ITERATIONS = 10; # BW Don't iterate more than this, somethings likely wrong.

# (none/max) Type of AGC to apply to input files
$CFG_AGC = 'none';
# (current/none) Type of cepstral mean subtraction/normalization
# to apply to input files
$CFG_CMN = 'current';
# (yes/no) Normalize variance of input files to 1.0
$CFG_VARNORM = 'no';
# (yes/no) Use letter-to-sound rules to guess pronunciations of
# unknown words (English, 40-phone specific)
$CFG_LTSOOV = 'no';
# (yes/no) Train full covariance matrices
$CFG_FULLVAR = 'no';
# (yes/no) Use diagonals only of full covariance matrices for
# Forward-Backward evaluation (recommended if CFG_FULLVAR is yes)
$CFG_DIAGFULL = 'no';

# (yes/no) Perform vocal tract length normalization in training.  This
# will result in a "normalized" model which requires VTLN to be done
# during decoding as well.
$CFG_VTLN = 'no';
# Starting warp factor for VTLN
$CFG_VTLN_START = 0.80;
# Ending warp factor for VTLN
$CFG_VTLN_END = 1.40;
# Step size of warping factors
$CFG_VTLN_STEP = 0.05;

# Directory to write queue manager logs to
$CFG_QMGR_DIR = "$CFG_BASE_DIR/qmanager";
# Directory to write training logs to
$CFG_LOG_DIR = "$CFG_BASE_DIR/logdir";
# Directory for re-estimation counts
$CFG_BWACCUM_DIR = "$CFG_BASE_DIR/bwaccumdir";
# Directory to write model parameter files to
$CFG_MODEL_DIR = "$CFG_BASE_DIR/model_parameters";

# Directory containing transcripts and control files for
# speaker-adaptive training
$CFG_LIST_DIR = "$CFG_BASE_DIR/etc";

#*******variables used in main training of models*******
$CFG_DICTIONARY     = "$CFG_LIST_DIR/$CFG_DB_NAME.dic";
$CFG_RAWPHONEFILE   = "$CFG_LIST_DIR/$CFG_DB_NAME.phone";
$CFG_FILLERDICT     = "$CFG_LIST_DIR/$CFG_DB_NAME.filler";
$CFG_LISTOFFILES    = "$CFG_LIST_DIR/${CFG_DB_NAME}_train.fileids";
$CFG_TRANSCRIPTFILE = "$CFG_LIST_DIR/${CFG_DB_NAME}_train.transcription";
$CFG_FEATPARAMS     = "$CFG_LIST_DIR/feat.params";

#*******variables used in characterizing models*******

$CFG_HMM_TYPE = '.cont.'; # Sphinx III
#$CFG_HMM_TYPE  = '.semi.'; # Sphinx II

if (($CFG_HMM_TYPE ne ".semi.") and ($CFG_HMM_TYPE ne ".cont.")) {
  die "Please choose one CFG_HMM_TYPE out of '.cont.' or '.semi.', " .
    "currently $CFG_HMM_TYPE\n";
}


if ($CFG_HMM_TYPE eq '.semi.') {
  $CFG_DIRLABEL = 'semi';
  $CFG_STATESPERHMM = 5;
  $CFG_SKIPSTATE = 'yes';
# Four (4) stream features for Sphinx II
  $CFG_FEATURE = "s2_4x";
  $CFG_NUM_STREAMS = 4;
  $CFG_INITIAL_NUM_DENSITIES = 256;
  $CFG_FINAL_NUM_DENSITIES = 256;
  die "For semi continuous models, the initial and final models have the same density"
    if ($CFG_INITIAL_NUM_DENSITIES != $CFG_FINAL_NUM_DENSITIES);
} elsif ($CFG_HMM_TYPE eq '.cont.') {
  $CFG_DIRLABEL = 'cont';
  $CFG_STATESPERHMM = 3;
  $CFG_SKIPSTATE = 'no';
# Single stream features - Sphinx 3
  $CFG_FEATURE = "1s_c_d_dd";
  $CFG_NUM_STREAMS = 1;
  $CFG_INITIAL_NUM_DENSITIES = 1;
  $CFG_FINAL_NUM_DENSITIES = 8;
  die "The initial has to be less than the final number of densities"
    if ($CFG_INITIAL_NUM_DENSITIES > $CFG_FINAL_NUM_DENSITIES);
}

# (yes/no) Train multiple-gaussian context-independent models (useful
# for alignment, use 'no' otherwise) in the models created
# specifically for forced alignment
$CFG_FALIGN_CI_MGAU = 'no';
# (yes/no) Train multiple-gaussian context-independent models (useful
# for alignment, use 'no' otherwise)
$CFG_CI_MGAU = 'no';
# Number of tied states (senones) to create in decision-tree clustering
$CFG_N_TIED_STATES = 1000;
# How many parts to run Forward-Backward estimatinon in
$CFG_NPART = 1;

# (yes/no) Train a single decision tree for all phones (actually one
# per state) (useful for grapheme-based models, use 'no' otherwise)
$CFG_CROSS_PHONE_TREES = 'no';

# Use force-aligned transcripts (if available) as input to training
$CFG_FORCEDALIGN = 'no';

# Use a specific set of models for force alignment.  If not defined,
# context-independent models for the current experiment will be used.
$CFG_FORCE_ALIGN_MDEF = "$CFG_BASE_DIR/model_architecture/$CFG_EXPTNAME.falign_ci.mdef";
if ($CFG_FALIGN_CI_MGAU eq  'yes') {
  $CFG_FORCE_ALIGN_MODELDIR = "$CFG_MODEL_DIR/$CFG_EXPTNAME.falign_ci_${CFG_DIRLABEL}_$CFG_FINAL_NUM_DENSITIES";
}
else {
  $CFG_FORCE_ALIGN_MODELDIR = "$CFG_MODEL_DIR/$CFG_EXPTNAME.falign_ci_$CFG_DIRLABEL";
}

# Use a specific dictionary and filler dictionary for force alignment.
# If these are not defined, a dictionary and filler dictionary will be
# created from $CFG_DICTIONARY and $CFG_FILLERDICT, with noise words
# removed from the filler dictionary and added to the dictionary (this
# is because the force alignment is not very good at inserting them)

# $CFG_FORCE_ALIGN_DICTIONARY = "$ST::CFG_BASE_DIR/falignout$ST::CFG_EXPTNAME.falign.dict";;
# $CFG_FORCE_ALIGN_FILLERDICT = "$ST::CFG_BASE_DIR/falignout/$ST::CFG_EXPTNAME.falign.fdict";;

# Use a particular beam width for force alignment.  The wider
# (i.e. smaller numerically) the beam, the fewer sentences will be
# rejected for bad alignment.
$CFG_FORCE_ALIGN_BEAM = 1e-60;

# Calculate an LDA/MLLT transform?
$CFG_LDA_MLLT = 'no';
# Dimensionality of LDA/MLLT output
$CFG_LDA_DIMENSION = 29;

#set convergence_ratio = 0.004
$CFG_CONVERGENCE_RATIO = 0.04;

# Queue::POSIX for multiple CPUs on a local machine
# Queue::PBS to use a PBS/TORQUE queue
$CFG_QUEUE_TYPE = "Queue";

# Name of queue to use for PBS/TORQUE
$CFG_QUEUE_NAME = "workq";

# (yes/no) Build questions for decision tree clustering automatically
$CFG_MAKE_QUESTS = "yes";
# If CFG_MAKE_QUESTS is yes, questions are written to this file.
# If CFG_MAKE_QUESTS is no, questions are read from this file.
$CFG_QUESTION_SET = "${CFG_BASE_DIR}/model_architecture/${CFG_EXPTNAME}.tree_questions";
#$CFG_QUESTION_SET = "${CFG_BASE_DIR}/linguistic_questions";

$CFG_CP_OPERATION = "${CFG_BASE_DIR}/model_architecture/${CFG_EXPTNAME}.cpmeanvar";

# This variable has to be defined, otherwise utils.pl will not load.
$CFG_DONE = 1;

return 1;
...

 

i think the directory is exactly defined, I am working on ubuntu system. i dont know why this error is happened. do reply. thank

--- (Edited on 5/24/2009 2:17 pm [GMT+0530] by sarvesh) ---

Re: problem in training stage using sohinx
User: kmaclean
Date: 5/25/2009 1:17 pm
Views: 72
Rating: 6

Hi sarvesh,

Did you convert your audio to feature files as nsh asked?  i.e. convert your "wav audio" files to "feature" files using the make_feats.plscript:?  The process is covered a bit in this post: Creating Sphinx Acoustic Models

Ken

--- (Edited on 5/25/2009 2:17 pm [GMT-0400] by kmaclean) ---

Re: problem in training stage using sohinx
User: sarvesh
Date: 5/25/2009 10:38 pm
Views: 70
Rating: 5

thanks for your reply,

yes I used audacity to record the audio at 16kHz,16bit rate in .wav format.

then generated the feature file using perl script make_feats.pl.

.mfc file was there in the directory. i dont know why it is not able to read the .mfc generated by make_feats.pl

--- (Edited on 5/26/2009 9:08 am [GMT+0530] by sarvesh) ---

Re: problem in training stage using sohinx
User: nsh
Date: 5/26/2009 3:14 am
Views: 74
Rating: 8

The issues is that your fileids file have Windows style newlines with CR+LF while sphinx expect only LF to be in the end of the line. So instead of file

a.mfc

it searches for a file

"a

.mfc"

You need to strip CR say with tr:

tr -d '\r' < oldfile.txt > newfile.txtDon't use a text editor that adds Windows style newline.  Also, I don't recommend you to use Cygwin for trainig. Install linux and dont bother us with dumb Windows problems anymore.

--- (Edited on 5/26/2009 3:14 am [GMT-0500] by nsh) ---

Re: problem in training stage using sohinx
User: Visitor
Date: 5/26/2009 7:11 am
Views: 114
Rating: 5

thank you very much.

--- (Edited on 5/26/2009 7:11 am [GMT-0500] by Visitor) ---

Re: problem in training stage using sphinx
User: sarvesh
Date: 5/27/2009 6:11 am
Views: 800
Rating: 6

while running the script 30.cd_hmm_untied/slave_convg.pl

it is not able to initialize

FATAL_ERROR: "main.c", line 1054: initialization failed

in the log file

WARN: "s3io.c", line 256: Unable to open ~/mixture_weights for reading.

there is no file that directory.

why it is not able to generate the file.

please do reply.

--- (Edited on 5/27/2009 4:41 pm [GMT+0530] by sarvesh) ---

Re: problem in training stage using sphinx
User: nsh
Date: 5/28/2009 5:00 pm
Views: 2070
Rating: 5

The real cause of this error happen before the moment you quoted. Search the logs to find it out :)

 

--- (Edited on 5/28/2009 5:00 pm [GMT-0500] by nsh) ---

Re: problem in training stage using sphinx
User: John
Date: 5/20/2010 10:22 am
Views: 404
Rating: 5

I have the same error

while running the script 30.cd_hmm_untied/slave_convg.pl it is not able to initialize FATAL_ERROR: "main.c", line 1054: initialization failed in the log file WARN: "s3io.c", line 256: Unable to open ~/mixture_weights for reading. there is no file that directory. why it is not able to generate the file.

 

Please help me

 

 

--- (Edited on 5/20/2010 10:22 am [GMT-0500] by Visitor) ---

PreviousNext