res_1124ca89We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. … (excerpt)
arXiv 2006.11477 · cs.CL cs.LG cs.SD
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
res_1124ca89We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. … (excerpt)
res_bf03e290Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets.
res_38ef14f8When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled … (excerpt)
res_a03f58beUsing just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER.
res_3c120780Our results show that jointly learning discrete speech units with contextualized representations achieves substantially better results than fixed units learned in a prior step [4].
res_bdd8b265We also demonstrate the feasibility of ultra-low resource speech recognition: when using only 10 minutes of labeled data, our approach achieves word error rate (WER) 4.8/8.2 on the clean/other test … (excerpt)
clm_ecf3a420We set a new state of the art on TIMIT phoneme recognition as well as the 100 hour clean subset of Librispeech.
res_b0df2872Moreover, when we lower the amount of labeled data to just one hour, we still outperform the previous state of the art self-training method of [ 42] while using 100 … (excerpt)
res_a311aaffWhen we use all 960 hours of labeled data from Librispeech, then our model achieves 1.8/3.3 WER (§ 4, § 5).
res_85de6010Models are optimized by minimizing a CTC loss [14] and we apply a modified version of SpecAugment [41] by masking to time-steps and channels during training which delays overfitting and … (excerpt)
res_da3bd830The LARGE model pre-trained on LV-60k and fine-tuned on only 10 minutes of labeled data achieves a word error rate of 5.2/8.6 on the Librispeech clean/other test sets.
res_b5dd5c9fOn the 100 hour subset of Librispeech, their method achieves WER 4.2/8.6 on test-clean/other which compares to WER 2.3/5.0 with the LARGE model in a like for like setup, a … (excerpt)
res_a343765dWhen the LARGE model uses an order of magnitude less labeled data (10h labeled), then it still achieves WER 3.2/6.1, an error reduction of 24%/29% relative to iterative self-training.
res_b6c49ea1Using only a single hour of labeled data, the same model achieves WER 3.9/7.6 which improves on both test-clean and test-other by 7%/12% - with two orders of magnitude less … (excerpt)
res_4eef6cfa1.6 3.0 1.8 3.3 light data splits contain both clean and noisy data leading to better accuracy on test-other compared to test-clean.
res_5f2afdfbIncreasing model size reduces WER on all setups with the largest improvements on test-other (BASE vs. LARGE both on LS-960) and increasing the amount of unlabeled training data also leads … (excerpt)
res_caafcf92Table 2 shows that our approach achieves WER 1.8/3.3 on test-clean/other on the full Librispeech benchmark.
res_d747f8c8This is despite a weaker baseline architecture: supervised training of our architecture achieves WER 2.1/4.6 (LARGE - from scratch) compared to WER 1.9/4.1 for ContextNet [17], the baseline architecture of … (excerpt)
res_854121b6Moreover, our result is achieved without any data balancing such as [ 42].
res_55baf831Table 3 shows that our approach can achieve a new state of the art on this dataset, reducing PER by a relative 23%/29% over the next best result on the … (excerpt)
res_d2376f6bOur experiments show the large potential of pre-training on unlabeled data for speech processing: when using only 10 minutes of labeled training data, or 48 recordings of 12.5 seconds on … (excerpt)
clm_ca5876e7Our model achieves results which achieve a new state of the art on the full Librispeech benchmark for noisy speech.
res_7ba4a5ddOn the clean 100 hour Librispeech setup, wav2vec 2.0 outperforms the previous best result while using 100 times less labeled data.
clm_07316e6bWe expect performance gains by switching to a seq2seq architecture and a word piece vocabulary.
res_79f19ed1The LARGE LV-60k model achieves WER 38.3 on dev-clean and adding a Transformer language model enables to choose more likely pronunciations during the search and gives a large WER improvement … (excerpt)
res_cf7d17aaFirst, we tried to improve the continuous input and continuous target model (§ 5.4) by adding an MLP on top of the continuous target representation and we also tried to … (excerpt)
res_5e40acaeDoubling the number of relative positional embeddings to 256 also did not help.
res_18e9e90aNext, increasing the number of negatives did not result in better performance (K = 200) and sampling negatives from the entire batch of utterances hurt performance, likely because candidates from … (excerpt)
clm_1bda0cd0Predicting only up to one time step performs poorly because there is little training signal from each utterance and predicting more time steps performs better but does not significantly outperform … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2006.11477) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.