MearcAll papers

arXiv 2006.11477 · cs.CL cs.LG cs.SD

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedres_1124ca89
We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bf03e290
Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_38ef14f8
When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a03f58be
Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3c120780
Our results show that jointly learning discrete speech units with contextualized representations achieves substantially better results than fixed units learned in a prior step [4].

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bdd8b265
We also demonstrate the feasibility of ultra-low resource speech recognition: when using only 10 minutes of labeled data, our approach achieves word error rate (WER) 4.8/8.2 on the clean/other test … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ecf3a420
We set a new state of the art on TIMIT phoneme recognition as well as the 100 hour clean subset of Librispeech.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b0df2872
Moreover, when we lower the amount of labeled data to just one hour, we still outperform the previous state of the art self-training method of [ 42] while using 100 … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a311aaff
When we use all 960 hours of labeled data from Librispeech, then our model achieves 1.8/3.3 WER (§ 4, § 5).

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_85de6010
Models are optimized by minimizing a CTC loss [14] and we apply a modified version of SpecAugment [41] by masking to time-steps and channels during training which delays overfitting and … (excerpt)

Fine-tuning · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_da3bd830
The LARGE model pre-trained on LV-60k and fine-tuned on only 10 minutes of labeled data achieves a word error rate of 5.2/8.6 on the Librispeech clean/other test sets.

Low-Resource Labeled Data Evaluation · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b5dd5c9f
On the 100 hour subset of Librispeech, their method achieves WER 4.2/8.6 on test-clean/other which compares to WER 2.3/5.0 with the LARGE model in a like for like setup, a … (excerpt)

Low-Resource Labeled Data Evaluation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a343765d
When the LARGE model uses an order of magnitude less labeled data (10h labeled), then it still achieves WER 3.2/6.1, an error reduction of 24%/29% relative to iterative self-training.

Low-Resource Labeled Data Evaluation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b6c49ea1
Using only a single hour of labeled data, the same model achieves WER 3.9/7.6 which improves on both test-clean and test-other by 7%/12% - with two orders of magnitude less … (excerpt)

Low-Resource Labeled Data Evaluation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4eef6cfa
1.6 3.0 1.8 3.3 light data splits contain both clean and noisy data leading to better accuracy on test-other compared to test-clean.

Low-Resource Labeled Data Evaluation · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5f2afdfb
Increasing model size reduces WER on all setups with the largest improvements on test-other (BASE vs. LARGE both on LS-960) and increasing the amount of unlabeled training data also leads … (excerpt)

Low-Resource Labeled Data Evaluation · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_caafcf92
Table 2 shows that our approach achieves WER 1.8/3.3 on test-clean/other on the full Librispeech benchmark.

High-Resource Labeled Data Evaluation on Librispeech · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d747f8c8
This is despite a weaker baseline architecture: supervised training of our architecture achieves WER 2.1/4.6 (LARGE - from scratch) compared to WER 1.9/4.1 for ContextNet [17], the baseline architecture of … (excerpt)

High-Resource Labeled Data Evaluation on Librispeech · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_854121b6
Moreover, our result is achieved without any data balancing such as [ 42].

High-Resource Labeled Data Evaluation on Librispeech · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_55baf831
Table 3 shows that our approach can achieve a new state of the art on this dataset, reducing PER by a relative 23%/29% over the next best result on the … (excerpt)

Phoneme Recognition on TIMIT · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d2376f6b
Our experiments show the large potential of pre-training on unlabeled data for speech processing: when using only 10 minutes of labeled training data, or 48 recordings of 12.5 seconds on … (excerpt)

Conclusion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_ca5876e7
Our model achieves results which achieve a new state of the art on the full Librispeech benchmark for noisy speech.

Conclusion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7ba4a5dd
On the clean 100 hour Librispeech setup, wav2vec 2.0 outperforms the previous best result while using 100 times less labeled data.

Conclusion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_07316e6b
We expect performance gains by switching to a seq2seq architecture and a word piece vocabulary.

Conclusion · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_79f19ed1
The LARGE LV-60k model achieves WER 38.3 on dev-clean and adding a Transformer language model enables to choose more likely pronunciations during the search and gives a large WER improvement … (excerpt)

Speech Recognition Error Analysis · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cf7d17aa
First, we tried to improve the continuous input and continuous target model (§ 5.4) by adding an MLP on top of the continuous target representation and we also tried to … (excerpt)

Ablations · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5e40acae
Doubling the number of relative positional embeddings to 256 also did not help.

Ablations · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_18e9e90a
Next, increasing the number of negatives did not result in better performance (K = 200) and sampling negatives from the entire batch of utterances hurt performance, likely because candidates from … (excerpt)

Ablations · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1bda0cd0
Predicting only up to one time step performs poorly because there is little training signal from each utterance and predicting more time steps performs better but does not significantly outperform … (excerpt)

Ablations · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2006.11477) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.