MearcAll papers

arXiv 1810.04805 · cs.CL

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_3125340d
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_43a835ec
It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9f2638df
We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_aa6f8f5c
This is also in contrast to Peters et al. (2018a), which uses a shallow concatenation of independently trained left-to-right and right-to-left LMs. • We show that pre-trained representations reduce the … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_cf4cd1c2
BERT is the first finetuning based representation model that achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks, outperforming many task-specific architectures. • BERT advances the state … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_58917f32
7For example, the BERT SQuAD model can be trained in around 30 minutes on a single Cloud TPU to achieve a Dev F1 score of 91.0%.

GLUE · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e81105aa
Both BERTBASE and BERTLARGE outperform all systems on all tasks by a substantial margin, obtaining 4.5% and 7.0% respective average accuracy improvement over the prior state of the art.

GLUE · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b195c2d3
For the largest and most widely reported GLUE task, MNLI, BERT obtains a 4.6% absolute accuracy improvement.

GLUE · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_15bd4697
We find that BERT LARGE significantly outperforms BERTBASE across all tasks, especially those with very little training data.

GLUE · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f5fb9041
Our best performing system outperforms the top leaderboard system by +1.5 F1 in ensembling and +1.3 F1 as a single system.

GLUE · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_69d641d1
In fact, our single BERT model outperforms the top ensemble system in terms of F1 score.

GLUE · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b88e6fc2
We exclude entries that use BERT as one of their components. tuning data, we only lose 0.1-0.4 F1, still outperforming all existing systems by a wide margin.12 4.3 SQuAD v2.0 … (excerpt)

GLUE · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_11fc8ef9
We observe a +5.1 F1 improvement over the previous best system.

GLUE · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0f965d5d
BERT LARGE outperforms the authors’ baseline ESIM+ELMo system by +27.1% and OpenAI GPT by 8.3%.

SW AG · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b2fc0c00
We demonstrate the importance of the deep bidirectionality of BERT by evaluating two pretraining objectives using exactly the same pretraining data, fine-tuning scheme, and hyperparameters as BERTBASE : No NSP … (excerpt)

Effect of Pre-training Tasks · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_81e3d5de
In Table 5, we show that removing NSP hurts performance significantly on QNLI, MNLI, and SQuAD 1.1.

Effect of Pre-training Tasks · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6a82687c
We can see that larger models lead to a strict accuracy improvement across all four datasets, even for MRPC which only has 3,600 labeled training examples, and is substantially different … (excerpt)

Effect of Model Size · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6ae76035
In Section C.1 we demonstrate that MLM does converge marginally slower than a leftto-right model (which predicts every token), but the empirical improvements of the MLM model far outweigh the … (excerpt)

Illustration of the Pre-training Tasks · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_78a7b95d
The core argument of this work is that the bi-directionality and the two pretraining tasks presented in Section 3.1 account for the majority of the empirical improvements, but we do … (excerpt)

Comparison of BERT, ELMo ,and · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_55d3b154
To isolate the effect of these differences, we perform ablation experiments in Section 5.1 which demonstrate that the majority of the improvements are in fact coming from the two pre-training … (excerpt)

Comparison of BERT, ELMo ,and · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b183f9ad
The GLUE webpage notes that there are issues with the construction of this dataset, 15 and every trained system that’s been submitted to GLUE has performed worse than the 65.1 … (excerpt)

Detailed Descriptions for the GLUE · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8c351eb9
a large amount of pre-training (128,000 words/batch * 1,000,000 steps) to achieve high fine-tuning accuracy?

Question: Does BERT really need such · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d3fd2bf1
Answer: Yes, BERT BASE achieves almost 1.0% additional accuracy on MNLI when trained on 1M steps compared to 500k steps.

Question: Does BERT really need such · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4d59439d
slower than LTR pre-training, since only 15% of words are predicted in each batch rather than every word?

Question: Does MLM pre-training converge · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_06df6f4d
However, in terms of absolute accuracy the MLM model begins to outperform the LTR model almost immediately.

Question: Does MLM pre-training converge · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1810.04805) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.