clm_3125340dWe introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.
arXiv 1810.04805 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_3125340dWe introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.
res_43a835ecIt obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD … (excerpt)
clm_9f2638dfWe argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches.
clm_aa6f8f5cThis is also in contrast to Peters et al. (2018a), which uses a shallow concatenation of independently trained left-to-right and right-to-left LMs. • We show that pre-trained representations reduce the … (excerpt)
clm_cf4cd1c2BERT is the first finetuning based representation model that achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks, outperforming many task-specific architectures. • BERT advances the state … (excerpt)
res_58917f327For example, the BERT SQuAD model can be trained in around 30 minutes on a single Cloud TPU to achieve a Dev F1 score of 91.0%.
res_e81105aaBoth BERTBASE and BERTLARGE outperform all systems on all tasks by a substantial margin, obtaining 4.5% and 7.0% respective average accuracy improvement over the prior state of the art.
res_b195c2d3For the largest and most widely reported GLUE task, MNLI, BERT obtains a 4.6% absolute accuracy improvement.
clm_15bd4697We find that BERT LARGE significantly outperforms BERTBASE across all tasks, especially those with very little training data.
res_f5fb9041Our best performing system outperforms the top leaderboard system by +1.5 F1 in ensembling and +1.3 F1 as a single system.
clm_69d641d1In fact, our single BERT model outperforms the top ensemble system in terms of F1 score.
res_b88e6fc2We exclude entries that use BERT as one of their components. tuning data, we only lose 0.1-0.4 F1, still outperforming all existing systems by a wide margin.12 4.3 SQuAD v2.0 … (excerpt)
res_11fc8ef9We observe a +5.1 F1 improvement over the previous best system.
res_0f965d5dBERT LARGE outperforms the authors’ baseline ESIM+ELMo system by +27.1% and OpenAI GPT by 8.3%.
clm_b2fc0c00We demonstrate the importance of the deep bidirectionality of BERT by evaluating two pretraining objectives using exactly the same pretraining data, fine-tuning scheme, and hyperparameters as BERTBASE : No NSP … (excerpt)
clm_81e3d5deIn Table 5, we show that removing NSP hurts performance significantly on QNLI, MNLI, and SQuAD 1.1.
res_6a82687cWe can see that larger models lead to a strict accuracy improvement across all four datasets, even for MRPC which only has 3,600 labeled training examples, and is substantially different … (excerpt)
clm_6ae76035In Section C.1 we demonstrate that MLM does converge marginally slower than a leftto-right model (which predicts every token), but the empirical improvements of the MLM model far outweigh the … (excerpt)
res_78a7b95dThe core argument of this work is that the bi-directionality and the two pretraining tasks presented in Section 3.1 account for the majority of the empirical improvements, but we do … (excerpt)
res_55d3b154To isolate the effect of these differences, we perform ablation experiments in Section 5.1 which demonstrate that the majority of the improvements are in fact coming from the two pre-training … (excerpt)
res_b183f9adThe GLUE webpage notes that there are issues with the construction of this dataset, 15 and every trained system that’s been submitted to GLUE has performed worse than the 65.1 … (excerpt)
res_8c351eb9a large amount of pre-training (128,000 words/batch * 1,000,000 steps) to achieve high fine-tuning accuracy?
res_d3fd2bf1Answer: Yes, BERT BASE achieves almost 1.0% additional accuracy on MNLI when trained on 1M steps compared to 500k steps.
res_4d59439dslower than LTR pre-training, since only 15% of words are predicted in each batch rather than every word?
clm_06df6f4dHowever, in terms of absolute accuracy the MLM model begins to outperform the LTR model almost immediately.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1810.04805) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.