MearcAll papers

arXiv 1802.05365 · cs.CL

Deep contextualized word representations

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and 1 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_4182a7a7
We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d461c826
We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_520ac84b
In this paper, we introduce a new type of deep contextualized word representation that directly addresses both challenges, can be easily integrated into existing models, and significantly improves the state … (excerpt)

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_43dcf70e
Using intrinsic evaluations, we show that the higher-level LSTM states capture context-dependent aspects of word meaning (e.g., they can be used without modification to perform well on supervised word sense … (excerpt)

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6a4f81a2
The addition of ELMo representations alone significantly improves the state of the art in every case, including up to 20% relative error reductions.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cabe9dd4
Finally, an analysis of both ELMo and CoVe reveals that deep representations outperform arXiv:1802.05365v2 [cs.CL] 22 Mar 2018 those derived from just the top layer of an LSTM.

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_49dfdbf6
After adding ELMo to the baseline model, test set F 1 improved by 4.7% from 81.1% to 85.8%, a 24.9% relative error reduction over the baseline, and improving the overall … (excerpt)

Evaluation · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e8aba076
Overall, adding ELMo to the ESIM model improves accuracy by an average of 0.7% across five random seeds.

Evaluation · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a1020956
In our experiments with the OntoNotes coreference annotations from the CoNLL 2012 shared task (Pradhan et al., 2012), adding ELMo improved the average F1 by 3.2% from 67.2 to 70.4, … (excerpt)

Evaluation · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_95a34663
As shown in Table 1, our ELMo enhanced biLSTM-CRF achieves 92.22% F 1 averaged over five runs.

Evaluation · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e684f958
The key difference between our system and the previous state of the art from Peters et al. (2017) is that we allowed the task model to learn a weighted average … (excerpt)

Evaluation · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_11dd5237
As shown in Sec. 5.1, using all layers instead of just the last layer improves performance across multiple tasks.

Evaluation · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5923e0f0
Our baseline model is the biattentive classification network (BCN) from McCann et al. (2017), which also held the prior state-of-the-art result when augmented with CoVe embeddings.

Evaluation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9027003f
Replacing CoVe with ELMo in the BCN model results in a 1.0% absolute accuracy improvement over the state of the art.

Evaluation · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4a4483da
Sec. 5.1 shows that using deep contextual representations in downstream tasks improves performance over previous work that uses just the top layer, regardless of whether they are produced from a … (excerpt)

Analysis · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d432092e
Sec. 5.3 explores the different types of contextual information captured in biLMs and uses two intrinsic evaluations to show that syntactic information is better represented at lower layers while semantic … (excerpt)

Analysis · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bfacc75a
For example, in the case of SQuAD, using just the last biLM layer improves development F1 by 3.9% over the baseline.

Alternate layer weighting schemes · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bb7d4d10
Averaging all biLM layers instead of using just the last layer improves F 1 another 0.3% (comparing “Last Only” to λ=1 columns), and allowing the task model to learn individual … (excerpt)

Alternate layer weighting schemes · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a92b5456
For SNLI, averaging all layers withλ=1 improves development accuracy from 88.2 to 88.7% over using just the last layer.

Alternate layer weighting schemes · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7dc73a9e
As shown in Table 3, including ELMo at both the input and output layers for SNLI and SQuAD improves over just the input layer, but for SRL (and coreference resolution, … (excerpt)

Alternate layer weighting schemes · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3fc52dff
The CoVe biLSTM layers follow a similar pattern to those from the biLM (higher overall performance at the second layer compared to the first); however, our biLM outperforms the CoVe … (excerpt)

What information is captured by the · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_528e9a73
In addition, the biLM’s representations are more transferable to WSD and POS tagging than those in CoVe, helping to illustrate why ELMo outperforms CoVe in downstream tasks.

What information is captured by the · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8cbb695b
For example, the SRL model reaches a maximum development F 1 after 486 epochs of training without ELMo.

What information is captured by the · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_bfcdb94e
We have introduced a general approach for learning high-quality deep context-dependent representations from biLMs, and shown large improvements when applying ELMo to a broad range of NLP tasks.

Conclusion · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c64b2ec6
Through ablations and other controlled experiments, we have also confirmed that the biLM layers efficiently encode different types of syntactic and semantic information about wordsin-context, and that using all layers … (excerpt)

Conclusion · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_69ad75b4
Overall, adding ELMo to the ESIM model improved accuracy by 0.7% establishing a new single model state-of-the-art of 88.7%, and a five member ensemble pushes the overall accuracy to 89.3%.

Textual Entailment · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f566d1ef
Overall, our submission had the highest single model and ensemble results, improving the previous single model result (SAN) by 1.4% F 1 and our baseline by 4.2%.

Question Answering · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_da78cd66
Models are trained for 500 epochs or until validation F1 does not improve for 200 epochs, whichever is sooner.

Semantic Role Labeling · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_add85507
Our single model score of 84.6 F1 represents a new state-of-the-art result on the CONLL 2012 Semantic Role Labeling task, surpassing the previous single model result by 2.9 F1 and … (excerpt)

Semantic Role Labeling · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d4d9f3e8
Overall, we improve the single model state-of-the-art by 3.2% average F1, and our single model result improves the previous ensemble best by 1.6% F 1.

Coreference resolution · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1802.05365) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.