clm_4182a7a7We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across … (excerpt)
arXiv 1802.05365 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_4182a7a7We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across … (excerpt)
clm_d461c826We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment … (excerpt)
clm_520ac84bIn this paper, we introduce a new type of deep contextualized word representation that directly addresses both challenges, can be easily integrated into existing models, and significantly improves the state … (excerpt)
clm_43dcf70eUsing intrinsic evaluations, we show that the higher-level LSTM states capture context-dependent aspects of word meaning (e.g., they can be used without modification to perform well on supervised word sense … (excerpt)
res_6a4f81a2The addition of ELMo representations alone significantly improves the state of the art in every case, including up to 20% relative error reductions.
res_cabe9dd4Finally, an analysis of both ELMo and CoVe reveals that deep representations outperform arXiv:1802.05365v2 [cs.CL] 22 Mar 2018 those derived from just the top layer of an LSTM.
res_49dfdbf6After adding ELMo to the baseline model, test set F 1 improved by 4.7% from 81.1% to 85.8%, a 24.9% relative error reduction over the baseline, and improving the overall … (excerpt)
res_e8aba076Overall, adding ELMo to the ESIM model improves accuracy by an average of 0.7% across five random seeds.
res_a1020956In our experiments with the OntoNotes coreference annotations from the CoNLL 2012 shared task (Pradhan et al., 2012), adding ELMo improved the average F1 by 3.2% from 67.2 to 70.4, … (excerpt)
res_95a34663As shown in Table 1, our ELMo enhanced biLSTM-CRF achieves 92.22% F 1 averaged over five runs.
clm_e684f958The key difference between our system and the previous state of the art from Peters et al. (2017) is that we allowed the task model to learn a weighted average … (excerpt)
res_11dd5237As shown in Sec. 5.1, using all layers instead of just the last layer improves performance across multiple tasks.
clm_5923e0f0Our baseline model is the biattentive classification network (BCN) from McCann et al. (2017), which also held the prior state-of-the-art result when augmented with CoVe embeddings.
res_9027003fReplacing CoVe with ELMo in the BCN model results in a 1.0% absolute accuracy improvement over the state of the art.
res_4a4483daSec. 5.1 shows that using deep contextual representations in downstream tasks improves performance over previous work that uses just the top layer, regardless of whether they are produced from a … (excerpt)
res_d432092eSec. 5.3 explores the different types of contextual information captured in biLMs and uses two intrinsic evaluations to show that syntactic information is better represented at lower layers while semantic … (excerpt)
res_bfacc75aFor example, in the case of SQuAD, using just the last biLM layer improves development F1 by 3.9% over the baseline.
res_bb7d4d10Averaging all biLM layers instead of using just the last layer improves F 1 another 0.3% (comparing “Last Only” to λ=1 columns), and allowing the task model to learn individual … (excerpt)
res_a92b5456For SNLI, averaging all layers withλ=1 improves development accuracy from 88.2 to 88.7% over using just the last layer.
res_7dc73a9eAs shown in Table 3, including ELMo at both the input and output layers for SNLI and SQuAD improves over just the input layer, but for SRL (and coreference resolution, … (excerpt)
clm_3fc52dffThe CoVe biLSTM layers follow a similar pattern to those from the biLM (higher overall performance at the second layer compared to the first); however, our biLM outperforms the CoVe … (excerpt)
clm_528e9a73In addition, the biLM’s representations are more transferable to WSD and POS tagging than those in CoVe, helping to illustrate why ELMo outperforms CoVe in downstream tasks.
res_8cbb695bFor example, the SRL model reaches a maximum development F 1 after 486 epochs of training without ELMo.
clm_bfcdb94eWe have introduced a general approach for learning high-quality deep context-dependent representations from biLMs, and shown large improvements when applying ELMo to a broad range of NLP tasks.
clm_c64b2ec6Through ablations and other controlled experiments, we have also confirmed that the biLM layers efficiently encode different types of syntactic and semantic information about wordsin-context, and that using all layers … (excerpt)
res_69ad75b4Overall, adding ELMo to the ESIM model improved accuracy by 0.7% establishing a new single model state-of-the-art of 88.7%, and a five member ensemble pushes the overall accuracy to 89.3%.
res_f566d1efOverall, our submission had the highest single model and ensemble results, improving the previous single model result (SAN) by 1.4% F 1 and our baseline by 4.2%.
res_da78cd66Models are trained for 500 epochs or until validation F1 does not improve for 200 epochs, whichever is sooner.
res_add85507Our single model score of 84.6 F1 represents a new state-of-the-art result on the CONLL 2012 Semantic Role Labeling task, surpassing the previous single model result by 2.9 F1 and … (excerpt)
res_d4d9f3e8Overall, we improve the single model state-of-the-art by 3.2% average F1, and our single model result improves the previous ensemble best by 1.6% F 1.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1802.05365) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.