clm_61a22a8aTo better understand the challenged posed by GLUE, we conduct experiments with simple baselines and state-of-the-art sentence representation models.
arXiv 1804.07461 · cs.CL
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_61a22a8aTo better understand the challenged posed by GLUE, we conduct experiments with simple baselines and state-of-the-art sentence representation models.
clm_cabe761fWe find that unified multi-task trained models slightly outperform comparable models trained on each task separately.
res_829fded8As in MRPC, the class distribution in QQP is unbalanced (63% negative), so we report both accuracy and F1 score.
res_064cc308The average R3 score among the annotators is 0.80, much higher than any of the baseline systems described in Section 5.
res_0157e07fWe use the macro-average score as the validation metric and stop training when the learning rate drops below 10−5 or performance does not improve after 5 validation checks.
clm_b5c87022Relative to the models trained directly on the GLUE tasks, InferSent is competitive and GenSen outperforms all but the two best.
res_54ce23deOn WNLI, no model exceeds most-frequent-class guessing (65.1%) and we substitute the model predictions for the most-frequent baseline.
res_2ccc10e4Coarse Categories Overall performance is low for all models: The highest total score of 28 still denotes poor absolute performance.
clm_92bcf09dUnlike on the main benchmark, the multi-task models are almost always outperformed by their single-task counterparts.
clm_923f6ce9The models trained on the GLUE tasks largely outperform the pretrained sentence representation models, with the exception of GenSen.
clm_709d6aebWe confirm the utility of attention mechanisms and transfer learning methods such as ELMo in NLU systems, which combine to outperform the best sentence representation models on the GLUE benchmark, … (excerpt)
res_ad6e8f6fWe divide the learning rate by 5 whenever validation performance does not improve.
res_fd020d68We stop training when the learning rate drops below 10−5 or performance does not improve after 5 validation checks.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (1804.07461) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.