clm_506586a7Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information.
arXiv 2104.09864 · cs.CL cs.AI cs.LG
Read on arXiv · CC BY-NC-ND 4.0
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_506586a7Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information.
clm_0dac35e2Our experiments show that it consistently overcomes its alternatives.
clm_af2b5199In this paper, we introduce a novel method, namely Rotary Position Embedding(RoPE), to leverage the positional information into the learning process of PLMS.
clm_d748b0c1We introduce a novel method, namely Rotary Position Embedding(RoPE), to leverage the positional information into the learning process of PLMS.
clm_5a03967eOur experiments show that it consistently achieves better performance compared to its alternatives.
clm_fd49546dWe establish a formal description of the position encoding problem in self-attention architecture and revisit previous works in Section (2).
clm_00b2cb85In the next section, we show that our proposed RoPE is related to this intuition from the sinusoidal function perspective.
clm_2a7c568aNext, we show that the derived approach is more interpretable by incorporating relative position information with the rotation of context representations.
clm_9a5e52e2As can be seen, RoFormer can significantly outperform BERT in three out of six datasets, and the improvements are considerable.
clm_2f6f5c8eWe demonstrate its performance with the pre-training task of language modeling.
clm_dc92463fTo better illustrate the efficacy of RoPE, we report the loss curves of the pre-training process with and without RoPE under the same settings, i.e., learning rate 1e-4, batch size … (excerpt)
res_a9795284As shown on the right plot of Figure (3), substituting RoPE into Performer leads to rapid convergence and lower loss under the same amount of training steps.
res_a06851d1To validate the performance of RoFormer on long texts, we conduct experiments on long documents whose length exceeds 512 characters.
res_8e82d171With short text cut-offs, i.e., 512, the result from RoFormer is comparable to WoBERT and is slightly better than the BERT implementation.
res_6e248852However, when increasing the maximum input text length to 1024, RoFormer outperforms WoBERT by an absolute improvement of 1.5%.
clm_c4a66983The experimental results also show that our proposed RoFormer can achieve better performance on long texts task.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2104.09864) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.