MearcAll papers

arXiv 2104.09864 · cs.CL cs.AI cs.LG

RoFormer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_506586a7
Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_0dac35e2
Our experiments show that it consistently overcomes its alternatives.

ABSTRACT · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_af2b5199
In this paper, we introduce a novel method, namely Rotary Position Embedding(RoPE), to leverage the positional information into the learning process of PLMS.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d748b0c1
We introduce a novel method, namely Rotary Position Embedding(RoPE), to leverage the positional information into the learning process of PLMS.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_5a03967e
Our experiments show that it consistently achieves better performance compared to its alternatives.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_fd49546d
We establish a formal description of the position encoding problem in self-attention architecture and revisit previous works in Section (2).

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_00b2cb85
In the next section, we show that our proposed RoPE is related to this intuition from the sinusoidal function perspective.

Absolute position embedding · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2a7c568a
Next, we show that the derived approach is more interpretable by incorporating relative position information with the rotation of context representations.

Relative position embedding · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9a5e52e2
As can be seen, RoFormer can significantly outperform BERT in three out of six datasets, and the improvements are considerable.

Results · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2f6f5c8e
We demonstrate its performance with the pre-training task of language modeling.

Performer with RoPE · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_dc92463f
To better illustrate the efficacy of RoPE, we report the loss curves of the pre-training process with and without RoPE under the same settings, i.e., learning rate 1e-4, batch size … (excerpt)

Implementation details · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a9795284
As shown on the right plot of Figure (3), substituting RoPE into Performer leads to rapid convergence and lower loss under the same amount of training steps.

Results · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a06851d1
To validate the performance of RoFormer on long texts, we conduct experiments on long documents whose length exceeds 512 characters.

Evaluation on Chinese Data · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8e82d171
With short text cut-offs, i.e., 512, the result from RoFormer is comparable to WoBERT and is slightly better than the BERT implementation.

Results · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6e248852
However, when increasing the maximum input text length to 1024, RoFormer outperforms WoBERT by an absolute improvement of 1.5%.

Results · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c4a66983
The experimental results also show that our proposed RoFormer can achieve better performance on long texts task.

Conclusions · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2104.09864) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.