MearcAll papers

arXiv 2205.14135 · cs.LG

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_614532bb
We argue that a missing principle is making attention algorithmsIOaware—accounting for reads and writes between levels of GPU memory.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_196319f8
FlashAttention trains Transformers faster than existing baselines: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3 speedup on GPT-2 (seq. length … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3da96736
In this paper, we argue that a missing principle is making attention algorithmsIO-aware [1]—that is, carefully accounting for reads and writes to different levels of fast and slow memory (e.g., … (excerpt)

Introduction · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7ba14fcf
We propose FlashAttention, a new attention algorithm that computes exact attention with far fewer memory accesses.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_886a5a9f
Even with the increased FLOPs due to recomputation, our algorithm bothruns faster(up to 7.6x on GPT-2 [67], Figure 1 right) anduses less memory—linear in sequence length—than standard attention, thanks to … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b595ea5a
We prove that block-sparseFlashAttention has better IO complexity thanFlashAttention by a factor proportional to the sparsity ratio.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_446ab88b
We train BERT-large (seq. length 512) 15% faster than the training speed record in MLPerf 1.1 [58], GPT2 (seq. length 1K) 3 faster than baseline implementations from HuggingFace [87] and … (excerpt)

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1ebd2258
We observe a 0.7 improvement in perplexity on GPT-2 and 6.4 points of lift from modeling longer sequences on long-document classification [13].FlashAttention enables the first Transformer that can achieve better-than-chance … (excerpt)

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b6edbb9b
Block-sparseFlashAttentionenables a Transformer to scale to even longer sequences (64K), resulting in the first model that can achieve better-than-chance performance on Path-256. • Benchmarking Attention.

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_9874b916
We show how to compute exact attention with fewer HBM reads/writes and without storing large intermediate matrices for the backward pass.

FlashA ttention: Algorithm, Analysis, and Extensions · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5bbd5d23
We also provide a lower bound, proving that no exact attention algorithm can 2This style of aggregation is called algebraic aggregation [33].

Analysis: IO Complexity of FlashA ttention · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f717f695
Fewer HBM accesses result in faster runtime, up to a point.Right: The runtime (for seq. length 4K) of block-sparse FlashAttention is faster thanFlashAttention by a factor proportional to the sparsity. … (excerpt)

Analysis: IO Complexity of FlashA ttention · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e79b8dc9
For typical values of𝑑 (64-128) and𝑀 (around 100KB),𝑑2 is many times smaller than𝑀, and thus FlashAttentionrequires many times fewer HBM accesses than standard implementation.

Analysis: IO Complexity of FlashA ttention · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e2a98a72
This leads to both faster execution and lower memory footprint, which we validate in Section 4.3.

Analysis: IO Complexity of FlashA ttention · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9d0564a2
In Fig. 2 (left), we see that even thoughFlashAttentionhas higher FLOP count compared to standard attention (due to recomputation in the backward pass), it has much fewer HBM accesses, resulting … (excerpt)

Analysis: IO Complexity of FlashA ttention · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2ed989ec
We extendFlashAttention to approximate attention: we propose block-sparseFlashAttention, whose IO complexity is smaller thanFlashAttention by a factor proportional to the sparsity.

Extension: Block-Sparse FlashA ttention · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7708e4e0
In Fig. 2 (right), we validate that as the sparsity increases, the runtime of block-sparseFlashAttention improves proportionally.

Extension: Block-Sparse FlashA ttention · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b543e83f
FlashAttentionscales Transformers to longer sequences, yielding higher quality.FlashAttention trains GPT-2 with context length 4K faster than Megatron trains GPT-2 with context length 1K, while achieving 0.7 better perplexity.

Experiments · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b524721e
Finally,FlashAttentionyields thefirst Transformerthat can achieve better-than-random performance on the challenging Path-X task (sequence length 16K), and block-sparse FlashAttentionyields thefirst sequence modelthat we know of that can achieve better-than-random performance on … (excerpt)

Experiments · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a5a8f11e
We follow the implementation and experimental setting in Tay et al.[80]and Xiong et al.[90].3 Table 3 shows thatFlashAttention achieves up 2.4 speed-up compared to standard attention.

Faster Models with FlashA ttention · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c9697f5b
The runtime and memory-efficiency ofFlashAttention allow us to increase the context length of GPT-2 by 4 while still running faster than the optimized implementation from Megatron-LM.

Better Models with Longer Sequences · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_59f71b6f
Table 4 shows that that GPT-2 withFlashAttentionand context length 4K is still 30% faster than GPT-2 from Megatron with context length 1K, while achieving 0.7 better perplexity.

Better Models with Longer Sequences · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8b0f3412
Table 5 shows that sequence length 16K outperforms length 512 by 4.3 points on MIMIC, and that length 8K outperforms length 512 by 8.5 points on ECtHR.

Better Models with Longer Sequences · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d95df36e
Additionally, block-sparse FlashAttentionenables the Transformers to scale to sequence length 64K, achieving 63.1 accuracy4 on Path-256.

Better Models with Longer Sequences · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4b7ca3a4
4Path-256 requires longer sequences but has relatively shorter paths than Path-X, so it is easier to obtain a higher accuracy.

Benchmarking Attention · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bc198b68
We describe theFlashAttention algorithm to implement both the forward 17 and the backward passes on GPUs that reduces HBM accesses, leading to both faster runtime and smaller memory footprint.

Algorithm Details · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4b9e94ff
We prove the IO-complexity of block-sparseFlashAttention.

Block-sparse FlashA ttention · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dc75ed95
The training is stopped once the validation accuracy (for masked language modeling) reaches the target 72.0%, and the wall-clock run-time is measured.

BERT · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3ed7c986
To be generous to the baseline methods, if we are unable to reproduce the performance of any baseline for any of the five tasks, we report the better performance from … (excerpt)

LRA details · page 27 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_59d7f45f
Compared to FMHA, the overall runtime ofFlashAttention is about 4% slower for sequence length 128, 8% faster for sequence length 256, and 5% faster for sequence length 512.

Comparison with Apex FMHA · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2205.14135) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.