clm_614532bbWe argue that a missing principle is making attention algorithmsIOaware—accounting for reads and writes between levels of GPU memory.
arXiv 2205.14135 · cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_614532bbWe argue that a missing principle is making attention algorithmsIOaware—accounting for reads and writes between levels of GPU memory.
res_196319f8FlashAttention trains Transformers faster than existing baselines: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3 speedup on GPT-2 (seq. length … (excerpt)
clm_3da96736In this paper, we argue that a missing principle is making attention algorithmsIO-aware [1]—that is, carefully accounting for reads and writes to different levels of fast and slow memory (e.g., … (excerpt)
clm_7ba14fcfWe propose FlashAttention, a new attention algorithm that computes exact attention with far fewer memory accesses.
res_886a5a9fEven with the increased FLOPs due to recomputation, our algorithm bothruns faster(up to 7.6x on GPT-2 [67], Figure 1 right) anduses less memory—linear in sequence length—than standard attention, thanks to … (excerpt)
clm_b595ea5aWe prove that block-sparseFlashAttention has better IO complexity thanFlashAttention by a factor proportional to the sparsity ratio.
res_446ab88bWe train BERT-large (seq. length 512) 15% faster than the training speed record in MLPerf 1.1 [58], GPT2 (seq. length 1K) 3 faster than baseline implementations from HuggingFace [87] and … (excerpt)
res_1ebd2258We observe a 0.7 improvement in perplexity on GPT-2 and 6.4 points of lift from modeling longer sequences on long-document classification [13].FlashAttention enables the first Transformer that can achieve better-than-chance … (excerpt)
res_b6edbb9bBlock-sparseFlashAttentionenables a Transformer to scale to even longer sequences (64K), resulting in the first model that can achieve better-than-chance performance on Path-256. • Benchmarking Attention.
clm_9874b916We show how to compute exact attention with fewer HBM reads/writes and without storing large intermediate matrices for the backward pass.
res_5bbd5d23We also provide a lower bound, proving that no exact attention algorithm can 2This style of aggregation is called algebraic aggregation [33].
res_f717f695Fewer HBM accesses result in faster runtime, up to a point.Right: The runtime (for seq. length 4K) of block-sparse FlashAttention is faster thanFlashAttention by a factor proportional to the sparsity. … (excerpt)
res_e79b8dc9For typical values of𝑑 (64-128) and𝑀 (around 100KB),𝑑2 is many times smaller than𝑀, and thus FlashAttentionrequires many times fewer HBM accesses than standard implementation.
res_e2a98a72This leads to both faster execution and lower memory footprint, which we validate in Section 4.3.
res_9d0564a2In Fig. 2 (left), we see that even thoughFlashAttentionhas higher FLOP count compared to standard attention (due to recomputation in the backward pass), it has much fewer HBM accesses, resulting … (excerpt)
clm_2ed989ecWe extendFlashAttention to approximate attention: we propose block-sparseFlashAttention, whose IO complexity is smaller thanFlashAttention by a factor proportional to the sparsity.
res_7708e4e0In Fig. 2 (right), we validate that as the sparsity increases, the runtime of block-sparseFlashAttention improves proportionally.
res_b543e83fFlashAttentionscales Transformers to longer sequences, yielding higher quality.FlashAttention trains GPT-2 with context length 4K faster than Megatron trains GPT-2 with context length 1K, while achieving 0.7 better perplexity.
res_b524721eFinally,FlashAttentionyields thefirst Transformerthat can achieve better-than-random performance on the challenging Path-X task (sequence length 16K), and block-sparse FlashAttentionyields thefirst sequence modelthat we know of that can achieve better-than-random performance on … (excerpt)
res_a5a8f11eWe follow the implementation and experimental setting in Tay et al.[80]and Xiong et al.[90].3 Table 3 shows thatFlashAttention achieves up 2.4 speed-up compared to standard attention.
res_c9697f5bThe runtime and memory-efficiency ofFlashAttention allow us to increase the context length of GPT-2 by 4 while still running faster than the optimized implementation from Megatron-LM.
res_59f71b6fTable 4 shows that that GPT-2 withFlashAttentionand context length 4K is still 30% faster than GPT-2 from Megatron with context length 1K, while achieving 0.7 better perplexity.
res_8b0f3412Table 5 shows that sequence length 16K outperforms length 512 by 4.3 points on MIMIC, and that length 8K outperforms length 512 by 8.5 points on ECtHR.
res_d95df36eAdditionally, block-sparse FlashAttentionenables the Transformers to scale to sequence length 64K, achieving 63.1 accuracy4 on Path-256.
res_4b7ca3a44Path-256 requires longer sequences but has relatively shorter paths than Path-X, so it is easier to obtain a higher accuracy.
res_bc198b68We describe theFlashAttention algorithm to implement both the forward 17 and the backward passes on GPUs that reduces HBM accesses, leading to both faster runtime and smaller memory footprint.
clm_4b9e94ffWe prove the IO-complexity of block-sparseFlashAttention.
res_dc75ed95The training is stopped once the validation accuracy (for masked language modeling) reaches the target 72.0%, and the wall-clock run-time is measured.
clm_3ed7c986To be generous to the baseline methods, if we are unable to reproduce the performance of any baseline for any of the five tasks, we report the better performance from … (excerpt)
res_59d7f45fCompared to FMHA, the overall runtime ofFlashAttention is about 4% slower for sequence length 128, 8% faster for sequence length 256, and 5% faster for sequence length 512.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2205.14135) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.