MearcAll papers

arXiv 2309.06180 · cs.LG cs.DC

Efficient Memory Management for Large Language Model Serving with PagedAttention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, and 3 more

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedclm_4cf4b453
To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_a82b4381
On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bfbe0a48
Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_f03a0450
Indeed, our profiling results in Fig. 2 show that only 20.4% - 38.2% of the KV cache memory is used to store the actual token states in the existing systems.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_13d44c17
In this work, we buildvLLM, a high-throughput distributed LLM serving engine on top of PagedAttention that achieves near-zero waste in KV cache memory. vLLM uses block-level memory management and preemptive request scheduling that are co-designed with PagedAttention. vLLM supports popular LLMs such as GPT [5], OPT [62], and LLaMA [52] with varying sizes, including the ones exceeding the memory capacity of a single GPU.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_b6c35fea
Our evaluations on various models and workloads show that vLLM improves the LLM serving throughput by 2-4× compared to the state-of-the-art systems [31, 60], without affecting the model accuracy at all.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_252782fc
The improvements are more pronounced with longer sequences, larger models, and more complex decoding algorithms (§4.3).

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_631af717
In summary, we make the following contributions: • We identify the challenges in memory allocation in serving LLMs and quantify their impact on serving performance. • We propose PagedAttention, an attention algorithm that operates on KV cache stored in non-contiguous paged memory, which is inspired by the virtual memory and paging in OS. • We design and implement vLLM, a distributed LLM serving engine built on top of PagedAttention. • We evaluate vLLM on various scenarios and demonstrate that it substantially outperforms the previous state-of-theart solutions such as FasterTransformer [31] and Orca [60].

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_019da2e6
Then, we show how this design facilitates effective memory management for various decoding methods (§4.4) and handles the variable length input and output sequences (§4.5).

Method · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_a1207df9
Finally, we show how the system design of vLLM works in a distributed setting (§4.6).

Method · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7098a05a
As shown in the first row of Fig. 14, with a larger number of sequences to sample, vLLM brings more improvement over the Orca baselines.

Parallel Sampling and Beam Search · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1e5e4752
The improvement of vLLM over Orca (Oracle) on OPT-13B and the Alpaca dataset goes from 1.3× in basic sampling to 2.3× in beam search with a width of 6.

Parallel Sampling and Beam Search · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c29632b8
As shown in Fig. 16 (a), vLLM achieves 1.67× higher throughput than Orca (Oracle) when the one-shot prefix is shared.

Shared prefix · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_308fe5f8
Furthermore, when more examples are shared (Fig. 16 (b)), vLLM achieves 3.58× higher throughput than Orca (Oracle).

Shared prefix · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6b09f30c
As shown in Fig. 18a, this leads to 20–26% higher attention kernel latency, compared to the highly-optimized FasterTransformer implementation.

Kernel Microbenchmark · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_0f0b0848
Despite the overhead, PagedAttention makes vLLM significantly outperform FasterTransformer in end-to-end performance (§6).

Kernel Microbenchmark · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_1bcac111
In practice, we find that the block size 16 is large enough to efficiently utilize the GPU and small enough to avoid significant internal fragmentation in most workloads.

Impact of Block Size · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fa735c26
Thus, recomputation is more efficient when the block size is small, while swapping is more efficient when the block size is large, though recomputation overhead is never higher than 20% of swapping’s latency.

Comparing Recomputation and Swapping · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_117f1f49
Inspired by operating systems, we demonstrate how established techniques, such as virtual memory and copy-on-write, can be adapted to efficiently manage KV cache and handle various decoding algorithms in LLM serving.

Conclusion · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c8712085
Our experiments show that vLLM achieves 2-4× throughput improvements over the state-of-the-art systems.

Conclusion · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2309.06180) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.