MearcAll papers

arXiv 2312.00752 · cs.LG cs.AI

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Albert Gu, Tri Dao

Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.

No reproduction recordedres_5f22698a
Mamba enjoys fast inference (5× higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4a600433
As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c34e1665
On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e72f11db
(iii) Long context: the quality and efficiency together yield performance improvements on real data up to sequence length 1M.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_42246fbb
With scaling laws up to 1B parameters, we show that Mamba exceeds the performance of a large range of baselines, including very strong modern Transformer training recipes based on LLaMa (Touvron et al. 2023).

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d6399016
Structured SSMs independently map each channel (e.g. 𝐷 = 5) of an input 𝑥 to output𝑦 through a higher dimensional latent stateℎ (e.g.𝑁 = 4).

State Space Models · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_810deaa5
We argue that a fundamental problem of sequence modeling is compressing context into a smaller state .

Motivation: Selection as a Means of Compression · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_efbded95
In turn, we propose that a fundamental principle for building sequence models isselectivity: or the context-aware ability to focus on or filter out inputs into a sequential state.

Motivation: Selection as a Means of Compression · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ed42abd3
In fact, Theorem 1 is an improvement of Gu, Johnson, Goel, et al. (2021, Lemma 3.1) generalizing to the ZOH discretization and input-dependent gates (proof in Appendix C).

Connection to Gating Mechanisms · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8599f778
On the other hand, selective models can simply reset their state at any time to remove extraneous history, and thus their performance in principle improves monotonicly with context length (e.g. Section 4.3.2).

Interpretation of Selection Mechanisms · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c1708067
Table 1 confirms that gated architectures such as H3 and Mamba only partially improve performance, while the selection mechanism (modifying S4 to S6) easily solves this task, particularly when combined with these more powerful architectures.

Selective Copying · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_8820c4b3
Mamba scales better than all other attention-free models and is the first to match the performance of a very strong “Transformer++” recipe that has now become standard, particularly as the sequence length grows.

Induction Heads · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c21a7aa8
(Left) Fixing short context length 210 = 1024 and increasing size from≈ 200𝐾 to≈ 40𝑀 parameters, Mamba scales better than baselines.

DNA Modeling · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9a5e0cde
Figure 5 (Left) shows that Mamba’s pretraining perplexity improves smoothly with model size, and that Mamba scales better than both HyenaDNA and Transformer++.

Scaling: Model Size · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_795fcbbc
Figure 5 (Right) shows that Mamba is able to make use of longer context even up to extremely long sequences of length 1M , and its pretraining perplexity improves as the context increases.

Scaling: Context Length · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c12e5f54
We modify the task to be significantly more challenging by classifying between the five great apes species {human, chimpanzee, gorilla, orangutan, bonobo}, which are known to share 99% of their DNA.

Synthetic Species Classification · page 13 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e8e38542
Mamba improves performance over prior state-of-the-art (Sashimi) in autoregressive audio modeling, while improving up to minute-long context or million-length sequences (controlling for computation).

Synthetic Species Classification · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_212b79e5
A small Mamba model outperforms the state-of-the-art (and much larger) GANand diffusion- based models.

Autoregressive Speech Generation · page 14 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_3440735f
Mamba achieves 4-5× higher inference throughput than a Transformer of similar size, since without the KV cache it can use much higher batch sizes.

Speed and Memory Benchmarks · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_82bac6bd
(Right) Inference: as a recurrent model, Mamba can achieve 5× higher throughput than Transformers.

Speed and Memory Benchmarks · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d1391b60
We find that • Among previous non-selective (LTI) SSMs, which are equivalent to global convolutions, performance is very similar. • Replacing the complex-valued S4 variant from previous work with a real-valued one does not affect performance much, suggesting that (at least for LM) real-valued SSMs may be a better choice when accounting for hardware efficiency. • Replacing any of these with a selective SSM (S6) significantly improves performance, validating the motivation of Section 3.

Architecture · page 15 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_75756b21
The more standard initializations based on S4D-Lin (Gu, Gupta, et al. 2022) perform worse than S4D-Real or a random initialization, when the SSM is selective. 𝑨𝑛 Initialization Field Perplexity 𝑨𝑛 =− 12+𝑛𝑖 Complex 9.16 𝑨𝑛 =−1/2 Real 8.85 𝑨𝑛 =−(𝑛+ 1) Real 8.71 𝑨𝑛∼ exp(N( 0, 1)) Real 8.71 • The Mamba architecture performs similarly to the H3 architecture (and seems slightly better when using a selective layer).

Architecture · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_e4e31802
On language modeling, we find that simpler real-valued diagonal initializations (S4D-Real, row 3) instead of more standard complex-valued parameterizations (S4D-Lin, row 1) perform better.

Selective SSM · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_246a9c4e
Of particular note is the dramatic improvement of the selective SSM when the state size𝑁 is increased, with over a 1.0 perplexity improvement for a cost of only 1% additional parameters.

Selective SSM · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9131b0d9
1 provides a large increase in performance; increasing it further provides further improvements at the cost of a modest increase in parameters.

Discussion · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_92a758a3
Increasing the SSM state dimension𝑁 , which can be viewed as an expansion factor on the dimension of the recurrent state, can significantly improve performance for a negligible cost in parameters/FLOPs, but only when 𝑩 and 𝑪 are also selective.

Discussion · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4d72a7a7
When incorporated into a simple attention-free architecture, Mamba achieves state-of-the-art results on a diverse set of domains, where it matches or exceeds the performance of strong Transformer models.

Conclusion · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_c6f89983
Our results suggest that Mamba is a strong candidate to be a general sequence model backbone.

Conclusion · page 17 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_3fdabc38
Thus, while selection mechanisms could be considered a special case of ideas such as architectural gating, hypernetworks, or data-dependence, so can an enormous range of other constructions—essentially anything with a multiplication, including standard attention mechanisms (Bahdanau, Cho, and Bengio 2015; Vaswani et al. 2017) as well—and we find it uninformative to think of them as such.

Discussion: Selection Mechanism · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_4ee0813e
It is the first to generalize this formulation of linear attention to more general recurrences, which is also the basis of later architectures. • Selective S4 (J.

SSM Architectures · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_57cb002e
For the LTI H3 and Hyena models, results are reported after the 10th epoch (81920 steps) because they had converged by then and failed to improve further.

Synthetic Tasks · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4884b01f
Our models are: • Transformer: The standard Transformer based on GPT3 (Table 12). • Transformer++: A Transformer with an improved architecture, namely rotary positional encodings (Su et al. 2021) and SwiGLU MLP (Shazeer 2020), and the improved training recipe above. • Hyena: Interleaving a Hyena block (the H3 block with S4 replaced by a global convolution parameterized by an MLP) with standard MLP blocks.

Scaling Law Details · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2fd7bb6c
The MLP blocks have expansion factor 2 instead of 4 and the number of layers is correspondingly increased by 1.5× to preserve parameter count. • H3++: The H3 architecture with a few modifications, including (i) using the same “thin” Hyena dimensions above (ii) the improved training recipe above (iii) a linear attention head dimension of 8. • RWKV: The default RWKV model from B.

Scaling Law Details · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_bed83da5
Next we ablate differences between the Hyena and H3++ models, our weakest and strongest models outside of Transformer++ and Mamba, particularly to isolate the effect of training recipes. • Hyena: The Hyena block with its original architecture and GPT3 training recipe (same as Figure 4). • Hyena+: The same architecture but with the improved training recipe described above. • H3+: The same architecture as Hyena+ but with the Hyena convolution kernel swapped out for S4D convolution kernel. • H3++: The same as H3+, but with a linear attention head dimension of 8.

Additional Scaling Law Ablations · page 31 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_182e9680
Figure 9 (Right) shows that • A large improvement is achieved by the improved training recipe, which was used for many of the models in the main Figure 4 (RetNet, H3++, Transformer++, Mamba). • The choice of the inner LTI SSM does not matter (e.g. Hyena vs. S4), consistent with findings throughout this paper. • The head dimension expansion improves performance, consistent with one of our main themes that expanded state dimension improves performance for SSMs (Section 3).

Additional Scaling Law Ablations · page 31 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_f0cf4cf6
Clark et al. 2018) • ARC-easy: an easy subset of ARC-challenge • WinoGrande (Sakaguchi et al. 2021) We report accuracy for LAMBADA, WinoGrande, PIQA, and ARC-easy, and accuracy normalized by sequence length for HellaSwag and ARC-challenge (since normalized accuracy is higher for almost all models for these task).

Downstream Evaluation Details · page 32 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2155984e
The optimal Mamba learning rate was 8e-3; note that Mamba performed better than baselines with matched learning rates (2e-3), but was more stable and improved even more at higher learning rates.

Scaling: Model Size Details · page 33 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5024a613
Table 15 shows that Mamba’s memory requirement is comparable to a similar-sized Transformer with an extremely optimized implementation, and we expect further improvement in Mamba’s memory footprint in the future.

Efficiency Benchmark · page 36 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2312.00752) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.