res_5f22698aMamba enjoys fast inference (5× higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences.
arXiv 2312.00752 · cs.LG cs.AI
Read on arXiv · CC BY 4.0
Claims were extracted automatically by reducer-1 on 2026-11-12. Claims are quoted in full under the paper's license.
res_5f22698aMamba enjoys fast inference (5× higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences.
clm_4a600433As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics.
res_c34e1665On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.
res_e72f11db(iii) Long context: the quality and efficiency together yield performance improvements on real data up to sequence length 1M.
clm_42246fbbWith scaling laws up to 1B parameters, we show that Mamba exceeds the performance of a large range of baselines, including very strong modern Transformer training recipes based on LLaMa (Touvron et al. 2023).
res_d6399016Structured SSMs independently map each channel (e.g. 𝐷 = 5) of an input 𝑥 to output𝑦 through a higher dimensional latent stateℎ (e.g.𝑁 = 4).
clm_810deaa5We argue that a fundamental problem of sequence modeling is compressing context into a smaller state .
clm_efbded95In turn, we propose that a fundamental principle for building sequence models isselectivity: or the context-aware ability to focus on or filter out inputs into a sequential state.
res_ed42abd3In fact, Theorem 1 is an improvement of Gu, Johnson, Goel, et al. (2021, Lemma 3.1) generalizing to the ZOH discretization and input-dependent gates (proof in Appendix C).
res_8599f778On the other hand, selective models can simply reset their state at any time to remove extraneous history, and thus their performance in principle improves monotonicly with context length (e.g. Section 4.3.2).
res_c1708067Table 1 confirms that gated architectures such as H3 and Mamba only partially improve performance, while the selection mechanism (modifying S4 to S6) easily solves this task, particularly when combined with these more powerful architectures.
clm_8820c4b3Mamba scales better than all other attention-free models and is the first to match the performance of a very strong “Transformer++” recipe that has now become standard, particularly as the sequence length grows.
res_c21a7aa8(Left) Fixing short context length 210 = 1024 and increasing size from≈ 200𝐾 to≈ 40𝑀 parameters, Mamba scales better than baselines.
res_9a5e0cdeFigure 5 (Left) shows that Mamba’s pretraining perplexity improves smoothly with model size, and that Mamba scales better than both HyenaDNA and Transformer++.
res_795fcbbcFigure 5 (Right) shows that Mamba is able to make use of longer context even up to extremely long sequences of length 1M , and its pretraining perplexity improves as the context increases.
res_c12e5f54We modify the task to be significantly more challenging by classifying between the five great apes species {human, chimpanzee, gorilla, orangutan, bonobo}, which are known to share 99% of their DNA.
clm_e8e38542Mamba improves performance over prior state-of-the-art (Sashimi) in autoregressive audio modeling, while improving up to minute-long context or million-length sequences (controlling for computation).
clm_212b79e5A small Mamba model outperforms the state-of-the-art (and much larger) GANand diffusion- based models.
res_3440735fMamba achieves 4-5× higher inference throughput than a Transformer of similar size, since without the KV cache it can use much higher batch sizes.
res_82bac6bd(Right) Inference: as a recurrent model, Mamba can achieve 5× higher throughput than Transformers.
res_d1391b60We find that • Among previous non-selective (LTI) SSMs, which are equivalent to global convolutions, performance is very similar. • Replacing the complex-valued S4 variant from previous work with a real-valued one does not affect performance much, suggesting that (at least for LM) real-valued SSMs may be a better choice when accounting for hardware efficiency. • Replacing any of these with a selective SSM (S6) significantly improves performance, validating the motivation of Section 3.
res_75756b21The more standard initializations based on S4D-Lin (Gu, Gupta, et al. 2022) perform worse than S4D-Real or a random initialization, when the SSM is selective. 𝑨𝑛 Initialization Field Perplexity 𝑨𝑛 =− 12+𝑛𝑖 Complex 9.16 𝑨𝑛 =−1/2 Real 8.85 𝑨𝑛 =−(𝑛+ 1) Real 8.71 𝑨𝑛∼ exp(N( 0, 1)) Real 8.71 • The Mamba architecture performs similarly to the H3 architecture (and seems slightly better when using a selective layer).
clm_e4e31802On language modeling, we find that simpler real-valued diagonal initializations (S4D-Real, row 3) instead of more standard complex-valued parameterizations (S4D-Lin, row 1) perform better.
res_246a9c4eOf particular note is the dramatic improvement of the selective SSM when the state size𝑁 is increased, with over a 1.0 perplexity improvement for a cost of only 1% additional parameters.
res_9131b0d91 provides a large increase in performance; increasing it further provides further improvements at the cost of a modest increase in parameters.
clm_92a758a3Increasing the SSM state dimension𝑁 , which can be viewed as an expansion factor on the dimension of the recurrent state, can significantly improve performance for a negligible cost in parameters/FLOPs, but only when 𝑩 and 𝑪 are also selective.
clm_4d72a7a7When incorporated into a simple attention-free architecture, Mamba achieves state-of-the-art results on a diverse set of domains, where it matches or exceeds the performance of strong Transformer models.
clm_c6f89983Our results suggest that Mamba is a strong candidate to be a general sequence model backbone.
clm_3fdabc38Thus, while selection mechanisms could be considered a special case of ideas such as architectural gating, hypernetworks, or data-dependence, so can an enormous range of other constructions—essentially anything with a multiplication, including standard attention mechanisms (Bahdanau, Cho, and Bengio 2015; Vaswani et al. 2017) as well—and we find it uninformative to think of them as such.
clm_4ee0813eIt is the first to generalize this formulation of linear attention to more general recurrences, which is also the basis of later architectures. • Selective S4 (J.
res_57cb002eFor the LTI H3 and Hyena models, results are reported after the 10th epoch (81920 steps) because they had converged by then and failed to improve further.
res_4884b01fOur models are: • Transformer: The standard Transformer based on GPT3 (Table 12). • Transformer++: A Transformer with an improved architecture, namely rotary positional encodings (Su et al. 2021) and SwiGLU MLP (Shazeer 2020), and the improved training recipe above. • Hyena: Interleaving a Hyena block (the H3 block with S4 replaced by a global convolution parameterized by an MLP) with standard MLP blocks.
res_2fd7bb6cThe MLP blocks have expansion factor 2 instead of 4 and the number of layers is correspondingly increased by 1.5× to preserve parameter count. • H3++: The H3 architecture with a few modifications, including (i) using the same “thin” Hyena dimensions above (ii) the improved training recipe above (iii) a linear attention head dimension of 8. • RWKV: The default RWKV model from B.
res_bed83da5Next we ablate differences between the Hyena and H3++ models, our weakest and strongest models outside of Transformer++ and Mamba, particularly to isolate the effect of training recipes. • Hyena: The Hyena block with its original architecture and GPT3 training recipe (same as Figure 4). • Hyena+: The same architecture but with the improved training recipe described above. • H3+: The same architecture as Hyena+ but with the Hyena convolution kernel swapped out for S4D convolution kernel. • H3++: The same as H3+, but with a linear attention head dimension of 8.
res_182e9680Figure 9 (Right) shows that • A large improvement is achieved by the improved training recipe, which was used for many of the models in the main Figure 4 (RetNet, H3++, Transformer++, Mamba). • The choice of the inner LTI SSM does not matter (e.g. Hyena vs. S4), consistent with findings throughout this paper. • The head dimension expansion improves performance, consistent with one of our main themes that expanded state dimension improves performance for SSMs (Section 3).
clm_f0cf4cf6Clark et al. 2018) • ARC-easy: an easy subset of ARC-challenge • WinoGrande (Sakaguchi et al. 2021) We report accuracy for LAMBADA, WinoGrande, PIQA, and ARC-easy, and accuracy normalized by sequence length for HellaSwag and ARC-challenge (since normalized accuracy is higher for almost all models for these task).
res_2155984eThe optimal Mamba learning rate was 8e-3; note that Mamba performed better than baselines with matched learning rates (2e-3), but was more stable and improved even more at higher learning rates.
res_5024a613Table 15 shows that Mamba’s memory requirement is comparable to a similar-sized Transformer with an extremely optimized implementation, and we expect further improvement in Mamba’s memory footprint in the future.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2312.00752) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.