MearcAll papers

arXiv 2111.00396 · cs.LG

Efficiently Modeling Long Sequences with Structured State Spaces

Albert Gu, Karan Goel, Christopher Ré

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_f7a2e79e
We propose the Structured State Space sequence model (S4) based on a new parameterization for the SSM, and show that it can be computed much more efficiently than prior approaches … (excerpt)

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_eb5cc028
Although theoretically efficient algorithms for the LSSL were proposed, we show that these are numerically unstable.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ef72c2f7
Combining these two ideas, we show that the low-rank term can be corrected by the Woodbury identity while the normal term can be diagonalized stably, ultimately reducing to a well-studied … (excerpt)

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6b74b49b
On the LRA benchmark for efficient sequence models, S4 is as fast as all baselines while outperforming them by 20+ points on average.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ee8c74c7
With no architectural changes, S4 surpasses Speech CNNs on speech classification, outperforms the specialized Informer model on time-series forecasting problems, and matches a 2-D ResNet on sequential CIFAR with over … (excerpt)

Introduction · page 3 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_ab5fe7cf
For example, the LSSL found that simply modifying an SSM from a random matrix A to equation (2) improved its performance on the sequential MNIST benchmark from 60% to 98%.

Addressing Long-Range Dependencies with HiPPO · page 4 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7bd4d6ca
In Appendix B, we prove that it is also numerically unstable for related reasons.

Motivation: Diagonalization · page 5 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_342b850f
S4 substantially advances the SoTA, outperforming all baselines on all tasks and averaging 80 .48% compared to less than 60% for every baseline.

Learning Long Range Dependencies · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c2839ff4
S4 achieves 98.3% accuracy, higher than all baselines that use the 100× shorter MFCC features, and validates that a powerful LRD model is able to extract more information from the … (excerpt)

Learning Long Range Dependencies · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_172c9e2e
We show that alternative models to Transformers can still be competitive in these settings.

S4 as a General Sequence Model · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_6ceed908
On both CIFAR-10 and WikiText-103, we report the throughput of various models at generation time, with S4 around 60 × faster than a vanilla Transformer on both tasks (details in … (excerpt)

S4 as a General Sequence Model · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_40939656
Without re-training, S4 achieves 96.3% accuracy at 0.5× the frequency on Speech Commands 10 (Table 5), simply by changing its internal step size ∆ (Section 2.3).

S4 as a General Sequence Model · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9c39503c
A simple application of S4 that treats forecasting as a masked sequence-to-sequence transformation (Fig. 5) outperforms the Informer and other baselines on 40 /50 settings across 5 forecasting tasks.

S4 as a General Sequence Model · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_fa7502e5
Notably, S4 is better on the 10 longest setting in each task, e.g. reducing MSE by 37% when forecasting 30 days of weather data (Table 9).

S4 as a General Sequence Model · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_40a3bded
By contrast, S4 is competitive with a larger ResNet18 (7.9M vs. 11.0M parameters), both with ( 93.16% vs. 95.62%) or without ( 91.12% vs. 89.46%) data augmentation.

S4 as a General Sequence Model · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_94f84245
Moreover, it is much more robust to other architectural choices (e.g. 90.46% vs. 79.52% when swapping BatchNorm for LayerNorm).

S4 as a General Sequence Model · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8c352207
With 0.1 Dropout, the same trends still hold, and the HiPPO initialization—in other words, the full S4 method—achieves 84.27% test accuracy with just 100K parameters.

SSM Ablations: the Importance of HiPPO · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_d79d7b20
S4 achieves SotA accuracy on sequential CIFAR-10 with just 100K parameters.

SSM Ablations: the Importance of HiPPO · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_bd7bda29
We introduce S4, a sequence model that uses a new parameterization for the state space model’s continuoustime, recurrent, and convolutional views to efficiently model LRDs in a principled manner.

Conclusion · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_0f4a8ade
In this work, we show that S4 can address a wide variety of data effectively.

Discussion · page 16 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_7eb6ce9b
We show that this is exactly the same as a Cauchy kernel, which is a well-studied problem with fast and stable numerical algorithms.

Computing the Convolutional View · page 25 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cbf1006f
However, the learning rate on HiPPO parameters (in particular Λ, P, Q, B, C, ∆) were reduced to a maximum starting LR of 0.001, which improves stability since the HiPPO … (excerpt)

Long-Range Dependencies · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_7f7b53b9
We note that most of these results were trained for far longer than what was necessary to achieve SotA results (e.g., the Image task reaches SotA in 1 epoch).

Long-Range Dependencies · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8fac6164
The above hyperparameters describe the results reported in the original paper, shown in Table 10, which have since been improved.

Long-Range Dependencies · page 26 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4d04915c
WaveGAN Discriminator [11] The WaveGAN-D in Table 5 is actually our improved version of the discriminator network from the recent WaveGAN model for speech [11].

Long-Range Dependencies · page 27 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_168da70a
Other settings are reported in [ 2] that include more context at training and evaluation time and improves the score.

WikiText-103 Language Modeling · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_751618f7
WikiText-103 The original version of this paper used an S4 model with batch size 8, context size 1024 which achieved a validation perplexity of 20.88 and test perplexity of 21.28.

Reproduction · page 31 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_e6118e91
It was later retrained with a batch size of 1 and context size 8192 which achieved a validation perplexity of 19.69 and test perplexity of 20.95, and a model checkpoint … (excerpt)

Reproduction · page 31 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2111.00396) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.