clm_f7a2e79eWe propose the Structured State Space sequence model (S4) based on a new parameterization for the SSM, and show that it can be computed much more efficiently than prior approaches … (excerpt)
arXiv 2111.00396 · cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_f7a2e79eWe propose the Structured State Space sequence model (S4) based on a new parameterization for the SSM, and show that it can be computed much more efficiently than prior approaches … (excerpt)
clm_eb5cc028Although theoretically efficient algorithms for the LSSL were proposed, we show that these are numerically unstable.
res_ef72c2f7Combining these two ideas, we show that the low-rank term can be corrected by the Woodbury identity while the normal term can be diagonalized stably, ultimately reducing to a well-studied … (excerpt)
res_6b74b49bOn the LRA benchmark for efficient sequence models, S4 is as fast as all baselines while outperforming them by 20+ points on average.
res_ee8c74c7With no architectural changes, S4 surpasses Speech CNNs on speech classification, outperforms the specialized Informer model on time-series forecasting problems, and matches a 2-D ResNet on sequential CIFAR with over … (excerpt)
res_ab5fe7cfFor example, the LSSL found that simply modifying an SSM from a random matrix A to equation (2) improved its performance on the sequential MNIST benchmark from 60% to 98%.
clm_7bd4d6caIn Appendix B, we prove that it is also numerically unstable for related reasons.
res_342b850fS4 substantially advances the SoTA, outperforming all baselines on all tasks and averaging 80 .48% compared to less than 60% for every baseline.
res_c2839ff4S4 achieves 98.3% accuracy, higher than all baselines that use the 100× shorter MFCC features, and validates that a powerful LRD model is able to extract more information from the … (excerpt)
clm_172c9e2eWe show that alternative models to Transformers can still be competitive in these settings.
clm_6ceed908On both CIFAR-10 and WikiText-103, we report the throughput of various models at generation time, with S4 around 60 × faster than a vanilla Transformer on both tasks (details in … (excerpt)
res_40939656Without re-training, S4 achieves 96.3% accuracy at 0.5× the frequency on Speech Commands 10 (Table 5), simply by changing its internal step size ∆ (Section 2.3).
res_9c39503cA simple application of S4 that treats forecasting as a masked sequence-to-sequence transformation (Fig. 5) outperforms the Informer and other baselines on 40 /50 settings across 5 forecasting tasks.
res_fa7502e5Notably, S4 is better on the 10 longest setting in each task, e.g. reducing MSE by 37% when forecasting 30 days of weather data (Table 9).
res_40a3bdedBy contrast, S4 is competitive with a larger ResNet18 (7.9M vs. 11.0M parameters), both with ( 93.16% vs. 95.62%) or without ( 91.12% vs. 89.46%) data augmentation.
res_94f84245Moreover, it is much more robust to other architectural choices (e.g. 90.46% vs. 79.52% when swapping BatchNorm for LayerNorm).
res_8c352207With 0.1 Dropout, the same trends still hold, and the HiPPO initialization—in other words, the full S4 method—achieves 84.27% test accuracy with just 100K parameters.
res_d79d7b20S4 achieves SotA accuracy on sequential CIFAR-10 with just 100K parameters.
clm_bd7bda29We introduce S4, a sequence model that uses a new parameterization for the state space model’s continuoustime, recurrent, and convolutional views to efficiently model LRDs in a principled manner.
clm_0f4a8adeIn this work, we show that S4 can address a wide variety of data effectively.
clm_7eb6ce9bWe show that this is exactly the same as a Cauchy kernel, which is a well-studied problem with fast and stable numerical algorithms.
res_cbf1006fHowever, the learning rate on HiPPO parameters (in particular Λ, P, Q, B, C, ∆) were reduced to a maximum starting LR of 0.001, which improves stability since the HiPPO … (excerpt)
res_7f7b53b9We note that most of these results were trained for far longer than what was necessary to achieve SotA results (e.g., the Image task reaches SotA in 1 epoch).
res_8fac6164The above hyperparameters describe the results reported in the original paper, shown in Table 10, which have since been improved.
res_4d04915cWaveGAN Discriminator [11] The WaveGAN-D in Table 5 is actually our improved version of the discriminator network from the recent WaveGAN model for speech [11].
res_168da70aOther settings are reported in [ 2] that include more context at training and evaluation time and improves the score.
res_751618f7WikiText-103 The original version of this paper used an S4 model with batch size 8, context size 1024 which achieved a validation perplexity of 20.88 and test perplexity of 21.28.
res_e6118e91It was later retrained with a batch size of 1 and context size 8192 which achieved a validation perplexity of 19.69 and test perplexity of 20.95, and a model checkpoint … (excerpt)
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2111.00396) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.