NeurIPS 2026 · SpotlightNeuroscience meets state-space models

Prospective Coding Improves Learningin Deep Continuous-Time Recurrent Networks

Memory takes time. A little look-ahead helps deep recurrent networks keep up.

Shivang Rawat1,*Mirko Morello1Flaviano Morone1,2David J. Heeger1,2

1 Telepath2 New York University* shivang@telepath.com

A layer of RQFs, listening
A bank of RQFs responds to a frequency sweep; each filter is tuned to a different frequency.
A changing-frequency input, a bank of filters. Each unit turns at its own frequency; the ones tuned to the input respond most.
Follow the idea

The contributionA recurrent substrate. A better input path.

2 input taps0 added parametersSame parallel scan
01 / The timing problem

Deep stacks of slow neurons fall behind.

A neuron integrates its input over time. That gives it memory, but also makes it late. Stack these neurons, and the delays add up.

A teaching signal applied at the output refers to activity that early layers produced some time ago. The same temporal integration that makes a recurrent network useful can make a deep one harder to train.

Each layer is a leaky integratorτẏℓ = −yℓ + yℓ−1The layer below supplies the input.

Try it

Put the layers back in step.

With ordinary input, each layer delays and attenuates a 4 Hz signal. Ideal look-ahead cancels the leaky-integrator lag.
111 ms

lag at the final layer

51%

of the input amplitude

Leaky integrators · τ = 20 ms ·
4 Hz input · ideal steady state

The key operator
χτ=1+τddt

Neurons can look ahead.

The prospective operator adds a signal’s local slope to its current value: z + τż. For a leaky integrator it is the exact differential inverse, cancelling its integration lag. As a time advance, it approximates z(t + τ) to first order.

It uses present activity and its slope. No future samples are needed. Prospective responses in cortex, retina, cerebellum, and muscle spindles motivate this construction.

02 / A model from cortical theory

Balance what comes in
with what comes next.

An energy principle gives us a neuron. An oscillatory prediction gives it a rhythm.

Start from an energy

Input meets an internal prediction.

A unit’s activity y balances a bottom-up input z and an internal prediction ŷ. The parameter γ sets their relative influence.

E=γ|y−z|2+(1−γ)|y−y^|2

Descending this energy with respect to the state, holding the drives fixed, gives a leaky integrator that relaxes toward their blend:

τy˙=−y+γz+(1−γ)y^

Following the energy-based cortical theory of Heeger (PNAS, 2017); paper, Appendix A.1.

The geometry of a compromiseDrag γ below
Gamma moves the energy minimum between input and prediction.
Prediction-drivenInput-driven
γ = 0.5555% input · 45% prediction
× marks the energy minimum. Input and prediction are held fixed here to show the energy geometry.
The look-ahead follows the tangent to an oscillation, approximating its exact rotation.
State yFirst-order look-ahead (1 + iωτ)yExact rotation eiωτy
Give the prediction a model

The simplest oscillation is a rotation.

In the complex plane, ẏ = iωy rotates a state at a constant angular frequency. Its exact rotation over one time constant is eiωτy. The prospective operator gives its first-order version:

y^=χτ(y)|y˙=iωy=(1+iωτ)y

The prediction leads the state in phase. The state keeps chasing that prediction, producing a damped oscillation.

Look-ahead appears twice in this work: first in the recurrent prediction, then in the bottom-up input. Both input variants keep the same recurrent prediction.

03 / The recurrent substrate

Meet the Recursive
Quadrature Filter.

Put the oscillatory prediction into the energy-derived dynamics. What remains is a stable, single-pole complex filter.

The RQF, in one line
α = γ / τ decay rateω0 = (1 − γ)ω tuning frequencyRe(λ) < 0 stable for 0 < γ < 1
Smaller γ → narrower tuning, longer memory
Larger ω → faster oscillations
8.3τeffective memory τ / γ
1.58tuning frequency ω0τ
a

A state that spirals

A stable RQF spirals toward the origin when the input is removed.
Free response in the complex plane.
The real and imaginary states rotate as they decay.
b

A pair of temporal filters

The impulse response has decaying real and imaginary sinusoidal components.
RealImaginary
Impulse response divided by α; time in units of τ.
c

A preferred frequency

The complex transfer function has a peak at the filter's tuning frequency.
Complex transfer magnitude |H(iΩ)|.
Peak at ω0; half-power bandwidth 2α.

Two learnable dynamical parameters per unit: γ and ω. For real signals, the two components approximate a quadrature pair near resonance in the narrow-band regime.

The bridge to machine learning

Derived from cortex.
Trained like an SSM.

Collect these filters into a layer and mix its inputs with W. You get a diagonal state-space model, with each channel’s input gain tied to its own decay rate.

y˙=diag(λ)y+diag(α)Wx

Every RQF layer is a diagonal SSM. The inclusion is one-way: a general diagonal SSM need not have this pole–input-gain coupling.

Parallel scansFFT convolutionsLong sequences
RQFs are a constrained subset of diagonal state-space models Diagonal state-space models RQFs α = γ / τ S4D S5 LRU DSS
The SSM toolkit carries over to RQFs.
04 / The input-side correction

Two taps.
Zero new parameters.

Now use look-ahead a second time: drive each layer with z + τż. The recurrent dynamics stay the same. The input gets one extra path.

yn+1=A¯yn+[B¯+γA¯]zn+1−γA¯zn
A recurrent state path, a current input path, and a prospective path from the previous input yn zn+1 yn+1 A B + γA + zn −γA one extra path: the previous input same state transition

Exact impulse-ZOH update for an RQF channel.
A = eλh, B = (α/λ)(A − 1). Initialize the previous input to zero.

One recurrent step
# coefficients from the same RQF parameters
B_plus  = B_bar + gamma * A_bar
B_minus = -gamma * A_bar

# two input taps, the same recurrent transition
y = A_bar * y + B_plus * z + B_minus * z_prev
z_prev = z
0 additional parametersSame diagonal transitionSame associative scanOne lagged input buffer

The added work is elementwise and linear in sequence length. For a general diagonal SSM, the extra coefficients are ±τAB; the same construction applies beyond RQFs.

How does the continuous look-ahead become two discrete taps?

Hold each input constant over an integration interval of length h. At the boundary, the input changes from zn to zn+1, so its derivative is an impulse with that difference as its weight. Integrating the linear dynamics exactly propagates this impulse with A. Since τα = γ, the extra contribution is γA(zn+1 − zn).

This is the paper’s impulse-ZOH convention, rather than a finite-difference approximation to a smooth derivative. See Section 3 and Appendix B for the derivation and the general diagonal-SSM form.

05 / Why it helps learning

Help the error survive
the trip down.

The smaller the time step, the less error an instantaneous input passes between layers.

To isolate this effect, detach earlier recurrent states and let errors travel only across layers within a step. Under this spatial-only backpropagation diagnostic, the input’s per-hop Euler coefficient changes from:

Instantaneous(h / τ)γO(h / τ)
Prospective(1 + h / τ)γO(1) as h / τ → 0

The exact ZOH update has the same leading-order scaling. Look-ahead removes this explicit step-size penalty at every spatial hop.

What the result says

It removes a discretization-induced attenuation factor. Activation slopes, weights, and damping can still shrink the full gradient. Spatial-only training is a diagnostic; the main experiments also use full BPTT.

Error reaching each layerMeasured · log scale
ProspectiveInstantaneous
Prospective gradients are 10,290 times larger at layer one in this six-layer spatial-only diagnostic.
Six-layer, width-64 raw-audio RQF; h / τ = 0.2.
Weight-gradient Frobenius norms, paper Table 9.
10,290×

larger early-layer weight gradient
at layer 1 in this spatial-only diagnostic

Read the gradient analysis
What happens as the time step gets smaller?

The measured prospective per-hop error gain stays near 0.6 as the step size shrinks. The instantaneous gain falls roughly in proportion to h / τ, so their ratio grows. The ratio predicted from the exact ZOH coefficients follows this trend.

Spatial-only diagnostic, six-layer width-64 raw-audio RQF; paper, Table 10.

h / τ Prospective gain Instantaneous gain Measured ratio Predicted ratio
1.00 0.619 0.361 1.72× 1.96×
0.50 0.594 0.220 2.70× 2.95×
0.20 0.585 0.104 5.65× 5.93×
0.10 0.598 0.0565 10.59× 10.91×
0.05 0.624 0.0305 20.43× 20.89×

Per-hop gain is fitted over layers 6 to 2 from the RMS error norms. These are diagnostic measurements, not accuracy results.

06 / What the models learn

Small models.
Long sequences.

From raw audio to nonlocal visual structure and content-aware memory. All numbers below come from the paper.

96.09%

Raw-audio Speech Commands

31.9k parameters. Six layers, width 32. No residual connections, gating, or normalization.

± 0.23 · 16,000 steps · full BPTT
83.36%

Path-X · 16,384 steps

A minimal six-layer RQF reads a flattened 128 × 128 image. Input-selective sRQF reaches 86.31%.

± 1.71 · width 64 · full BPTT
99.83%

Selective Copying

A four-layer selective RQF recalls 16 tokens hidden among 4,096 steps. Prospective input succeeds in 5/5 seeds; its control in 2/5.

± 0.07 · 33.0k parameters · full BPTT
Full backpropagation through time

Prospective variants match or improve mean test accuracy in all 35 comparisons on Speech Commands and Path-X.

RQF · αP-S5 · ORGaNICs · sRQF
A closer look

Explore the comparisons.

Mean ± sample s.d. · five seeds¹

RQF · width 32 · Raw audio · 16,000 steps · Full BPTT

Layers Parameters Prospective Instantaneous Difference
2 23.5k 94.59 ± 0.28% 93.96 ± 0.28% +0.63 pts
4 27.7k 95.66 ± 0.23% 95.29 ± 0.22% +0.37 pts
6 31.9k 96.09 ± 0.23% 95.27 ± 0.22% +0.82 pts

Non-residual RQF stacks without normalization or gating. Source: paper, Table 6.

¹ Selective Copying accuracy is computed over seeds that exceed chance; success counts are reported separately. The αP-S5 comparison includes input-gain scaling, the second tap, and pole clipping. These are mean-accuracy comparisons, not claims of statistical significance.

When the input decides what to remember

From RQF to selective RQF.

An sRQF makes each channel’s input gain depend on the current input. For Path-X, that is the only selective component. For Selective Copying, the bandwidth γ and frequency ω also become input-dependent. The recurrence stays diagonal and retains its parallel scan; fixed-kernel FFT convolution applies to the time-invariant RQF.

07 / Keep exploring

The paper, the poster,
and the details.

Read the abstract

Temporal integration gives continuous-time recurrent networks memory, but in deep stacks it also delays bottom-up signals and attenuates top-down errors. We compensate for this delay by making each layer’s bottom-up input prospective, and we develop Recursive Quadrature Filters (RQFs), biologically motivated temporal band-pass filters that are a special case of diagonal state-space models (SSMs). Starting from an energy model, we derive the RQF dynamics and show that each RQF is a band-pass filter whose learnable parameters control its tuning frequency and bandwidth. We then make each layer’s bottom-up input prospective using a parameter-free two-tap update that leaves the recurrent transition and parallel scan unchanged. We extend this correction to general diagonal SSMs and show that it mitigates depth-dependent gradient attenuation when temporal gradients are truncated, i.e., spatial-only backpropagation.

We evaluate the performance of RQFs, S5, and ORGaNICs (a nonlinear gated RNN with built-in attentional modulation and recurrent normalization) trained using full backpropagation through time (BPTT) and spatial-only backpropagation. Under full BPTT, the prospective variants of these models match or outperform their non-prospective controls in every configuration. A non-residual width-32 six-layer RQF achieves 96.09 ± 0.23% accuracy on raw-audio Speech Commands with 31.9k parameters; a width-64 six-layer RQF achieves 83.36 ± 1.71% on the 16,384-step Path-X task. We also introduce selective RQFs (sRQFs), which make each filter’s input gain and, optionally, its bandwidth and tuning frequency input-dependent while retaining the parallel scan. With the input gain alone, the same six-layer model achieves 86.31 ± 0.51% on Path-X; with all three input-dependent, a four-layer prospective sRQF solves Selective Copying (99.83 ± 0.07%). These results identify RQFs as a parameter-efficient recurrent substrate and prospective-input coding as an input-side correction for deep continuous-time recurrent networks.

Build on this work

Citation

@misc{rawat2026prospective,
  title = {Prospective Coding Improves Learning in Deep
           Continuous-Time Recurrent Networks},
  author = {Rawat, Shivang and Morello, Mirko and
            Morone, Flaviano and Heeger, David J.},
  year = {2026},
  eprint = {2609.04134},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  note = {NeurIPS 2026},
  url = {https://arxiv.org/abs/2609.04134}
}