Verkko Research Programme · Technical brief
Beyond
Transformers.
Sustaining edge intelligence with spiking neural networks.
88%
Network sparsity
<5%
Dead neurons
16
Internal timesteps
7B
Spiking parameters
Always-on intelligence at the edge is not a scaling problem. It is an activation density problem. This brief sets out the architecture VOLTAIC uses to make computation an event rather than a clock tick, the runtime invariants that hold sparsity in place, and the sparse-attention work that comes next.
The ceiling
Dense inference collides with the edge power budget.
In a dense transformer, every parameter participates in every token. Attention and the feed-forward blocks both perform the full matrix multiply whether or not the input carries new information, which means cost is set by model size and sequence length alone.
A robot, a sensor node or an autonomous platform cannot sustain transformer-class wattage continuously. Making the model smaller buys a little headroom and then hits the same wall, because the constraint is architectural rather than a matter of parameter count.
Continuous edge intelligence needs an event-driven compute model, not a denser always-on one.
The primitive
Spikes replace clocks.
A spiking neuron contributes compute only when its membrane potential crosses threshold. No fire, no multiply-accumulate. Energy then tracks how much the input is actually changing, which inverts the cost model that makes dense inference unaffordable in the field.
Dense tensor
[B, T, D]
A full activation for every batch item, every position and every channel, recomputed at every step regardless of what the input did.
SpikeTensor
sparse events · zeros dominate
Discrete spike events scattered across internal timesteps. The overwhelming majority of entries are zero, and a zero costs nothing to compute.
The contract
Four modalities, one spike contract.
Text, vision, audio and structured signals are each encoded into the same event representation. Because everything downstream sees spikes and nothing else, a single stack fuses modalities without dense embedding towers standing between the sensor and the model at inference time.
Text
Token embeddings become rate or latency spike codes — meaning is carried by how often and how early a neuron fires.
Image
A Difference-of-Gaussians front end emits polarity spikes, so only contrast that changes produces events at all.
Audio
Poisson rate coding over spectro-temporal features turns a continuous signal into a sparse event stream.
Structured
Gaussian population codes spread numeric and categorical fields across a bank of tuned neurons.
Memory in the neuron
ALIF neurons store time as state.
A transformer recovers temporal context by rereading the sequence, which is why its cost grows with everything it has already seen. An adaptive leaky integrate-and-fire neuron instead carries that context in its own membrane potential and threshold.
Compact stateful dynamics replace sequence rereads.
Membrane decay
An exponential leak integrates spike history, so recent events weigh more than distant ones without any explicit window.
Adaptive threshold
The firing threshold rises after each spike and relaxes over time, stabilising the neuron’s rate against runaway activity.
Surrogate gradients
A smooth proxy stands in for the Heaviside spike during backpropagation, making the discontinuous network trainable.
The enforcement
Sparsity is a constraint, not an outcome.
The Spiking Feed-Forward Network expands and projects through spike normalisation and ALIF neurons, unrolled over sixteen internal timesteps. An explicit sparsity loss and a dead-neuron regulariser sit in the objective itself, so the network is held between two bounds throughout training rather than pruned towards them afterwards.
SFFN path
Unrolled over 16 internal timesteps.
88%
Sparsity floor
The share of the network that stays silent is a property the architecture guarantees, measured at runtime rather than estimated from a training run.
<5%
Dead-neuron ceiling
Silence is only useful if the quiet neurons can still fire. Under five percent of them go permanently dark, which is what separates enforced sparsity from slow capacity loss.
The barrier
Spiking networks stalled at 200M. VOLTAIC runs at 7B.
Spiking networks have historically become unstable somewhere beyond a couple of hundred million parameters. Firing rates drift, neurons fall silent for good, and the training signal through the surrogate gradient stops being informative. That ceiling is the reason most neuromorphic work has stayed at the scale of keyword spotting and gesture recognition.
VOLTAIC is a fully spiking 7B-parameter model — roughly thirty-five times past that barrier, reached by making the sparsity and liveness bounds part of the architecture instead of hoping they emerge.
Prior art
~200M
Where pure spiking networks have historically lost stability.
Now
7B · 88%
Spiking parameters at eighty-eight percent sparsity, with the dead-neuron ceiling held under five percent.
The destination
70B · 95%
A homeostatic loss penalises firing above the target rate, so a bigger model is held to a stricter bound, not a looser one.
The economics
Sparsity tightens as we scale. Energy does not.
Dense scaling compounds energy super-linearly, because every added parameter fires on every token. That is the arithmetic that keeps large models in data centres.
When the energy budget is set by spike activity rather than parameter count, the relationship breaks. A larger VOLTAIC model is trained against a stricter firing target, so growth in capability does not have to be paid for in watts.
If sparsity scales with the model, capability and power consumption stop being the same curve.
What comes next
Sparse spiking self-attention.
The encoders and the spiking feed-forward core are built. The remaining step is to carry the same enforced sparsity through attention itself, which is where dense computation still concentrates.
Step 1
Multimodal spike encoders
BuiltStep 2
ALIF-based spiking feed-forward
BuiltStep 3
Spiking multi-head self-attention
In researchLIF queries and keys
Q and K are built from leaky integrate-and-fire neurons, so a match is a coincidence in spike timing rather than a dense dot product.
ALIF value streams
V is carried by adaptive neurons that hold state, letting a value persist across timesteps instead of being recomputed.
STDP temporal weighting
Softmax with a segment bias produces the scores, then spike-timing-dependent weighting replaces uniform aggregation before the output projection — recent coincident spikes count for more than flat attention mass.
Open problems
The frontier is still open.
Four problems stand between event-driven inference and general deployment. None of them is solved, and we would rather say so.
- Sparsifying attention at scale without collapsing quality
- On-device and online training through ALIF surrogate gradients
- Cross-modal spike fusion at runtime under a single contract
- Continuous learning in human–swarm context (HSC)
Work on this with us
Verkko is building the sparse stack where continuous intelligence has to live — at the edge, under a power ceiling, on spike time.
Architecture deep dive
The SFFN and ALIF design in detail, including the sixteen-step internal dynamics.
Runtime metrics
Measured sparsity and dead-neuron figures from live inference, not training-time estimates.
Stress-test the invariants
Bring your own traces and see whether the sparsity bounds hold on your workload.