The Mathematics of Explainable Spike Timing

Explainability in AI is often treated as a post-hoc concern: train the model first, then try to interpret it afterward. Gradient-based saliency maps, attention visualizations, SHAP values—all attempts to peer inside a black box that was never designed to be transparent. This approach produces explanations that are approximate, fragile, and often misleading.

Spiking neural networks offer a different path. Explainability is not an add-on; it is a consequence of the computational substrate. The timing of spikes encodes information, and the propagation of spikes through the network is governed by equations that preserve causal structure. This means decisions are not just interpretable—they are inherently traceable. Every output spike has a provenance that can be reconstructed from the input.

This post lays out the formal framework for understanding why spike-timing-based computation enables explainability by design. We will move through the mathematics carefully, starting from first principles and building toward a general result: spike-based inference produces decision traces that are complete, auditable, and causally faithful.

The structure of a spiking neuron

A spiking neuron maintains a membrane potential $V(t)$, which evolves according to incoming spikes and intrinsic dynamics. The standard model is the Leaky Integrate-and-Fire (LIF) neuron, whose dynamics are:

$$ \tau_m \frac{dV}{dt} = -(V - V_{\text{rest}}) + R \sum_{j} w_j \sum_{t_j^{(k)}} \delta(t - t_j^{(k)}) $$

Here:

• $\tau_m$ is the membrane time constant
• $V_{\text{rest}}$ is the resting potential
• $R$ is the input resistance
• $w_j$ is the synaptic weight from neuron $j$
• $t_j^{(k)}$ are the spike times from presynaptic neuron $j$
• $\delta(t)$ is the Dirac delta function

When $V(t)$ crosses a threshold $V_{\text{th}}$, the neuron emits a spike at time $t_{\text{spike}}$ and resets $V$ to $V_{\text{reset}}$.

This formulation is deterministic. Given a sequence of input spike times ${t_j^{(k)}}$ and initial conditions, the output spike train is uniquely determined. This determinism is the foundation of traceability.

Spike trains as sequences of events

A spike train from neuron $i$ is a point process:

$$ S_i(t) = \sum_{k=1}^{N_i} \delta(t - t_i^{(k)}) $$

where $t_i^{(k)}$ is the time of the $k$-th spike. Unlike continuous-valued activations in traditional neural networks, spike trains are discrete events in time. Each spike carries two pieces of information:

1. Identity: which neuron fired
2. Timing: when it fired

This discrete structure makes the information flow through the network explicit. You do not have to infer relevance from gradient magnitudes or activation values. The spikes themselves are the computation.

Causal propagation and the spike dependency graph

Consider a feedforward spiking network with $L$ layers. An output spike at time $t_{\text{out}}$ in layer $L$ was triggered by a specific subset of spikes in layer $L-1$. Each of those spikes, in turn, was triggered by spikes in layer $L-2$, and so on back to the input.

We can formalize this with a spike dependency graph $G = (V, E)$, where:

• $V$ is the set of all spikes across all layers
• $E$ contains a directed edge $(s_i, s_j)$ if spike $s_i$ contributed to triggering spike $s_j$

A spike $s_i$ contributes to $s_j$ if:

1. $s_i$ occurred before $s_j$
2. The postsynaptic potential (PSP) from $s_i$ had not fully decayed by time $t_j$
3. The cumulative PSP at $t_j$ would not have crossed threshold without $s_i$

This third condition is critical. It distinguishes necessary spikes (those without which the target would not have fired) from present spikes (those that merely occurred earlier).

Formal definition of necessity

Let $V_j(t)$ be the membrane potential of neuron $j$ at time $t$, and let $S_{\text{pre}}$ be the set of all presynaptic spikes up to time $t$. Spike $s_i \in S_{\text{pre}}$ is necessary for spike $s_j$ at time $t_j$ if:

$$ V_j(t_j) \geq V_{\text{th}} \quad \text{and} \quad V_j(t_j) - \Delta V_i < V_{\text{th}} $$

where $\Delta V_i$ is the contribution of spike $s_i$ to $V_j(t_j)$.

This gives us a precise, computable criterion for causal relevance. We can walk backward from any output spike and identify the exact set of input spikes that were causally necessary for it.

Traceability as path analysis

For an output spike $s_{\text{out}}$ in the final layer, the causal trace is the set of all directed paths in $G$ from input spikes to $s_{\text{out}}$. Each path represents a chain of dependencies:

$$ s_{\text{in}}^{(1)} \to s_{\text{h1}}^{(1)} \to s_{\text{h2}}^{(1)} \to \cdots \to s_{\text{out}} $$

The union of all such paths forms the explanation for why the network produced $s_{\text{out}}$.

Importantly, this trace is:

• Complete: It includes all necessary dependencies. No relevant spike is omitted.
• Minimal: It excludes spikes that were present but not necessary.
• Auditable: Each edge in the graph corresponds to a synaptic weight and a temporal contribution that can be verified by recomputing the membrane dynamics.

This is fundamentally different from gradient-based explanations. Gradients tell you how a small change in input would affect output. The spike dependency graph tells you which inputs did affect the output, in the actual forward pass that produced the decision.

Temporal precision and information bottlenecks

Spike timing allows the network to encode information with temporal precision. A neuron that fires at $t = 10.3$ ms conveys different information than one that fires at $t = 10.8$ ms. This precision enables the network to multiplex information in time, not just in population codes.

For explainability, this matters because it allows disambiguation. If two input features both activate a downstream neuron, but one arrives earlier, the temporal ordering reveals which feature dominated. In rate-coded networks, this ordering is lost. In spiking networks, it is preserved in the spike times.

Example: temporal coincidence detection

Consider a neuron that fires only when it receives spikes from neurons A and B within a narrow time window $\Delta t$. The dependency graph will show edges from both $s_A$ and $s_B$ to the output spike, and the edge labels will record the spike times. An auditor can verify:

$$ |t_A - t_B| \leq \Delta t $$

This is a logical rule encoded in the temporal dynamics. The network is implementing a conjunctive condition ("A AND B within $\Delta t$"), and the trace makes this explicit.

From spike traces to symbolic explanations

The spike dependency graph is a low-level trace: a list of neurons, times, and synaptic weights. For human interpretability, we need to map this trace onto higher-level concepts.

In an INN (Intuitive Neural Network), this mapping is built into the architecture. The network is structured so that layers correspond to semantic stages: feature extraction, relation detection, decision integration. Each layer is populated by neurons that represent interpretable units (edges, objects, relationships).

When we construct the causal trace, we can annotate it with these labels:

• Spike $s_{47}$ at $t = 8.2$ ms: "Vertical edge detected at (x=120, y=340)"
• Spike $s_{93}$ at $t = 11.5$ ms: "Corner formed by edges $s_{47}$ and $s_{52}$"
• Spike $s_{201}$ at $t = 15.0$ ms: "Rectangle object identified"

The trace becomes a narrative:

"The network detected a rectangle because:

1. Vertical edge at (120, 340) fired at 8.2 ms
2. Horizontal edge at (340, 120) fired at 9.1 ms
3. Corner detector fired at 11.5 ms when both edges were active
4. Rectangle classifier fired at 15.0 ms when four corners were present"

This is not approximate. It is not a post-hoc rationalization. It is the actual computation, expressed in human-readable terms.

Formal guarantees: soundness and completeness

We can state the explainability theorem precisely.

Theorem (Spike Trace Soundness): Let $N$ be a feedforward spiking network, and let $s_{\text{out}}$ be an output spike at time $t_{\text{out}}$. The causal trace $T(s_{\text{out}})$ computed by backward traversal of the spike dependency graph is sound: every spike in $T(s_{\text{out}})$ was causally necessary for $s_{\text{out}}$.

Proof sketch: The trace is constructed by iterating backward from $s_{\text{out}}$. At each step, we include only spikes $s_i$ such that the membrane potential at the target neuron would not have reached threshold without the PSP contribution from $s_i$. By construction, all spikes in the trace satisfy the necessity condition defined earlier. ∎

Theorem (Spike Trace Completeness): The causal trace $T(s_{\text{out}})$ is complete: every spike that was causally necessary for $s_{\text{out}}$ is included in $T(s_{\text{out}})$.

Proof sketch: Suppose spike $s_i$ was necessary for $s_{\text{out}}$ but not included in $T(s_{\text{out}})$. Then there exists a path from $s_i$ to $s_{\text{out}}$ in the dependency graph that was not traversed. But our backward traversal algorithm visits all incoming edges of each spike in the trace, so this is a contradiction. ∎

These theorems establish that spike traces are faithful: they correspond exactly to the causal structure of the computation.

Practical implementation: trace extraction

To extract traces in a deployed INN, we maintain a sparse event log during inference. For each spike:

1. Record neuron ID, spike time, and membrane potential at threshold crossing
2. Record all presynaptic spikes that contributed to the PSP within the temporal window
3. Compute necessity: for each presynaptic spike, check if removing its PSP would have prevented threshold crossing

This produces a dependency graph in linear time relative to the number of spikes. For sparse spiking networks (typical firing rates of 1–10 Hz per neuron), this overhead is minimal.

When a decision is queried, we traverse the graph backward from the output layer, collecting all necessary spikes and their semantic labels. The result is returned as a structured trace (JSON, protobuf, etc.) that can be inspected by operators or fed into automated auditing tools.

Comparison with gradient-based explanations

Gradient-based methods (Grad-CAM, Integrated Gradients, etc.) approximate feature importance by computing:

$$ \frac{\partial y}{\partial x_i} $$

This tells you how sensitive the output $y$ is to input $x_i$ in a local neighborhood. But:

• Gradients are local; they do not capture the global path from input to output
• Gradients can be noisy or saturated in deep networks
• Gradients assume differentiability, which does not hold for discrete decisions

Spike traces, by contrast, are:

• Global: they capture the full causal chain
• Exact: they reflect the actual computation, not a linearization
• Discrete: they work naturally with threshold-based decisions

The price is that spike traces require explicit event tracking. But this is a small cost when explainability is a hard requirement.

Closing: explainability as a mathematical property

The key insight is this: explainability is not a feature you add to a neural network. It is a property of the computational model. In continuous-activation networks, information flows through real-valued vectors, and causal structure is entangled in weight matrices and nonlinear activations. In spiking networks, information flows through discrete events, and causal structure is encoded in spike times and dependencies.

This discrete structure allows us to trace decisions with the same rigor we apply to symbolic programs. Every spike has a timestamp. Every connection has a weight. Every threshold crossing has a membrane potential. The computation is deterministic, observable, and auditable.

For systems deployed in regulated domains (healthcare, finance, autonomous vehicles), this matters. Regulators do not accept "the model is 95% accurate" as an explanation. They need to know why the model made a specific decision on a specific input. Spike traces provide exactly this: a complete, minimal, and verifiable account of the computation.

Explainability by design. Not as aspiration, but as theorem.