August 14, 2026 · 22 min read

BatchNorm Folding SNN Guide

BatchNorm must merge into the preceding conv or linear before you replace activations with integrate-and-fire neurons. This page is the algebra, the NeuroCUDA API, and the collapse you get if you skip it. It is not a second ANN-to-SNN theory hub.

batchnorm folding SNN conversion in NeuroCUDA merges BatchNorm into the preceding conv or linear before IF replace. Install with pip install neurocuda from https://github.com/Krishnav1/neurocuda. Use neurocuda.fold_batchnorm, or let convert() fold. Unfolded BatchNorm after spikes shifts activations and collapses accuracy. This is not the ANN-to-SNN theory hub.

This page vs nearby pages: this URL is why BatchNorm must fold before IF replace. /blog/snn-accuracy-drop-after-conversion is the four-bug checklist (reset, T, lambda, BN). /blog/qcfs-ann-to-snn is calibration method. /ann-to-snn is conversion theory. convert PyTorch to SNN is the eight-step HowTo. Do not index this page as a clone of the theory hub.

Almost every modern CNN you would convert has BatchNorm. ResNet-18 on CIFAR is built from conv-BN-ReLU blocks. If you replace ReLU with an integrate-and-fire neuron and leave BatchNorm in the graph, you have not converted the network the ANN computed. You have inserted a normalizer that was fit to a continuous tensor into a stream of spikes. That mismatch is one of the four ways a 95% ANN becomes a 50% SNN, documented as bug four on SNN accuracy drop after conversion. This page exists so that bug has a dedicated URL: algebra, API, checks, and what not to claim.

NeuroCUDA folds as part of conversion. Install with pip install neurocuda. Source is MIT at github.com/Krishnav1/neurocuda. Author: Krishna Santosh Varma, QuantaraCore Technologies LLP. You still need to know what folding is, because convert() will not stop you from reintroducing nn.BatchNorm2d later, and it will not explain a shifted activation histogram. The local env HowTo is pip install neurocuda guide. Stay here for the fold.

Why batchnorm folding SNN must happen before IF replace

Write the ANN block the way PyTorch actually runs it:

y = Conv(x; W, b)
y = BatchNorm(y; gamma, beta, mean, var, eps)
y = ReLU(y)

BatchNorm at eval is an affine transform:

y_hat = gamma * (y - mean) / sqrt(var + eps) + beta

mean and var are running statistics collected on real-valued conv outputs during ANN training. gamma and beta are learned scale and shift. Together they keep each channel on a scale ReLU likes. QCFS then learns a per-channel threshold on that ReLU scale. IF replace turns the calibrated activation into a membrane and a fire. If BatchNorm is still sitting between conv and IF, two things are wrong at once:

1. The neuron is integrating a tensor that is still being recentered by statistics from a different domain (continuous activations, not spike counts).
2. The QCFS threshold was (or will be) fit as if the ReLU saw the folded affine. A live BN after spikes is a second, drifting affine the threshold does not know.

Folding removes BN by absorbing the affine into W and b. After a correct fold, the block is:

y = Conv(x; W_fold, b_fold)
y = ReLU(y)          # then IF replace this ReLU
# no BatchNorm

That is the only graph IF replace is allowed to see. batchnorm folding SNN conversion is this merge, not a new learning algorithm and not a restatement of ANN-to-SNN theory. The theory hub remains /ann-to-snn. QCFS remains QCFS ANN to SNN. T as a budget remains SNN timesteps T explained.

Order is not optional. Fold on the ANN. Calibrate QCFS on the folded (or jointly converted) graph. Replace ReLU with IF. Finetune with BPTT. If you replace first, there is no honest fold left: the preceding tensor is spikes, and the BN formula is a lie about what it is normalizing. Restore the ANN checkpoint and start again.

The fold algebra (conv and linear)

Let s = gamma / sqrt(var + eps). For a convolution, scale the output channels of W and rewrite bias:

W_fold[c, ...] = s[c] * W[c, ...]
b_fold[c]      = s[c] * (b[c] - mean[c]) + beta[c]

If the conv had no bias, treat b as zero and create one. For nn.Linear the same identities hold on the output features. For BatchNorm2d after a conv, c is the channel. For BatchNorm1d after a linear, c is the feature. Depthwise convs still fold per output channel. Grouped convs fold per output channel as long as BN is per-channel, which it is in the usual ResNet pattern.

Numerically, a folded ANN (still ReLU, no spikes) should match the original ANN to floating-point noise on eval. That check is independent of SNN conversion. Run it. If the folded ANN already disagrees, you folded the wrong pairing (BN not adjacent to that conv), you were still in train mode (batch stats instead of running stats), or you hit a graph the folder does not support (BN in a residual add with no preceding conv on that branch, BN after a concat). Fix the ANN graph before you blame IF neurons.

ResNet shortcuts sometimes have a 1×1 conv plus BN on the projection branch, and identity on the other. Fold each BN into its own conv. Do not fold across the add. The add is a sum of two already-normalized streams in the ANN; after fold it is a sum of two scaled convs, which is the same function. NeuroCUDA's residual executor is about keeping that add honest in NIR, which is the ResNet tutorial's job, not a second copy of residual algebra here.

What folding does not do: it does not train. It does not set QCFS lambda. It does not pick T. It does not measure energy. It is a compiler pass that should be close to lossless on the ANN, then mandatory before spikes.

API: neurocuda.fold_batchnorm and convert()

Two entry points. Use both when you debug. Use convert() when you just want the SNN.

import torch
import neurocuda

model.eval()  # running stats, not batch stats

# Explicit fold: inspect weights, unit-test the ANN match
folded = neurocuda.fold_batchnorm(model)
assert not any("bn" in n.lower() or isinstance(m, torch.nn.modules.batchnorm._BatchNorm)
               for n, m in folded.named_modules())

# Default conversion path already folds inside convert()
snn, meta = neurocuda.convert(model, calib_loader)
neurocuda.compile(snn, target="gpu")
ann_acc = neurocuda.evaluate(model, test_loader)
snn_acc = neurocuda.evaluate(snn, test_loader)
print(ann_acc, snn_acc)

Inside convert() the stages stay the ones NeuroCUDA documents: QCFS calibration on in-distribution batches, batchnorm folding SNN merge into conv/linear, IF replace, BPTT finetune with an atan surrogate. meta records conversion. Log it. Do not invent a schema for it in a blog. Compile selects a backend (gpu, cpu, loihi2_sim). Extra finetune after you change T is a different experiment; T lives on the timesteps page.

Call fold_batchnorm on a copy if you need the original module intact. Conversion should not silently destroy the only ANN checkpoint you have. Keep that checkpoint as the regression baseline, the same advice as convert PyTorch to SNN.

If you are writing a test in CI, the assertion above is the point: after fold, BatchNorm modules are gone. After convert, they should still be gone. A Docker job that only checks accuracy can miss a graph that still contains BN and happens to score okay on a tiny subset. Container setup is neurocuda docker. Citation labels are cite NeuroCUDA.

Failure if unfolded: shifted activations, accuracy collapse

Unfolded means the SNN still has BatchNorm modules, or you reinserted them, or you converted a submodule and then wrapped it with the original BN. Symptoms, in the order you will actually see them:

Shifted activations

Log per-channel means of the tensor that enters the IF neuron. On a healthy folded net they look like scaled conv outputs. On an unfolded net they look like (spike_count - running_mean) / sqrt(running_var), where running_mean was fit to ReLU-era floats. Spike counts are small integers, often 0/1 per step. Subtracting a ReLU-era mean of, say, 2.4 from a 0/1 stream drives the channel negative. Gamma then amplifies the nonsense. The membrane integrates a centered-and-scaled object the ANN never computed.

Threshold mismatch

QCFS lambda is a scale for the activation QCFS replaced. If BN still moves the tensor after that scale was chosen, lambda is in the wrong units. Thresholds that "learned" during convert can look moved in meta and still be wrong at test because the live BN is an extra operator. People then open the frozen-lambda page by mistake. Frozen lambda is a different bug (shared LR, dead floor gradient) on QCFS threshold not learning. Moved lambda plus live BN is this page.

Accuracy collapse

Unlike the membrane-reset bug, which often lands on chance (about 10% on CIFAR-10), unfolded BN can land anywhere: 40%, 60%, a weirdly confident wrong class. That is why it is easy to misread as "conversion gap." The accuracy-drop article's rule still holds: a gap larger than about 10-15 points is a bug until proven otherwise. Published healthy gaps are small: N-MNIST SNN 99.88% ± 0.02% vs ANN 99.70%; ResNet-18 CIFAR-10 SNN 94.61% ± 0.14% vs ANN 95.56% at T=32. Those are software-backend results from the report. Skipping fold is not how you get a new paper number. It is how you miss them.

Reset still comes first in a debugging order because chance-level is unambiguous. T comes second because a sweep is cheap once the graph is folded. Lambda third. Folding fourth in that checklist only because it is slightly slower to reason about, not because it is rare. On ResNets it is the default foot-gun. Confirm fold before you write "SNNs do not like residual nets."

How to verify folding in a few minutes

import torch
import neurocuda

def bn_keys(module):
    keys = []
    for name, _ in module.named_parameters():
        if "running_mean" in name or "running_var" in name:
            keys.append(name)
    for name, buf in module.named_buffers():
        if "running_mean" in name or "running_var" in name:
            keys.append(name)
    mods = [n for n, m in module.named_modules()
            if isinstance(m, torch.nn.modules.batchnorm._BatchNorm)]
    return keys, mods

model.eval()
folded = neurocuda.fold_batchnorm(model)
keys, mods = bn_keys(folded)
print("BN buffers:", keys)
print("BN modules:", mods)

# ANN identity check (still ReLU, no spikes)
x, _ = next(iter(calib_loader))
with torch.no_grad():
    a = model(x)
    b = folded(x)
print("max abs diff", (a - b).abs().max().item())

Expect empty BN lists and a tiny diff. Then convert. After snn, meta = neurocuda.convert(model, calib_loader), run bn_keys(snn) again. If BN returned, folding did not stick. Do not compile and evaluate as if the SNN were valid.

Train mode is a silent breaker. model.train() makes BatchNorm use batch statistics. Folding must use eval running stats, the same stats deployment uses. Convert on a model left in train() can fold the wrong numbers or skip the intended path. Call eval() first. Calibration batches are for QCFS, not for refreshing BN means at convert time, unless you have a documented protocol that says otherwise. The published protocol is in paper.pdf and reproduce NeuroCUDA results.

Where folding sits next to QCFS, T, and backends

QCFS needs a stable activation scale. Folding is how conv+BN becomes one linear map so that scale is the map QCFS sees. If you want why clip-floor-shift exists, that is the QCFS page. If you want why T=8 vs 16 vs 32, that is SNN timesteps T. Folding does not pick T. It makes a T sweep meaningful. Sweeping T on an unfolded ResNet is how you spend four converts to learn nothing.

Backends do not relax the fold. GPU and CPU are cross-checked software backends. Loihi 2 in NeuroCUDA is an IF-neuron simulator, not Intel silicon. SpiNNaker-1 jobs #420148 and #420186 are a 2-neuron smoke test, not ResNet-on-chip. An unfolded BN graph exported to NIR is still wrong on every backend. NIR export is export PyTorch to NIR when it exists in your reading list; residual identity is not a substitute for folding. FPGA HLS C++ is not a bitstream (NeuroCUDA FPGA HLS). Akida is honest status, not a physical claim (NeuroCUDA Akida). CUDA field map: /neuromorphic-cuda.

Published accuracies stay labeled software-backend:

N-MNIST HowTo: N-MNIST SNN conversion. ResNet HowTo: ResNet-18 SNN conversion tutorial. Neither HowTo replaces this fold page. ResNet is the architecture that makes folding non-optional. N-MNIST may use a small CNN with or without BN; if BN is present, fold it. If BN is absent, fold_batchnorm should be a no-op, not a crash. If it crashes on a BN-free MLP, that is a compiler bug to file on GitHub, not a reason to skip fold on ResNet.

What this page will not clone

It will not retell ANN-to-SNN conversion theory. That hub is /ann-to-snn. It will not retell QCFS clip-floor-shift. It will not retell the four-bug checklist beyond the BN row. It will not become the convert HowTo. It will not invent ImageNet accuracy. It will not say folding ran on Loihi silicon. It will not say SpiNNaker measured ResNet. GPU time is not Loihi energy; that sentence belongs in the T article and in citations, and it is true here too when someone tries to "save energy" by skipping fold (you do not). Skipping fold saves nothing. It spends accuracy.

Runnable checks you can paste

Full path from a trained ReLU CNN. CIFAR normalize belongs on CIFAR. Keep eval() on.

import copy
import torch
import neurocuda

def convert_with_fold_check(model, calib_loader, test_loader):
    model = copy.deepcopy(model).eval()
    folded = neurocuda.fold_batchnorm(model)
    leftover = [
        n for n, m in folded.named_modules()
        if isinstance(m, torch.nn.modules.batchnorm._BatchNorm)
    ]
    if leftover:
        raise RuntimeError(f"unfolded BatchNorm remain: {leftover}")

    snn, meta = neurocuda.convert(model, calib_loader)
    leftover_snn = [
        n for n, m in snn.named_modules()
        if isinstance(m, torch.nn.modules.batchnorm._BatchNorm)
    ]
    if leftover_snn:
        raise RuntimeError(f"SNN still has BN: {leftover_snn}")

    neurocuda.compile(snn, target="gpu")
    return {
        "ann": neurocuda.evaluate(model, test_loader),
        "snn": neurocuda.evaluate(snn, test_loader),
        "meta": meta,
    }

If this raises, do not tune T. If this returns a 40-point gap with empty BN lists, go to reset and lambda. If this returns about a 1-point gap on ResNet-18/CIFAR at T=32, you are in the published neighborhood; match seeds on the reproduce page before you claim reproduction.

Colab users: folding is cheap; convert is not. Save the ANN and the SNN to Drive. Folding will not recover a session that died mid-BPTT. See NeuroCUDA Google Colab.

Cluster map

• /neurocuda - product hub
• GitHub - source, MIT
• paper.pdf - numbers
• convert PyTorch to SNN - eight-step API
• ResNet-18 tutorial - BN-heavy CIFAR net
• N-MNIST conversion - event path
• QCFS - calibration after a stable scale
• accuracy drop - four bugs
• reproduce - seeds
• SNN timesteps T - choose 8/16/32 after the graph is folded
• batchnorm folding SNN - this page (fold before IF replace)
• cite NeuroCUDA - BibTeX, honest backends
• NeuroCUDA Docker - CI
• Akida - honest status
• FPGA HLS - C++ PoC
• /ann-to-snn - theory hub this page is not
• pip install guide
• /neuromorphic-cuda - CUDA map

Frequently asked questions

Next: pip install neurocuda · neurocuda.fold_batchnorm(model) · snn, meta = neurocuda.convert(model, calib_loader) · Product page · Still dropping accuracy?