03. Deep Ensembles vs MC Dropout

• Primary References:

1. 1
   Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In ICML 2016 (PMLR 48:1050–1059).
2. 2
   Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In NeurIPS 2017.
3. 3
   Fort, S., Hu, H., & Lakshminarayanan, B. (2019). Deep Ensembles: A Loss Landscape Perspective. arXiv:1912.02757 (NeurIPS 2019 Workshop on Bayesian Deep Learning).
4. 4
   Ovadia, Y., et al. (2019). Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. In NeurIPS 2019.

• Why I Wrote This Note: In our paper on Ensemble Deep Learning for Real-Time Textile Fabric Defect Detection (ICCA 2024), combining diverse architectures markedly reduced false-negative defect misses compared to single-backbone inference. When extending these ideas to uncertainty-aware medical models, I needed a clear synthesis of when MC Dropout suffices as a zero-memory-overhead baseline versus when full multi-basin Deep Ensembles (or BatchEnsembles) are mandatory.

• MC Dropout Is Local Variational Inference: Keeping dropout layers active at test time (\(T\) stochastic passes) approximates a factorized Bernoulli variational distribution \(q_\phi(\theta)\) around a single local optimum.
• Deep Ensembles Perform Multi-Modal Functional Exploration: Training \(M\) networks (\(M \in [5, 10]\)) from independent random weight initializations \(\theta_0^{(m)}\) and shuffled mini-batch trajectories converges to distinct, functionally diverse basins of the non-convex loss surface (Fort et al., 2019).
• Functional Diversity Drives Epistemic Quality: Because all MC Dropout masks perturb the same underlying weight matrix \(W_l\), their predictions remain strongly correlated on OOD inputs. Deep Ensembles disagree sharply outside the training support, producing superior epistemic mutual information \(\mathcal{I}[y; \theta \mid x]\).
• Systems Trade-Off: MC Dropout costs \(1\times\) storage/VRAM and \(T\times\) sequential latency; Deep Ensembles cost \(M\times\) training compute and \(M\times\) parameter memory, but all \(M\) members can execute in parallel across devices or via grouped convolutions.

4.1 Mathematical Formulation of MC Dropout

In a neural network with \(L\) layers, applying dropout before weight matrix \(M_l \in \mathbb{R}^{K_l \times K_{l-1}}\) with drop probability \(p_l \in (0, 1)\) defines a stochastic weight matrix \(\widehat{W}_l\):

Gal & Ghahramani (2016) showed that minimizing the standard regularized cross-entropy loss with \(L_2\) weight decay \(\lambda\):

is mathematically equivalent to minimizing the Kullback-Leibler divergence \(\text{KL}\big(q_\phi(\widehat{W}) \,\|\, p(\widehat{W} \mid \mathcal{D})\big)\) in Variational Inference, where \(q_\phi(\widehat{W})\) is a bimodal Bernoulli mixture centered around \(M_l\) and \(0\). At inference time, rather than multiplying weights by \((1 - p_l)\), we draw \(T\) stochastic masks \(\{\widehat{W}^{(t)}\}_{t=1}^{T} \sim q_\phi(\widehat{W})\) and compute the Monte Carlo predictive mean and variance:

4.2 Mathematical Formulation of Deep Ensembles

Instead of imposing a parametric variational family around a single point estimate, Deep Ensembles (Lakshminarayanan et al., 2017) train \(M\) independent parameter vectors \(\{\theta^{(1)}, \dots, \theta^{(M)}\}\) initialized independently via \(\theta_0^{(m)} \sim p(\theta_0)\) and optimized with independent SGD noise. From a Bayesian perspective (Wilson & Izmailov, 2020), each \(\theta^{(m)}\) acts as an approximate MAP sample from a distinct mode of the posterior \(p(\theta \mid \mathcal{D})\), forming an empirical mixture of Dirac deltas:

For continuous regression tasks with heteroscedastic aleatoric noise, each ensemble member outputs both a predictive mean \(\mu_{\theta^{(m)}}(x)\) and variance \(\sigma^2_{\theta^{(m)}}(x) > 0\), trained via Gaussian Negative Log-Likelihood:

By the Law of Total Variance, the ensemble mixture mean \(\mu_\star(x) = \frac{1}{M}\sum_{m=1}^{M} \mu_{\theta^{(m)}}(x)\) has total predictive variance decomposed cleanly into aleatoric and epistemic components:

flowchart TD
    subgraph LossLandscape["Non-Convex Loss Surface L(θ)"]
        B1["Basin 1: θ⁽¹⁾<br/>Ensemble Member 1"]
        B2["Basin 2: θ⁽²⁾<br/>Ensemble Member 2"]
        B3["Basin 3: θ⁽³⁾<br/>Ensemble Member 3"]
        MCD["Local Neighborhood around θ⁽¹⁾<br/>MC Dropout Samples {Ŵ⁽¹⁾, ..., Ŵ⁽ᵀ⁾}"]
        B1 --- MCD
    end
    B1 & B2 & B3 --> DE_OUT["Deep Ensemble: High Functional Diversity on OOD"]
    MCD --> MC_OUT["MC Dropout: High Subspace Correlation on OOD"]

4.3 PyTorch Implementation: Selective MC Dropout vs. Deep Ensemble Decomposition

import torch
import torch.nn as nn

def enable_mc_dropout(model: nn.Module) -> None:
    # Keep BatchNorm in eval mode while activating only Dropout layers during test-time inference
    model.eval()
    for module in model.modules():
        if isinstance(module, (nn.Dropout, nn.Dropout2d, nn.Dropout3d, nn.AlphaDropout)):
            module.train()

@torch.no_grad()
def predict_with_decomposition(models_or_passes, x: torch.Tensor, is_mc_dropout: bool = False, n_passes: int = 10):
    # Compute predictive mean, total entropy, aleatoric entropy, and epistemic mutual information (BALD)
    if is_mc_dropout:
        model = models_or_passes
        enable_mc_dropout(model)
        probs_stack = torch.stack([torch.softmax(model(x), dim=-1) for _ in range(n_passes)], dim=0)
    else:
        probs_stack = torch.stack([torch.softmax(m.eval()(x), dim=-1) for m in models_or_passes], dim=0)

    # probs_stack has shape (M, B, C)
    mean_probs = probs_stack.mean(dim=0)  # (B, C)
    eps = 1e-12

    total_entropy = -(mean_probs * (mean_probs + eps).log()).sum(dim=-1)
    member_entropies = -(probs_stack * (probs_stack + eps).log()).sum(dim=-1)  # (M, B)
    aleatoric_entropy = member_entropies.mean(dim=0)
    epistemic_mi = (total_entropy - aleatoric_entropy).clamp_min(0.0)

    return {
        "mean_probs": mean_probs,
        "total_entropy": total_entropy,
        "aleatoric_entropy": aleatoric_entropy,
        "epistemic_mi": epistemic_mi,
    }

1. 1
   YOLOv8 + CNN Ensemble for Textile Defect Detection (ICCA 2024): In our real-time fabric inspection work, we observed firsthand why multi-model ensembling outperforms single-network perturbation: different convolutional inductive biases attend to complementary spatial frequencies (warp/weft texture vs. localized hole geometry).
2. 2
   TIG-UNet++ Lung Segmentation (ICICT 2025): Training \(5\) full 3D/2D UNet++ models is memory-heavy on clinical workstations, so activating spatial Dropout2d in the deepest decoder bottleneck blocks with \(T=8\) MC passes provides a practical spatial uncertainty map with zero extra disk footprint.
3. 3
   Bridging to Adaptive Inference: Because \(M=5\) Deep Ensembles multiply FLOPs by \(5\times\), my current research explores Early-Exit Ensembles and MoE routing so easy samples only evaluate 1 member or shallow layers, reserving full ensemble evaluation for high-entropy inputs.

• Why Bootstrap Sampling Often Hurts Deep Ensembles: Unlike classical Random Forests, training each deep ensemble member on a bootstrap resample (\(63.2\%\) unique data) reduces accuracy and calibration in deep learning because modern neural networks are data-hungry and SGD + random initialization already provides sufficient trajectory diversity (Lakshminarayanan et al., 2017).
• Last-Layer MC Dropout Is Too Weak: Placing dropout only before the final linear classifier head captures almost no feature-space epistemic uncertainty; if penultimate embeddings collapse on OOD inputs, all \(T\) linear heads receive the same collapsed vector.

• Implement enable_mc_dropout safeguard to prevent BatchNorm running-mean corruption during MC inference.Verified
• Implement BALD decomposition (total_entropy - aleatoric_entropy) for both ensemble lists and MC Dropout models.Verified
• Benchmark BatchEnsemble (Wen et al., ICLR 2020) rank-1 multiplicative adapters (\(W_m = W_0 \circ (r_m s_m^\top)\)) to achieve \(85\%\) of Deep Ensemble diversity at \(1.05\times\) memory cost.Pending
• Evaluate Ensemble Distribution Distillation (\(\text{EnD}^2\)) using a Dirichlet student head to compress \(M=5\) teachers into \(1\) forward pass.Pending

"MC Dropout jiggles weights inside a single valley of the loss landscape; Deep Ensembles send scouts into five completely different valleys."

Law of Total Variance for Heteroscedastic Ensembles:

• Connected Research Notes:
• Understanding Predictive Uncertainty — Defines the mutual information \(\mathcal{I}[y; \theta \mid x]\) estimated by predict_with_decomposition above.
• Early-Exit Neural Networks — How internal classifier heads across depth can act as an implicit ensemble without \(M\times\) backbone replication.
• Mixture-of-Experts Architecture — Sparse conditional routing across specialized sub-networks vs. dense ensemble averaging.
• Risk-Constrained AI Decision-Making — Using epistemic disagreement \(\mathcal{I}[y; \theta \mid x]\) as the abstention score in selective classification.