04. Shift-Calibrated Evidential Early Exiting for Selective Prediction in Resource-Constrained Vision Transformers Under Distribution Shift

Executive Summary Working Title: Dirichlet Evidential Early-Exit Vision Transformers with Selective Abstention

Early-Exit Vision Transformers (Early-Exit Neural Networks) reduce inference latency by attaching exit heads at intermediate self-attention blocks $m \in \{1, \dots, M\}$. Yet a fundamental theoretical question remains: when an intermediate exit head is uncertain, should the network spend more FLOPs running deeper layers, or is the input an out-of-distribution anomaly where deeper layers will only waste compute and hallucinate? Standard first-order softmax entropy $\mathcal{H}(\hat{p}_m)$ cannot answer this question because it conflates conflicting class evidence (which deeper self-attention blocks can resolve) with complete lack of evidence (OOD ignorance). By parameterizing intermediate exits with Shift-Calibrated Dirichlet Evidential Heads, we decompose uncertainty into Dissonance (conflicting evidence across classes $\implies$ continue to deeper layers) and Vacuity (lack of evidence $\implies$ flag for selective abstention if unresolved).

Tags: #Vision Transformers #Early-Exit Networks #Evidential Deep Learning #Dirichlet Distribution #Selective Prediction

Research Motivation & Feasibility

Very manageable experimental scope and a clean efficiency–reliability research question on Vision Transformers. The novelty burden here requires careful differentiation: both Early-Exit ViTs and Evidential Deep Learning are active areas, so we must directly address recent critiques of Evidential DL under shift (Jürgens et al., 2024).

Core Identified Research Gap

Existing Early-Exit Vision Transformers halt based on first-order softmax entropy, which cannot separate in-distribution class overlap (aleatoric) from out-of-distribution unfamiliarity (epistemic). Conversely, standard Evidential Deep Learning (Sensoy et al., 2018) placed only at the final layer suffers from vanishing evidence gradients and fails to use intermediate block trajectory consistency to detect covariate shift.

Basic Information

Field	Details
Core Focus	Clean Scoped Efficiency–Reliability Topic
Final Research Title	Shift-Calibrated Evidential Early Exiting for Selective Prediction in Resource-Constrained Vision Transformers Under Distribution Shift
Working Title	Vacuity-and-Dissonance Guided Routing Along Vision Transformer Depth
Research Area	Computer Vision, Uncertainty Quantification & Dynamic Neural Networks
Subfield	Evidential Deep Learning (EDL) & Early-Exit Vision Transformers
Target Academic Level	MSc Thesis (Highly Manageable Single-GPU Scope)
Current Status	Scoped & Literature Audited

Idea Summary

2.1 One-Sentence Core Thesis

Use second-order Dirichlet evidential heads across Vision Transformer blocks to separate dissonance (which benefits from deeper attention layers) from vacuity (which triggers selective abstention), corrected via cross-exit evidence trajectory regularization.

2.2 Research Motivation & Strategic Fit

Strengths: Extremely clean experimental setup. Using a pretrained DeiT-Small or ViT-Base backbone from timm, training $4\text{–}6$ intermediate evidential heads takes $< 2$ hours on a single GPU.
Why #4 (Honest Assessment of the Novelty Burden): Both Early-Exit ViTs and Evidential Deep Learning have substantial recent literature, and recent theoretical work (Jürgens et al., NeurIPS 2024; Bengs et al., ICML 2022) showed that naive EDL losses are not strictly proper scoring rules without careful regularization. To publish at a top venue, our novelty cannot be "just putting Sensoy et al. (2018) on a ViT"—it must be cross-depth evidence trajectory calibration and dissonance-vs-vacuity routing.

Problem Exploration & Identified Research Gap

3.1 Two Distinct Reasons an Early Exit Is Uncertain

Suppose at ViT Block $m=4$ (out of $12$), the [CLS] token representation $\mathbf{h}_4(x)$ is ambiguous:

1
Case 1: High Dissonance (Resolvable Fine-Grained Ambiguity): Both class $c_1$ ("Husky") and class $c_2$ ("Malamute") have strong positive evidence ($e_{4, c_1} = 12, e_{4, c_2} = 11$). Deeper attention blocks ($m = 5 \dots 12$) are specifically needed to resolve fine-grained spatial features.
2
Case 2: High Vacuity (Severe Corruption / OOD): None of the classes elicit positive evidence ($\mathbf{e}_4 \approx \mathbf{0} \implies u_4 = C / S_4 \approx 1$). Even worse, in standard networks, running deeper layers on corrupted inputs often causes overthinking (Kaya et al., 2019), fabricating spurious evidence at Block 12.

3.2 Explicit Research Gap

Existing Early-Exit ViTs (Bakhtiarnia et al., 2021;LGViT, Xu et al., 2023) use a single scalar halting threshold on softmax confidence or entropy, failing to distinguish between inputs that require deeper computation (dissonance) and inputs that should be rejected/abstained (vacuity). Meanwhile, single-exit Evidential DL fails to leverage the monotonic progression of evidence $S_1(x) \le S_2(x) \le \dots \le S_M(x)$ across network depth to detect abnormal feature drift.

Proposed Research Direction

4.1 Mathematical Formulation: Dirichlet Vacuity vs. Dissonance Routing

At exit block $m \in \{1, \dots, M\}$, an evidential head maps [CLS] token $\mathbf{h}_m(x)$ to non-negative evidence $\mathbf{e}_m = \exp(\text{clamp}(g_m(\mathbf{h}_m(x)), -10, 10)) \in \mathbb{R}_+^C$, parameterizing a Dirichlet distribution $\text{Dir}(\mathbf{p}_m \mid \boldsymbol{\alpha}_m)$ with concentration $\boldsymbol{\alpha}_m = \mathbf{e}_m + \mathbf{1}$ and total Dirichlet strength $S_m = \sum_{c=1}^{C} \alpha_{m,c}$.

We compute two orthogonal uncertainty quantities (Jøsang, 2016; Sensoy et al., 2018):

$$ \text{Vacuity (Lack of Evidence): } u_m(x) = \frac{C}{S_m(x)} \in (0, 1] $$

$$ \text{Dissonance (Conflicting Evidence): } \text{diss}_m(x) = \sum_{c=1}^{C} \left( \frac{e_{m,c}}{S_m - C} \frac{\sum_{j \neq c} e_{m,j}\, \text{Bal}(e_{m,c}, e_{m,j})}{\sum_{j \neq c} e_{m,j}} \right) $$

where $\text{Bal}(a, b) = 1 - |a - b| / (a + b)$ measures pairwise evidence conflict.

Halting Rule at Exit $m$:
If $u_m(x) < \tau_{\text{vac}}^{(m)}$ and $\text{diss}_m(x) < \tau_{\text{diss}}^{(m)}$ $\implies$ Early Exit & Predict $\hat{y}_m = \arg\max_c \alpha_{m,c}$.
If $\text{diss}_m(x) \ge \tau_{\text{diss}}^{(m)}$ $\implies$ Continue to Block $m+1$ (conflicting features need deeper attention refinement).
If at the final exit $M$ (or via monotonic trajectory drop $S_m(x) < S_{m-1}(x)$), $u_M(x) \ge \tau_{\text{vac}}^{(M)}$ $\implies$ Selective Abstention ($\perp$).

Interactive Diagram SpecificationMermaid Flow / Schedule

flowchart LR
    X["Input Image x"] --> B4["ViT Block m<br/>Dirichlet α_m = e_m + 1"]
    B4 --> DEC{"Evaluate (Vacuity u_m, Dissonance diss_m)"}
    DEC -- "Low u_m & Low diss_m" --> EXIT["Early Exit at Block m<br/>Save FLOPs"]
    DEC -- "High diss_m (Conflicting Classes)" --> NEXT["Route to Deeper Block m+1<br/>Refine Fine-Grained Features"]
    DEC -- "High u_m at Final Block M" --> ABST["Selective Abstention (⊥)<br/>Prevent Overconfident Error"]

Preliminary Technical Plan

Backbones: DeiT-Small (12 blocks, 384 dim) and Swin-Tiny with exit heads at blocks $\{3, 6, 9, 12\}$.
Datasets: CIFAR-100 / CIFAR-100-C, ImageNet-100 / ImageNet-100-C, and SVHN/Places365 as OOD abstention splits.
Baselines: Softmax Entropy Early-Exit, Patience-Based Early-Exit (PABEE, Zhou et al., 2020), Last-Layer Evidential DL (Sensoy et al., 2018), and Post-Hoc Temperature Scaled Early-Exit.
Metrics: Selective Risk-Coverage AURC at $50\%, 65\%, 80\%$ FLOPs budgets; OOD AUROC; Expected Calibration Error ($\text{ECE}$).

Expected Contributions

1
Dissonance-vs-Vacuity Depth Routing Policy: A principled separation between computational deferral (deeper layers) and epistemic abstention (human deferral).
2
Depth-Monotonic Evidence Regularizer: A training penalty enforcing that correct class evidence $e_{m, y}$ increases monotonically across depth on clean data while suppressing spurious evidence growth on OOD perturbations.

Feasibility, Scope, and Honest Risks

Feasibility: $10/10$ on a single GPU.
Main Risk & Defense: Reviewers familiar with Jürgens et al. (NeurIPS 2024) will question whether Dirichlet vacuity is calibrated without OOD exposure. Defense: Combine density-based feature normalization (Spectral Normalization / Mahalanobis distance on [CLS] tokens, Postels et al. / Charpentier et al., NatPN) with conformal risk calibration on the final abstention threshold.

Initial Literature Notes

[1]
M. Sensoy, L. Kaplan, and M. Kandemir, "Evidential Deep Learning to Quantify Classification Uncertainty," NeurIPS, 2018.
[2]
G. Xu et al., "LGViT: Dynamic Early Exiting for Accelerating Vision Transformer," ACM MM, 2023.
[3]
B. Charpentier, D. Zügner, and S. Günnemann, "Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts," NeurIPS, 2020.
[4]
M. Jürgens et al., "Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?" ICML / NeurIPS, 2024.

Idea Validation Checklist

Addressed known theoretical critique of vanilla EDL via density/trajectory regularizationVerified
Clear mathematical distinction between Dissonance (continue) and Vacuity (abstain)Verified
Fast single-GPU training setup with pretrained timm DeiT backbonesVerified
Plot empirical $(u_m, \text{diss}_m)$ scatter distributions across blocks $3, 6, 9, 12$ on CIFAR-100-CPending

Current Assessment

Criterion	Score	My Assessment Notes
Problem clarity	9 / 10	Very crisp formulation of early-exit + selective prediction.
Novelty potential	7 / 10	Crowded neighborhood; requires strong theoretical rigor on EDL calibration.
Technical feasibility	9 / 10	Easy to implement and debug on 1 GPU.
Resource feasibility	10 / 10	Minimal compute and storage required.
Dataset availability	10 / 10	Standard vision corruption benchmarks.
Publication potential	8 / 10	Solid fit for WACV, BMVC, ECCV/CVPR workshops, or Pattern Recognition.
Personal interest	9 / 10	Connects directly to my adaptive inference proposal work.

Next Actions

1
Implement the EvidentialExitHead with exponential evidence parameterization and compute vacuity and dissonance tensors across 4 exits of deit_small_patch16_224.
2
Verify whether high-dissonance samples at Block 6 actually get corrected by Block 12 more often than high-vacuity samples.

Open Questions and Working Notes

Does sharing a single low-rank projection across intermediate exit heads prevent early exits from distorting shallow ViT attention maps?