06. Calibration-Preserving Model Quantization for Shift-Robust Edge Vision

Executive Summary Working Title: Margin-and-Entropy Preserving Low-Bit Quantization Under Covariate Shift

Before any vision model is deployed onto an industrial camera, medical diagnostic tablet, or autonomous drone, its weights and activations are almost always quantized from 32-bit floating point ($\text{FP32}$) to 8-bit or 4-bit integers ($\text{W8A8}, \text{W4A8}, \text{W4A4}$). Quantization papers routinely report that top-1 accuracy drops by less than $0.5\text{–}1.0\%$ on clean test splits. Yet when I inspected the predictive probabilities and out-of-distribution energy scores of quantized models under covariate shift, a troubling discrepancy emerged: quantization clipping systematically distorts logit margins and destroys probabilistic calibration. Even when $\arg\max_c z_c$ stays the same on clean inputs, clamping extreme activation channels alters the temperature profile of the final layer, inflating Expected Calibration Error ($\text{ECE}$) by $2\times\text{–}4\times$ on corrupted inputs and causing downstream selective prediction gates to fail. This topic develops Calibration-Preserving Quantization (CP-PTQ) to guarantee that low-bit edge models remain trustworthy under distribution shift.

Tags: #Model Quantization #PTQ / QAT #Calibration (ECE) #Edge Vision #INT4 / INT8 #Shift Robustness

Research Motivation & Feasibility

Directly useful for real-world model deployment and efficient edge inference. To make this a genuine research contribution rather than an engineering exercise, the focus is strictly on preserving probabilistic calibration and shift robustness during low-bit quantization, not merely applying off-the-shelf PTQ.

Core Identified Research Gap

Standard Post-Training Quantization (PTQ, e.g., BRECQ, Li et al., 2021; AdaRound, Nagel et al., 2020) minimizes layer-wise activation MSE or clean task cross-entropy on a small calibration set. However, clipping activation outliers to fit 4-bit or 8-bit grids compresses logit dynamic range unevenly across classes—inflating Expected Calibration Error (ECE) under covariate shift and erasing the logit-norm / energy signal needed for OOD detection.

Basic Information

Field	Details
Core Focus	Core Efficient & Reliable Deployment Topic
Final Research Title	Calibration-Preserving Model Quantization for Shift-Robust Edge Vision
Working Title	Preserving Logit Geometry, Calibration, and Energy Separability in Low-Bit Post-Training Quantization
Research Area	Efficient Deep Learning, Model Compression & Uncertainty Calibration
Subfield	Post-Training Quantization (PTQ) & Quantization-Aware Training (QAT)
Target Academic Level	MSc Thesis (Fast Single-GPU Iteration)
Current Status	Problem Formulated

Idea Summary

2.1 One-Sentence Core Thesis

Formulate a block-wise Post-Training Quantization objective that regularizes both multi-temperature probability divergence and Helmholtz free-energy preservation across clean and synthetically perturbed calibration samples, preventing low-bit clipping from degrading ECE and OOD detection.

2.2 Research Motivation & Strategic Fit

High Practical Relevance: Every edge AI engineer quantizes models, and every safety-critical system needs calibrated probabilities.
Why #6: Care must be taken not to frame the project as "just running INT8 PTQ and applying Temperature Scaling at the end." Our contribution must prove why post-hoc Temperature Scaling cannot recover information lost to non-uniform channel clipping in 4-bit/8-bit ViT and CNN blocks, and solve it inside the quantization rounding/scale optimization itself.

Problem Exploration & Identified Research Gap

3.1 How Quantization Clipping Distorts Uncertainty

In uniform affine quantization with bit-width $b$, a real weight or activation tensor $x$ is mapped to an integer grid via scale $s > 0$ and zero-point $z$:

$$ \hat{x}_q = s \cdot \left( \text{clamp}\!\left( \left\lfloor \frac{x}{s} \right\rceil + z,\, 0,\, 2^b - 1 \right) - z \right) $$

Why does this hurt calibration and shift robustness?

1
Asymmetric Tail Compression in ViTs: Vision Transformers exhibit extreme post-LayerNorm and post-GELU outlier channels. When PTQ clips these outliers to reduce MSE on average tokens, the penultimate feature norm $\|h_q(x)\|_2$ saturates on high-confidence and OOD inputs alike.
2
Irreversible Rank Swap in Confidence Scores: Because different channels contribute to different classes, clipping channel $k$ reduces the winning logit $z_{(1)}$ on tail classes more than on head classes. A single scalar post-hoc temperature $T$ cannot undo class-asymmetric logit distortion!

3.2 Explicit Research Gap

State-of-the-art PTQ algorithms (AdaRound, Nagel et al., ICML 2020; BRECQ, Li et al., ICLR 2021; QDrop, Wei et al., ICLR 2022) optimize the block reconstruction loss using a local Fisher/Hessian approximation of task cross-entropy on clean calibration images. *No existing PTQ framework explicitly constrains the quantized network to preserve the full calibration curve (Brier/ECE) and Helmholtz energy norm of the FP32 teacher across distribution shifts.*

Proposed Research Direction

4.1 Mathematical Formulation

Given a small unlabeled calibration set $\mathcal{D}_{\text{cal}}$ ($N = 1024$ images) augmented with lightweight synthetic covariate perturbations $\tilde{x} \sim \mathcal{T}_{\text{shift}}(x)$, let $\mathbf{z}_{\text{FP}}(x)$ and $\mathbf{z}_{Q}(x; \mathbf{s}, \mathbf{v})$ denote the logits of the full-precision model and the quantized model parameterized by step sizes $\mathbf{s}$ and AdaRound soft-rounding variables $\mathbf{v} \in [0, 1]^P$. We minimize a Calibration-and-Energy Preserving Quantization Objective:

$$ \min_{\mathbf{s}, \mathbf{v}} \;\mathbb{E}_{x \in \mathcal{D}_{\text{cal}} \cup \tilde{\mathcal{D}}_{\text{cal}}} \left[ \sum_{T \in \mathcal{T}} T^2 \text{KL}\Big( \sigma(\mathbf{z}_{\text{FP}}(x)/T) \;\Big\|\; \sigma(\mathbf{z}_{Q}(x)/T) \Big) + \lambda_E \Big( E_{\text{FP}}(x) - E_{Q}(x) \Big)^2 + \lambda_{\text{reg}} \mathcal{R}_{\text{round}}(\mathbf{v}) \right] $$

where optimizing across a multi-temperature grid $\mathcal{T} = \{0.5, 1.0, 2.0, 4.0\}$ forces both the peak confidence ($T=0.5$) and tail entropy ($T=4.0$) of $\mathbf{z}_Q$ to match $\mathbf{z}_{\text{FP}}$, while the energy term preserves log-partition magnitudes $E(x) = -\log\sum_c \exp(z_c)$.

Interactive Diagram SpecificationMermaid Flow / Schedule

flowchart LR
    CAL["1,024 Calibration Images<br/>+ Mild Shift Perturbations"] --> FP["Frozen FP32 Teacher<br/>Logits z_FP & Energy E_FP"]
    CAL --> QMOD["Quantized Student (W4A8 / W4A4)<br/>Scale s & Rounding v"]
    FP & QMOD --> LOSS["Multi-Temp KL + Energy Preservation<br/>+ Hessian Block Reconstruction"]
    LOSS -- "Optimize (s, v)" --> QMOD

Preliminary Technical Plan

Models & Bit-Widths: ResNet-50, MobileNetV3-Large, DeiT-Small, and Swin-Tiny quantized to W8A8, W4A8, and W4A4.
Datasets: ImageNet-1K (clean), ImageNet-C (15 corruptions), ImageNet-R, and CIFAR-100-C.
Baselines: MinMax PTQ, OMSE, AdaRound (ICML 2020), BRECQ (ICLR 2021), QDrop (ICLR 2022), plus each baseline followed by post-hoc Temperature Scaling.
Metrics: Clean & Corrupted Top-1 Accuracy, Expected Calibration Error ($\text{ECE}$), Brier Score, Selective Prediction AURC, and OOD Detection AUROC.

Expected Contributions

1
Systematic Audit of Quantization-Induced Miscalibration: First comprehensive benchmark showing how W4A8/W4A4 PTQ degrades ECE, AURC, and OOD energy scores under covariate shift even when clean Top-1 accuracy appears intact.
2
CP-PTQ Algorithm: A fast, 1024-sample post-training quantization objective that restores FP32 calibration and shift robustness without full-dataset Quantization-Aware Training.

Feasibility, Scope, and Honest Risks

Feasibility: Extremely high ($10/10$). PTQ only optimizes over $1,024$ calibration images—no full ImageNet training epochs are required! A complete W4A8 PTQ run on ResNet-50 or DeiT-Small takes 15–25 minutes on one GPU.
Main Risk & Fallback: At extreme W4A4 on ViTs, activation outliers in post-GELU layers may be too large for uniform quantizers regardless of the loss function. Fallback: Integrate channel-wise activation migration (SmoothQuant / RepQ-ViT scaling) prior to running our CP-PTQ rounding objective.

Initial Literature Notes

[1]
M. Nagel et al., "Up or Down? Adaptive Rounding for Post-Training Quantization (AdaRound)," ICML, 2020.
[2]
Y. Li et al., "BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction," ICLR, 2021.
[3]
X. Wei et al., "QDrop: Randomly Dropping Quantization for Extremely Low-Bit Post-Training Quantization," ICLR, 2022.
[4]
Z. Li et al., "RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers," ICCV, 2023.

Idea Validation Checklist

Clear distinction from naive PTQ: targets ECE, AURC, and Energy preservation under shiftVerified
Ultra-fast single-GPU execution ($1,024$ calibration samples, $<25\text{ mins}$ per run)Verified
Addresses both CNNs (ResNet/MobileNet) and ViTs (DeiT/Swin)Verified
Run initial W4A8 BRECQ vs. FP32 reliability diagram comparison on CIFAR-100-CPending

Current Assessment

Criterion	Score	My Assessment Notes
Problem clarity	9 / 10	Quantization vs. calibration under shift is a sharp, practical question.
Novelty potential	8 / 10	Multi-temperature KL + Energy preservation inside PTQ is clean and under-studied.
Technical feasibility	9 / 10	Builds directly on open-source BRECQ/QDrop PyTorch codebases.
Resource feasibility	10 / 10	Only needs 1,024 images for calibration!
Dataset availability	10 / 10	Standard public benchmarks.
Publication potential	8 / 10	Great fit for CVPR/ECCV/ICCV, WACV, or MLSys/TMLR.
Personal interest	8 / 10	Strong bridge between hardware deployment and uncertainty calibration.

Next Actions

1
Measure FP32 vs. W8A8 vs. W4A8 ECE and AURC on DeiT-Small across ImageNet-C severities 1–5.
2
Implement the multi-temperature KL + Helmholtz energy loss wrapper around AdaRound/QDrop.

Open Questions and Working Notes

Does quantizing the final classifier linear layer to 8-bit while keeping intermediate blocks at 4-bit recover most of the ECE loss, or does calibration drift originate in middle attention blocks?