Rakibul Islam
Contact

Personal Research Knowledge Base · 11-Section Synthesis Format

Research Notes & Mathematical Syntheses

Working research notes written in my own workflow—connecting first-principles mathematical derivations, annotated PyTorch implementations, failure-mode analyses, and direct links to my research on uncertainty-aware adaptive AI, medical imaging, and graph representation learning.

Pinned Foundation Note · #01 Showing all 10 research notes
RN-2025-01 · Evergreen v2.4
Main Focus
Entropy, confidence, calibration
H[p(y|x)] = I[y; θ | x] + E_θ[H[p(y|x, θ)]]
11-Section Synthesis PyTorch + KaTeX
Uncertainty & Calibration Updated: Mar 02, 2025 Focus: Entropy, confidence, calibration

01. Understanding Predictive Uncertainty

A first-principles formulation of predictive uncertainty in deep classifiers—separating maximum softmax confidence from Shannon entropy, decomposing total uncertainty via mutual information, and diagnosing miscalibration with Expected Calibration Error (ECE) and Temperature Scaling.

14 min read · #Predictive Uncertainty #Shannon Entropy #Calibration #ECE
Open Research Note
01 · Uncertainty & Calibration RN-2025-01

Understanding Predictive Uncertainty

Focus: Entropy, confidence, calibration

A first-principles formulation of predictive uncertainty in deep classifiers—separating maximum softmax confidence from Shannon entropy, decomposing total uncertainty via mutual information, and diagnosing miscalibration with Expected Calibration Error (ECE) and Temperature Scaling.

H[p(y|x)] = I[y; θ | x] + E_θ[H[p(y|x, θ)]]
Predictive Uncertainty Shannon Entropy Calibration
14 min read Read Note →
02 · Robustness & Generalization RN-2025-02

Distribution Shift: A Practical Introduction

Focus: Covariate, label, concept shift

A rigorous breakdown of how joint distributions p(x, y) fracture between training and deployment—distinguishing covariate shift, label (prior) shift, and concept drift—paired with importance weighting, BBSE label-shift correction, and energy/Mahalanobis OOD detection.

p_train(x, y) != p_test(x, y) => Covariate / Label / Concept Shift
Distribution Shift Covariate Shift Label Shift
15 min read Read Note →
03 · Uncertainty & Calibration RN-2025-03

Deep Ensembles vs MC Dropout

Focus: Uncertainty estimation

An engineering and theoretical comparison of Monte Carlo (MC) Dropout and Deep Ensembles—analyzing why independent random initializations explore distinct loss basins while MC Dropout samples locally within a single mode, with latency/VRAM trade-offs and PyTorch code.

Var_θ[p(y|x, θ)] : Single-Basin (MC Dropout) vs Multi-Basin (Deep Ensembles)
Deep Ensembles MC Dropout Bayesian Deep Learning
15 min read Read Note →
04 · Adaptive Inference RN-2025-04

Early-Exit Neural Networks

Focus: Adaptive computation

A deep dive into sample-wise adaptive depth via Early-Exit Neural Networks—covering multi-exit joint optimization, calibrated entropy halting thresholds, gradient interference across shallow exits, and preventing accuracy degradation from network 'overthinking'.

m*(x) = min { m : H(p_m(y|x)) < τ_m } (Sample-Adaptive Halting)
Early-Exit Networks Adaptive Computation Dynamic Inference
14 min read Read Note →
05 · Adaptive Inference RN-2025-05

Mixture-of-Experts Architecture

Focus: Gating and expert selection

A comprehensive mathematical and systems note on Sparse Mixture-of-Experts (MoE)—deriving Top-K noisy gating, explaining why naive routers suffer from expert collapse, formulating the Switch/Mixtral auxiliary load-balancing loss, and implementing a sparse MoE layer in PyTorch.

y = Σ_{i ∈ TopK(g(x))} G_i(x) E_i(x) + L_aux = α N Σ_i f_i · P_i
Mixture of Experts Sparse Gating Top-K Routing
16 min read Read Note →
06 · Representation & Adaptation RN-2025-06

LoRA: Mathematical Foundations

Focus: Low-rank adaptation

A rigorous mathematical derivation of Low-Rank Adaptation (LoRA)—connecting the intrinsic dimensionality of over-parameterized fine-tuning to the ΔW = (α/r)BA decomposition, proving why asymmetric initialization (B=0, A~N) and α/r scaling stabilize hyperparameter transfer, and implementing mergeable LoRA layers in PyTorch.

W = W_0 + (α / r) B A, B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k}, r ≪ min(d, k)
LoRA Low-Rank Adaptation PEFT
15 min read Read Note →
07 · Evaluation & RAG RN-2025-07

RAG Evaluation Frameworks

Focus: Retrieval and generation metrics

A complete engineering and mathematical framework for evaluating Retrieval-Augmented Generation (RAG) pipelines—decomposing errors into retrieval rank quality (Recall@K, MRR, nDCG@K) and generation grounding (Context Precision, Atomic Claim Faithfulness, and Answer Relevance).

Faithfulness = |{c ∈ Claims(a) : C ⊨ c}| / |Claims(a)|, nDCG@K = DCG@K / IDCG@K
RAG Evaluation Information Retrieval nDCG
15 min read Read Note →
08 · Representation & Adaptation RN-2025-08

Graph Attention Networks

Focus: Attention-based message passing

A first-principles derivation of Graph Attention Networks (GAT and GATv2)—contrasting static degree-normalized GCN message passing with anisotropic neighborhood attention, proving why original GAT computes only 'static' attention and how GATv2 fixes it, and implementing a multi-head GATv2 layer in PyTorch.

α_ij = softmax_j( a^T LeakyReLU(W_l h_i + W_r h_j) ), h'_i = ||_{k=1}^K σ(Σ_{j∈N_i} α_ij^(k) W^(k) h_j)
Graph Attention Networks GAT GATv2
15 min read Read Note →
09 · Robustness & Generalization RN-2025-09

Test-Time Adaptation

Focus: Learning during inference

A practical and mathematical analysis of Test-Time Adaptation (TTA)—explaining how a deployed model can adapt online to unlabeled covariate-shifted test streams without access to source training data, covering AdaBN, TENT entropy minimization on affine normalization parameters, and catastrophic collapse failure modes under small batches or label skew.

min_{γ, β} H(p_θ(y | x_test)) = -Σ_c p_θ(y=c | x_test) log p_θ(y=c | x_test)
Test-Time Adaptation TTA TENT
15 min read Read Note →
10 · Decision Theory & Safety RN-2025-10

Risk-Constrained AI Decision-Making

Focus: Selective prediction and decision theory

The capstone decision-theoretic synthesis connecting uncertainty quantification to real-world safety—deriving Bayes-optimal cost-sensitive actions, Chow's optimal abstention rule, Selective Classification (Risk-Coverage curves and AURC), and distribution-free Split Conformal Prediction sets in PyTorch.

R(f, g) = E[ℓ(f(x), y) g(x)] / E[g(x)] ≤ r*, P(Y_test ∈ C(X_test)) ≥ 1 - α
Selective Prediction Decision Theory Risk-Coverage Curve
16 min read Read Note →

Structured Index

Complete Research Notes Syllabus (01–10)

View 11-Section Research Note Standard
# Research Note Main Focus Research Theme Action
01 Understanding Predictive Uncertainty Entropy, confidence, calibration Uncertainty & Calibration Open →
02 Distribution Shift: A Practical Introduction Covariate, label, concept shift Robustness & Generalization Open →
03 Deep Ensembles vs MC Dropout Uncertainty estimation Uncertainty & Calibration Open →
04 Early-Exit Neural Networks Adaptive computation Adaptive Inference Open →
05 Mixture-of-Experts Architecture Gating and expert selection Adaptive Inference Open →
06 LoRA: Mathematical Foundations Low-rank adaptation Representation & Adaptation Open →
07 RAG Evaluation Frameworks Retrieval and generation metrics Evaluation & RAG Open →
08 Graph Attention Networks Attention-based message passing Representation & Adaptation Open →
09 Test-Time Adaptation Learning during inference Robustness & Generalization Open →
10 Risk-Constrained AI Decision-Making Selective prediction and decision theory Decision Theory & Safety Open →

Connected Writing & Software

Explore Companion Technical Essays & Publications

Each research note connects directly to my published IEEE/ACM conference papers and my 10-part essay series on reliable AI systems.

Read 10 AI Essays Conference Papers