10. Risk-Constrained AI Decision-Making

• Primary References:

1. 1
   Chow, C. K. (1970). On Optimum Recognition Error and Reject Tradeoff. IEEE Transactions on Information Theory, 16(1), 41–46.
2. 2
   El-Yaniv, R., & Wiener, Y. (2010). On the Foundations of Noise-free Selective Classification. Journal of Machine Learning Research (JMLR), 11, 1605–1641.
3. 3
   Geifman, Y., & El-Yaniv, R. (2017). Selective Classification for Deep Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS 2017).
4. 4
   Angelopoulos, A. N., & Bates, S. (2023). Conformal Prediction: A Gentle Introduction. Foundations and Trends in Machine Learning, 16(4), 494–591.

• Why I Wrote This Note: In our paper on Risk Minimization by Early Readmission Prediction for Diabetes Patients (IEEE BECITHCON 2024), discharging a high-risk patient without follow-up (False Negative) incurs vastly higher clinical and financial cost than scheduling a preventative check-up for a stable patient (False Positive). I wrote this note to connect cost-sensitive decision theory, selective abstention, and conformal prediction into a single unified mathematical pipeline.

• Accuracy Assumes Symmetric \(0\text{–}1\) Loss: Standard \(\arg\max_c p(y=c\mid x)\) is only Bayes-optimal when all mistakes cost the same. Under a cost matrix \(L(a, y)\), the optimal decision minimizes expected posterior loss \(R(a \mid x) = \sum_{y} L(a, y) p(y \mid x)\), which depends directly on probabilistic calibration.
• Chow's Optimal Reject Rule (1970): When an abstention action \(a_{\text{reject}}\) incurs fixed review cost \(c \in (0, 1 - 1/C)\) while a wrong prediction costs \(1\), the Bayes-optimal policy predicts \(\hat{y}(x) = \arg\max_y p(y\mid x)\) if and only if \(\max_y p(y\mid x) \ge 1 - c\), and abstains otherwise.
• Selective Risk vs. Coverage (\(R(f, g)\) vs. \(\phi(g)\)): A selective classifier \((f, g)\) pairs a predictor \(f: \mathcal{X} \to \mathcal{Y}\) with a selection function \(g: \mathcal{X} \to \{0, 1\}\) thresholded on a confidence/uncertainty score \(\kappa(x) \ge \tau\). Sweeping \(\tau\) traces the Risk-Coverage curve, summarized globally by \(\text{AURC}\) (lower is better).
• Split Conformal Prediction: Instead of abstaining completely, Conformal Prediction outputs a variable-sized prediction set \(\mathcal{C}(x) \subseteq \{1, \dots, C\}\) (e.g., \(\{\text{Pneumonia}, \text{TB}\}\)) whose size \(|\mathcal{C}(x)|\) adapts automatically to input difficulty and satisfies the exact finite-sample marginal guarantee \(\mathbb{P}(Y_{\text{test}} \in \mathcal{C}(X_{\text{test}})) \ge 1 - \alpha\) for any black-box neural network under exchangeability.

4.1 Asymmetric Bayes Decision Theory & Chow's Reject Rule

Let \(\mathcal{Y} = \{1, \dots, C\}\) be the true state space and \(\mathcal{A} = \{1, \dots, C\} \cup \{\perp\}\) be the action space augmented with the abstention (reject) action \(\perp\). Given a loss matrix \(L: \mathcal{A} \times \mathcal{Y} \to \mathbb{R}_{\ge 0}\), the conditional Bayes risk of taking action \(a \in \mathcal{A}\) on input \(x\) is:

In Chow's (1970) setting, correct classification incurs cost \(0\), misclassification incurs cost \(1\), and abstention (\(a = \perp\)) incurs a fixed human-review cost \(c \in (0, 1)\):

For any prediction action \(a \in \{1, \dots, C\}\), its conditional risk is \(R(a \mid x) = \sum_{y \neq a} p(y\mid x) = 1 - p(y=a \mid x)\), whereas the risk of abstaining is \(R(\perp \mid x) = c \sum_y p(y\mid x) = c\). Minimizing \(R(a \mid x)\) over \(a \in \mathcal{A}\) immediately yields Chow's Optimal Rule:

4.2 Selective Classification: Coverage, Selective Risk, and AURC

When the explicit dollar/clinical cost \(c\) of abstention is unknown a priori, Geifman & El-Yaniv (2017) formulate Risk-Constrained Selective Prediction. A selective classifier is a pair \((f, g_\tau)\) where \(f: \mathcal{X} \to \mathcal{Y}\) is the prediction head and \(g_\tau(x) = \mathbf{1}\{\kappa(x) \ge \tau\} \in \{0, 1\}\) is a binary gating function controlled by a confidence score \(\kappa(x)\) (such as maximum softmax probability, negative Shannon entropy \(-\mathcal{H}(\hat{p})\), or negative epistemic mutual information \(-\mathcal{I}[y; \theta \mid x]\)).

We define two fundamental population quantities over \(p(x, y)\):

1. 1
   Coverage (\(\phi(g_\tau)\)): The fraction of the population on which the system commits to an automated prediction:

1. 1
   Selective Risk (\(R(f, g_\tau)\)): The expected loss conditioned strictly on the accepted subset:

Given a safety specification—such as "Automated chest X-ray screening must maintain selective error rate \(R(f, g_\tau) \le r^\star = 0.01\) (\(99\%\) accuracy on accepted scans)"—the Risk-Constrained Optimization Objective maximizes coverage subject to the risk ceiling \(r^\star\):

Sweeping \(\tau\) from \(-\infty\) (\(\phi = 1.0\)) to \(+\infty\) (\(\phi \to 0\)) traces the Risk-Coverage (RC) Curve. The overall quality of the uncertainty score \(\kappa(x)\) for selective prediction is measured by the Area Under the Risk-Coverage Curve (\(\text{AURC}\)):

where \(\hat{R}_k\) is the empirical risk on the top-\(k\) most confident samples sorted in descending order of \(\kappa(x_{(1)}) \ge \kappa(x_{(2)}) \ge \dots \ge \kappa(x_{(N)})\).

flowchart LR
    X["Input x"] --> MOD["Calibrated Model / Ensemble"]
    MOD --> KAP["Uncertainty / Non-Conformity Score s(x, y)"]
    KAP --> SEL["1. Selective Prediction (g_τ)<br/>κ(x) ≥ τ*  ⇒  Predict ŷ (Risk ≤ r*)<br/>κ(x) < τ*  ⇒  Abstain (⊥)"]
    KAP --> CP["2. Split Conformal Prediction<br/>C(x) = {y : s(x, y) ≤ q̂_{1-α}}<br/>Guarantee: P(Y ∈ C(x)) ≥ 1 - α"]

4.3 Distribution-Free Split Conformal Prediction

What if we want a strict finite-sample mathematical guarantee without assuming the model's probabilities are perfectly calibrated? Given a held-out calibration dataset \(\mathcal{D}_{\text{cal}} = \{(X_i, Y_i)\}_{i=1}^{n}\) exchangeable with \((X_{\text{test}}, Y_{\text{test}})\) and a user-chosen miscoverage rate \(\alpha \in (0, 1)\) (e.g., \(\alpha = 0.05\) for \(95\%\) coverage):

1. 1
   Define a Non-Conformity Score \(s(x, y) \in \mathbb{R}\) measuring how implausible label \(y\) is for input \(x\), e.g., \(s(x, y) = 1 - \hat{p}_\theta(y \mid x)\), or Adaptive Prediction Sets (APS, Romano et al., NeurIPS 2020) summing sorted probabilities up to the true class \(y\).
2. 2
   Compute non-conformity scores \(S_i = s(X_i, Y_i)\) for all \(i \in \{1, \dots, n\}\) in \(\mathcal{D}_{\text{cal}}\).
3. 3
   Compute the finite-sample corrected \((1 - \alpha)\) empirical quantile \(\hat{q}_{1-\alpha}\):

1. 1
   Form the prediction set for any new test input \(X_{\text{test}}\):

Theorem (Vovk et al., 2005; Lei et al., 2018): Under exchangeability of \(\{(X_1, Y_1), \dots, (X_n, Y_n), (X_{\text{test}}, Y_{\text{test}})\}\), if \(S_i\) are almost surely distinct:

Notice how \(|\mathcal{C}(X_{\text{test}})|\) acts as a natural decision controller:

• \(|\mathcal{C}(x)| = 1\): Singleton set \(\implies\) High-confidence automated decision!
• \(|\mathcal{C}(x)| > 1\): Ambiguous differential diagnosis set (e.g., \(2\) candidate classes) \(\implies\) Present narrowed choices to human specialist.
• \(|\mathcal{C}(x)| = 0\): Empty set \(\implies\) Extreme anomaly / OOD alert.

4.4 PyTorch Implementation: AURC, Risk-Constrained Threshold Finder, & Split Conformal Predictor

import math
import torch

def compute_risk_coverage_and_aurc(confidences: torch.Tensor, is_correct: torch.Tensor) -> dict[str, torch.Tensor | float]:
    # Sort samples from highest confidence to lowest confidence to compute exact RC curve and AURC
    sorted_idx = torch.argsort(confidences, descending=True)
    sorted_errors = (1.0 - is_correct.float())[sorted_idx]

    n = sorted_errors.numel()
    k_range = torch.arange(1, n + 1, device=confidences.device, dtype=torch.float32)
    coverages = k_range / float(n)
    selective_risks = torch.cumsum(sorted_errors, dim=0) / k_range
    aurc = float(selective_risks.mean().item())

    return {"coverages": coverages, "selective_risks": selective_risks, "aurc": aurc}

def find_threshold_for_target_risk(
    confidences: torch.Tensor, is_correct: torch.Tensor, target_risk: float = 0.02
) -> tuple[float, float, float]:
    # Find the threshold tau* that maximizes coverage while keeping empirical selective risk <= target_risk
    sorted_conf, sorted_idx = torch.sort(confidences, descending=True)
    sorted_errors = (1.0 - is_correct.float())[sorted_idx]
    n = sorted_errors.numel()
    k_range = torch.arange(1, n + 1, device=confidences.device, dtype=torch.float32)
    selective_risks = torch.cumsum(sorted_errors, dim=0) / k_range

    valid_k = torch.where(selective_risks <= target_risk)[0]
    if valid_k.numel() == 0:
        return 1.0, 0.0, float(selective_risks[0].item())
    best_idx = int(valid_k[-1].item())
    return (
        float(sorted_conf[best_idx].item()),
        float((best_idx + 1) / n),
        float(selective_risks[best_idx].item()),
    )

def split_conformal_prediction_sets(
    cal_probs: torch.Tensor,   # (n_cal, C) calibrated probabilities on calibration split
    cal_labels: torch.Tensor,  # (n_cal,) ground-truth class indices in {0, ..., C-1}
    test_probs: torch.Tensor,  # (n_test, C) probabilities on test split
    alpha: float = 0.05,
) -> tuple[torch.Tensor, float]:
    # Construct (1 - alpha) marginal coverage prediction sets C(X_test)
    n_cal = cal_probs.size(0)
    true_class_probs = cal_probs[torch.arange(n_cal, device=cal_probs.device), cal_labels]
    nonconformity_scores = 1.0 - true_class_probs  # (n_cal,)

    q_level = min(math.ceil((n_cal + 1) * (1.0 - alpha)) / float(n_cal), 1.0)
    q_hat = float(torch.quantile(nonconformity_scores, q_level, interpolation="higher").item())

    # Include class y in C(x) iff 1 - p(y|x) <= q_hat  <=>  p(y|x) >= 1 - q_hat
    pred_sets_mask = (1.0 - test_probs) <= q_hat  # boolean tensor of shape (n_test, C)
    return pred_sets_mask, q_hat

This note represents the unifying decision layer across my entire body of research:

1. 1
   Risk Minimization by Early Readmission Prediction for Diabetes Patients (IEEE BECITHCON 2024): By replacing a naive \(0.5\) probability cutoff with an asymmetric cost matrix \(L(\text{False Negative}) \gg L(\text{False Positive})\) and a selective abstention zone \([1 - c_{\text{FN}},\, c_{\text{FP}}]\), hospital care teams can automate low-risk discharges while routing borderline cases to nurse case managers.
2. 2
   Tuberculosis Screening (ICEEICT 2024) & TIG-UNet++ Lung Segmentation (ICICT 2025): Using Split Conformal Prediction on chest radiographs guarantees that the true diagnosis is contained within \(\mathcal{C}(X_{\text{test}})\) at \(1 - \alpha = 99\%\) sensitivity, flagging any scan with \(|\mathcal{C}(x)| > 1\) for radiologist review.
3. 3
   Conformal Early-Exit & RAG Abstention: In my ongoing work on Adaptive Inference, replacing heuristic entropy thresholds in Early-Exit Neural Networks and RAG Evaluation Frameworks with Conformal Risk Control (Angelopoulos et al., 2022) turns heuristic speedups into certified, risk-bounded AI systems.

• Why calibration Still Matters for Conformal Prediction: Even though Conformal Prediction guarantees \(1 - \alpha\) coverage for an uncalibrated model, poor calibration inflates the non-conformity quantile \(\hat{q}_{1-\alpha}\), causing the average set size \(\mathbb{E}[|\mathcal{C}(X)|]\) to bloat! Applying Temperature Scaling (Understanding Predictive Uncertainty) or Deep Ensembles (Deep Ensembles vs MC Dropout) before conformal calibration shrinks average prediction set size \(|\mathcal{C}(X)|\) while maintaining the exact same \(1 - \alpha\) coverage guarantee.
• Excess AURC (E-AURC): Because raw \(\text{AURC}\) depends on the classifier's base error rate at \(100\%\) coverage, comparing uncertainty estimators across backbones with different base accuracies requires subtracting the oracle AURC (\(\text{E-AURC} = \text{AURC} - \text{AURC}^\star\)).

• Implement compute_risk_coverage_and_aurc, find_threshold_for_target_risk, and split_conformal_prediction_sets in PyTorch.Verified
• Implement Class-Conditional (Mondrian) Conformal Prediction (computing separate quantiles \(\hat{q}_{1-\alpha}^{(y)}\) per class \(y\)) to guarantee equal sensitivity on minority disease classes.Pending
• Integrate Weighted Conformal Prediction (Tibshirani et al., NeurIPS 2019) using importance weights \(w(x) = p_T(x)/p_S(x)\) from Distribution Shift: A Practical Introduction to maintain valid coverage under covariate shift.Pending

"Uncertainty quantification tells a model how much it does not know; decision theory tells the system what it is allowed to do about it."

Selective Risk-Coverage & Split Conformal Quantile Card:

• Connected Research Notes:
• Understanding Predictive Uncertainty — Supplies the calibrated probabilities and Shannon entropy scores used as \(\kappa(x)\) in selective prediction.
• Deep Ensembles vs MC Dropout — Epistemic mutual information \(\mathcal{I}[y; \theta \mid x]\) achieves lower \(\text{AURC}\) than single-model softmax under dataset shift.
• Distribution Shift: A Practical Introduction — Explains when exchangeability breaks and how weighted conformal prediction restores coverage.
• Early-Exit Neural Networks — Halting early at stage \(m\) is mathematically isomorphic to selective prediction along network depth.
• Test-Time Adaptation — High-entropy samples rejected by the selective gate \(g_\tau(x) = 0\) are also masked from online TENT updates.