11. Evidence-Grounded Multimodal Retrieval-Augmented Generation with Risk-Aware Abstention for Industrial Maintenance

2.1 One-Sentence Core Thesis

Decompose technical maintenance queries into atomic modality-tagged requirements \(\mathcal{R}(q) = \{r_1, \dots, r_M\}\) and compute a bottleneck cross-modal entailment score over retrieved document patches and text chunks, abstaining with a targeted missing-evidence request whenever the minimum sub-fact entailment falls below a conformal risk threshold.

2.2 Research Motivation & Strategic Fit

• Strengths: Directly builds on my RAG Evaluation Frameworks and Risk-Constrained AI Decision-Making research notes, and solves a high-value enterprise problem.
• Why #11: Many student RAG projects get rejected by reviewers for being "system integration wrappers around APIs." To succeed, the core contribution must be the formal cross-modal sufficiency estimator and conformal abstention theorem, evaluated on controlled counterfactual/incomplete-evidence benchmarks.

3.1 The "Partial-Evidence Hallucination" Trap in Multimodal RAG

Suppose a query asks: "What is the bypass valve sequence and maximum pressure limit for Pump Model B-400 during cold start?"

• Retrieved Text Chunk \(d_1\) describes the cold-start valve sequence for Pump B-400.
• Retrieved Diagram/Table \(d_2\) shows pressure limits, but for Pump B-200 (because the B-400 table was not indexed or is missing).
• What Standard MM-RAG Does: Average retrieval cosine similarity is high (\(0.84\)), so the VLM splices the B-200 pressure limit into the B-400 procedure!
• Why Mean Retrieval Score Fails: Averaging relevance across chunks hides the fact that a conjunction of sub-requirements (\(r_1 \wedge r_2\)) is only as strong as its weakest link (\(\min_m\)).

3.2 Explicit Research Gap

Existing abstention methods in RAG (e.g., CRAG, Yan et al., 2024; Self-RAG, Asai et al., 2024) score overall passage relevance in text-only settings and do not verify conjunctive entity-and-modality alignment between visual diagram regions and textual specifications, nor do they provide finite-sample false-answer guarantees under missing modalities.

4.1 Mathematical Formulation: Conjunctive Sub-Fact Sufficiency & Conformal Abstention

Given query \(q\) and retrieved multimodal context \(\mathcal{C}_K = \{\text{text chunks } \mathcal{T}_K, \text{visual page patches } \mathcal{V}_K\}\):

1. 1
   Atomic Requirement Decomposition: Decompose \(q\) into \(M\) required information slots \(\mathcal{R}(q) = \{r_1, \dots, r_M\}\) with entity constraints \(E(r_m)\) (e.g., Model = B-400).
2. 2
   Conjunctive Min-Max Sufficiency Score: Using a lightweight cross-modal NLI/VQA verifier \(v_\theta(c, r_m) \in [0, 1]\) and an entity-mismatch penalty \(\psi_{\text{ent}}(c, r_m)\), the sufficiency score of \(\mathcal{C}_K\) for query \(q\) is governed by the fuzzy logical AND (\(\min\) over requirements, \(\max\) over evidence chunks):

1. 1
   Two-Stage Conformal Abstention:

• Pre-Generation Gate: If \(\text{Suff}(q, \mathcal{C}_K) < \hat{q}_{1-\alpha}^{\text{pre}}\), abstain immediately and report the exact unsatisfied requirement \(r_{m^\star} = \arg\min_m \max_c (\dots)\).
• Post-Generation Claim Gate: For generated response \(a\), verify atomic claim faithfulness \(\min_{s \in \mathcal{S}(a)} \mathbb{P}(\mathcal{C}_K \models s) \ge \hat{q}_{1-\alpha}^{\text{post}}\).

flowchart LR
    Q["Maintenance Query q"] --> DEC["Atomic Requirement Decomposer<br/>R(q) = {r₁, r₂, ..., r_M}"]
    Q --> RET["Multimodal Retriever (ColPali + BM25)<br/>Text Chunks T_K + Diagram Patches V_K"]
    DEC & RET --> SUFF["Conjunctive Sufficiency Verifier<br/>Suff(q, C_K) = min_m max_c P(c ⊨ r_m)"]
    SUFF --> GATE{"Suff(q, C_K) ≥ q̂_{1-α} ?"}
    GATE -- "Sufficient Cross-Modal Evidence" --> VLM["Grounded VLM Generation<br/>with Region/Chunk Citations"]
    GATE -- "Missing Modality or Entity Conflict" --> ABST["Abstain (⊥) + Report Missing Slot r_{m*}"]

• Models: ColPali-v1.2 + BGE-M3 (hybrid multimodal retrieval) and Qwen2-VL-7B-Instruct (generator & visual entailment probe).
• Controlled Perturbation Benchmark: Construct a 4-condition evaluation suite from ViDoRe and TechQA:

1. 1
   Complete Evidence (both text & diagram present),
2. 2
   Modality Ablation (diagram removed, text retained),
3. 3
   Entity Swap Conflict (diagram from sibling model B-200 paired with text from B-400),
4. 4
   Unanswerable Out-of-Scope Query.

• Metrics: Hallucination Escape Rate at \(\alpha = 5\%\) risk ceiling, Selective F1 / AURC, and Missing-Slot Identification Accuracy.

1. 1
   Conjunctive Min-Max Cross-Modal Sufficiency Metric: Outperforms mean retrieval similarity and holistic LLM self-confidence by explicitly catching missing conjuncts and sibling-entity conflicts.
2. 2
   Counterfactual Multimodal Abstention Benchmark: A systematic testbed evaluating MM-RAG under modality dropout and entity-swap distractors.

• Feasibility: Runs entirely on a single 24GB GPU using 4-bit/8-bit Qwen2-VL-7B and cached ColPali embeddings.
• Main Risk & Mitigation: If query decomposition into \(\mathcal{R}(q)\) is noisy on short queries, \(\min_m\) may be overly conservative. Mitigation: Weight each requirement \(r_m\) by its predicted necessity score \(w_m \in [0, 1]\) using a soft-min (\(\text{LogSumExp}\) temperature relaxation).

• [1]
  M. Faysse et al., "ColPali: Efficient Document Retrieval with Vision Language Models," ICLR, 2025.
• [2]
  A. Asai et al., "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection," ICLR, 2024.
• [3]
  S. Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation," EACL, 2024.
• [4]
  B. Y. Lin et al. / S.-Q. Yan et al., "Corrective Retrieval Augmented Generation (CRAG)," arXiv:2401.15884, 2024.

• Framed around a rigorous mathematical sufficiency bottleneck (\(\min_m \max_c\)) rather than app buildingVerified
• Includes controlled modality-ablation and entity-swap evaluation splitsVerified
• Single-GPU compatible with Qwen2-VL-7BVerified
• Prototype the 4-condition modality-ablation test split on 200 technical PDF pages from ViDoRePending

1. 1
   Sample 200 technical query-document pairs from ViDoRe and create the paired Complete vs. Diagram-Dropped vs. Sibling-Entity-Swapped evaluation triples.
2. 2
   Compare holistic VLM verbalized confidence against our Conjunctive Min-Max Sufficiency score on AUROC for detecting unanswerable/conflicting contexts.

• Can we extract the visual entailment score \(v_\theta(c \models r_m)\) directly from ColPali's MaxSim late-interaction patch-token heatmaps without requiring an extra VLM forward pass?