Research Outlines & Blueprints

• Research Topic: Uncertainty-Aware Spatio-Temporal Graph Attention
• Working Title: Conformal Evidential Message Passing on Dynamic Spatio-Temporal Graphs
• Research Area: Machine Learning & Geometric Deep Learning
• Subfield: Spatio-Temporal Graph Neural Networks (ST-GNNs) & Uncertainty Quantification
• Idea Status: Validating (Initial Idea → Under Exploration → Validating → Ready for Proposal)
• Created Date: 2026-10-12
• Last Updated: 2026-10-29
• Tags: ST-GNN, Graph-Attention, Conformal-Prediction, Sensor-Networks, Distribution-Shift

2.1 One-Sentence Idea

Equip spatio-temporal graph attention networks with variance-propagating message passing and adaptive conformal calibration so each sensor node outputs valid prediction intervals under topological and temporal shift.

2.2 Brief Description

Real-world sensor networks evolve continuously over space and time. While architectures such as Graph WaveNet and STGCN capture spatio-temporal correlations effectively, they output deterministic point estimates or uncalibrated Gaussian variances. We propose a single-pass attention layer that propagates both feature expectations and epistemic uncertainty scores across graph edges, followed by an online conformal quantile adjustment.

2.3 Motivation

Downstream decision systems—such as urban traffic routing and smart-grid load balancing—require worst-case bounds rather than average predictions. When a sensor fails or traffic enters an unseen congestion regime, uncalibrated GNNs propagate corrupted messages to neighboring nodes within \(L\) hops.

2.4 Intended Outcome

A modular uncertainty-aware graph attention layer and an empirical benchmark comparing coverage and interval width on METR-LA and PEMS-BAY under synthetic sensor blackouts.

3.1 Problem Statement

Given a dynamic graph \(\mathcal{G}_t = (\mathcal{V}, \mathcal{E}_t, \mathbf{X}_t)\) over \(T\) historical steps, standard ST-GNNs predict future node states \(\hat{\mathbf{Y}}_{t+1:t+H}\) without finite-sample coverage guarantees when the test distribution \(P_{\text{test}}(\mathcal{G}_t)\) shifts away from \(P_{\text{train}}(\mathcal{G}_t)\).

3.2 Current Limitations

• Bayesian GNNs & Deep Ensembles: Require \(M \times\) memory and inference passes, making real-time edge deployment infeasible.
• Homoscedastic Conformal Intervals: Standard split-conformal prediction assigns uniform interval widths across nodes, ignoring local topological degree and message uncertainty.

3.3 Research Gap (Preliminary)

Existing spatio-temporal transformers rarely couple attention-weight entropy along graph edges with node-adaptive conformal nonconformity scores to attenuate corrupted neighbor signals during message passing.

3.4 Why the Problem Matters

Reliable uncertainty bounds prevent cascading control errors in intelligent transportation systems, industrial telemetry, and environmental hazard forecasting.

4.1 Core Hypothesis

Hypothesis: Weighting neighbor aggregation by inverse edge-uncertainty and calibrating node residuals via adaptive conformal inference will maintain \(1 - \alpha \approx 90\%\) empirical coverage under \(20\%\) random node dropout while keeping inference overhead below \(12\%\).

4.2 Key Research Questions

• RQ1: Can edge-attention variance attenuate noisy messages from malfunctioning neighbor sensors?
• RQ2: How does graph-aware conformal calibration compare against MC-Dropout and 5-member Deep Ensembles in coverage vs. interval sharpness?
• RQ3: Does the calibration hold under abrupt temporal regime shifts (e.g., rush-hour anomalies)?

4.3 Proposed Approach

For node \(v\) and neighbor \(u \in \mathcal{N}(v)\), we compute an uncertainty-modulated attention coefficient \(\tilde{\alpha}_{vu}^{(t)}\) that penalizes high-variance neighbor embeddings:

4.4 Potential Novelty

• Variance-penalized graph attention that naturally down-weights corrupted sensors during message passing.
• Lightweight online conformal quantile tracking per node cluster with \(O(|\mathcal{E}|)\) complexity.

4.5 Alternative Approaches

• Variational Graph Autoencoders (VGAE) with temporal recurrent states.
• Quantile regression heads (\(\alpha/2, 1-\alpha/2\)) trained with pinball loss.

5.1 Potential Methods

• Spatio-Temporal Graph Attention (ST-GAT) backbone with dual mean-variance projection heads.
• Adaptive Conformal Inference (ACI) with online step-size \(\eta = 0.005\).

5.2 Potential Datasets

• METR-LA: 207 sensors, 5-minute intervals, Los Angeles highway network.
• PEMS-BAY: 325 sensors, Bay Area traffic speed benchmark.

5.3 Evaluation Metrics

• Forecasting Accuracy: MAE, RMSE, MAPE.
• Uncertainty Quality: Empirical Coverage (\(90\%\) target), Mean Prediction Interval Width (MPIW), Expected Calibration Error (ECE).

5.4 Baselines

• Deterministic STGCN and Graph WaveNet.
• ST-GAT + MC-Dropout (\(10\) forward passes) and Deep Ensemble (\(5\) models).

5.5 Required Resources

• Compute / Hardware: Single GPU workstation (1x RTX 4090 24GB).
• Software / Libraries: Python 3.11, PyTorch, PyTorch Geometric.
• Data: Public open-access benchmarks (~1.2 GB total).
• Estimated Budget: \(\\)0$ (uses existing local compute).
• Other Requirements: Reproducible seed-locked evaluation pipeline.

• Potential Research Contribution: Empirical characterization of how topological message passing amplifies or dampens epistemic uncertainty under sensor dropout.
• Potential Technical Contribution: Open-source PyTorch Geometric module (UncertainSTGATConv) with built-in conformal calibration.
• Potential Practical Impact: Safer anomaly-aware forecasting for urban traffic and telemetry networks.
• Potential Research Output: Workshop paper and full M.Sc. thesis proposal.

7.1 Scope

Focus strictly on closed-topology sensor graphs (fixed nodes, dynamic signals and missing-value masks) over 15-, 30-, and 60-minute horizons.

7.2 Main Challenges

• Preventing variance collapse during joint training of mean and uncertainty heads.
• Maintaining sharp (narrow) intervals during normal traffic regimes.

7.3 Risks

• Conformal intervals may become overly wide immediately after an abrupt regime shift before online adaptation catches up.

7.4 Fallback Plan

If joint variance training destabilizes gradients, freeze the pre-trained ST-GAT backbone and train a post-hoc graph residual estimator on the validation split.

8.1 Key Papers to Read

• [1]
  Z. Wu et al., “Graph WaveNet for Deep Spatial-Temporal Graph Modeling,” IJCAI, 2019 — Establishes self-adaptive adjacency matrices for traffic forecasting.
• [2]
  I. Gibbs and E. Candès, “Adaptive Conformal Inference Under Distribution Shift,” NeurIPS, 2021 — Provides online quantile update guarantees under non-exchangeable time series.
• [3]
  M. Sensoy et al., “Evidential Deep Learning to Quantify Classification Uncertainty,” NeurIPS, 2018 — Inspires single-pass higher-order uncertainty parameterization.

8.2 Related Concepts and Keywords

Non-exchangeable conformal prediction, Homophily under corruption, Dirichlet-Normal-Inverse-Gamma, Message-passing variance.

8.3 Preliminary Observations

Preliminary toy runs on METR-LA show that masking 15% of nodes degrades neighbor RMSE by 28% in standard GAT, whereas variance-damped attention limits degradation to 11%.

• The problem is clearly statedVerified
• The motivation is clearVerified
• Existing limitations are identifiedVerified
• The idea can be expressed in one sentenceVerified
• Research questions are testableVerified
• Potential datasets existVerified
• Evaluation metrics are definedVerified
• Required compute and resources are realisticVerified
• The scope is suitable for the available timeVerified
• Literature review has started (in progress — 12 of 25 target papers reviewed)Pending

Overall Decision: Develop into Proposal (Keep Exploring / Validate Further / Develop into Proposal / Park Idea / Drop Idea)

1. 1
   Complete 5-seed baseline runs for Graph WaveNet and vanilla ST-GAT on METR-LA with 10%, 20%, and 30% sensor dropout masks.
2. 2
   Implement the variance-penalized attention coefficient \(\tilde{\alpha}_{vu}^{(t)}\) and log empirical coverage vs. MPIW.
3. 3
   Compare online Adaptive Conformal Inference (ACI) against static split-conformal calibration during simulated rush-hour spikes.
4. 4
   Draft full 3,000-word Research Proposal specification from validated findings.

12.1 Unresolved Questions

• Should the penalty coefficient \(\gamma\) in \(\tilde{\alpha}_{vu}^{(t)}\) be learned per attention head or tuned via validation risk–coverage curves?
• How does multi-hop message passing affect conformal exchangeability assumptions across spatial neighborhoods?

12.2 Supervisor / Peer Feedback

12.3 Revision History

• Note Title: Convolutional Neural Networks & Receptive Field Scaling
• Note ID: DL-104
• Related Project / Topic: Evidential Uncertainty-Guided Adaptive Inference (#PROP-2026-01)
• Research Area: Deep Learning & Computer Vision
• Note Type: Technical Concept & Paper Reading (Paper Reading / Idea / Experiment / Dataset / Code / Meeting / Observation)
• Created Date: 2026-10-18
• Last Updated: 2026-10-29
• Status: Synthesized (Draft / Reviewed / Synthesized / Archived)
• Tags: CNN, Receptive-Field, Residual-Connections, CBAM, Lightweight-Backbone

2.1 Source Details

• Title: Deep Residual Learning for Image Recognition; CBAM: Convolutional Block Attention Module
• Authors: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (2016); Sanghyun Woo et al. (2018)
• Year: 2016 / 2018
• Venue / Source: IEEE CVPR / ECCV
• DOI / URL: https://arxiv.org/abs/1512.03385

2.2 Why I Am Recording This

Selecting and adapting the lightweight model \(f_L\) for CIFAR-10-C (\(32 \times 32\)) requires precise control of spatial downsampling and receptive field growth so early corruptions do not destroy high-frequency class evidence.

3.1 Main Takeaway

Replacing the standard ImageNet \(7 \times 7\) stride-2 stem with a \(3 \times 3\) stride-1 stem preserves spatial resolution on small \(32 \times 32\) inputs, while residual skip connections prevent gradient degradation and lightweight channel-spatial gates improve calibration under noise.

3.2 Brief Summary

This note synthesizes the mathematical formulation of 2D discrete cross-correlation, recurrence equations for effective receptive fields across strided/dilated layers, and a minimal PyTorch implementation of a Residual-CBAM block for the lightweight classifier \(f_L\).

4.1 Main Points

• Parameter sharing reduces dense layer complexity from \(O(H^2 W^2 C_{\text{in}} C_{\text{out}})\) to \(O(K^2 C_{\text{in}} C_{\text{out}})\).
• Stacking three \(3 \times 3\) conv layers achieves the same \(7 \times 7\) receptive field as a single \(7 \times 7\) kernel with \(45\%\) fewer parameters and three non-linearities.

4.2 Important Concepts or Definitions

• Translation Equivariance: Shifting the input image shifts the output feature map identically: \(f(T_s(I)) = T_s(f(I))\).
• Effective Receptive Field (ERF): The region of input pixels that exercises a non-negligible gradient influence on a center unit in layer \(l\).

4.3 Methods, Models, or Technical Details

Given a 2D input feature map \(I\) and kernel \(K\) of size \(M \times N\), the discrete cross-correlation feature response \(S(i, j)\) is:

4.4 Experimental Setup or Evidence

Minimal PyTorch implementation of the residual basic block used in our CIFAR-10 lightweight model \(f_L\):

import torch
import torch.nn as nn
import torch.nn.functional as F

class ResidualBlock(nn.Module):
    def __init__(self, in_channels: int, out_channels: int, stride: int = 1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)

        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(out_channels),
            )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        out = F.relu(self.bn1(self.conv1(x)), inplace=True)
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)
        return F.relu(out, inplace=True)

5.1 Strengths

• Extremely fast inference latency (\(<1.2\text{ ms}\) per batch on GPU) and deterministic FLOP profile.
• Strong local inductive bias requires less training data than unpretrained Vision Transformers.

5.2 Weaknesses or Limitations

• Fixed local kernels struggle with global geometric distortions unless receptive fields are sufficiently deep.
• BatchNorm statistics drift under high-severity impulse/Gaussian noise (motivates test-time BN calibration or GroupNorm).

5.3 Assumptions

• Assumes local stationarity of low-level visual primitives (edges, corners, color gradients).

5.4 What Is Missing or Unclear

• Need empirical comparison between BatchNorm2d and GroupNorm when computing Dirichlet evidence parameters on CIFAR-10-C severity 5.

6.1 How This Connects to My Topic

Forms Component A (Lightweight Visual Model \(f_L\)) in the Evidential Adaptive Inference proposal.

6.2 Useful Ideas to Adopt

• Use CIFAR-adapted stem (conv1: 3x3, stride=1, no maxpool) for ResNet-18 so \(32 \times 32\) feature maps are not prematurely crushed to \(8 \times 8\).

6.3 Differences From My Approach

• Standard ResNet uses a linear softmax head; we replace the final classifier with a nonnegative Softplus Evidential Dirichlet head \(\boldsymbol{\alpha} = \mathbf{e} + \mathbf{1}\).

6.4 Potential Baseline or Comparison Point

• Serves as the primary fixed lightweight baseline (\(C_L = 1.0\times\) relative compute unit).

7.1 Personal Commentary

High-frequency corruptions (Gaussian noise, shot noise) primarily perturb the first two convolutional stages; adding a lightweight channel gate helps suppress noisy channels before global average pooling.

7.2 New Ideas Triggered

• Extract intermediate feature norms from Stage 2 and Stage 3 to augment the evidential vacuity score \(u = K / S\).

7.3 Questions Raised

• Does replacing ReLU with Softplus in the penultimate block stabilize Dirichlet evidence gradients during early epochs?

• Verify main claim or methodVerified
• Check cited papersVerified
• Compare with existing baselineVerified
• Test idea on small dataset (CIFAR-10 clean split: \(94.8\%\) accuracy)Verified
• Add to literature reviewVerified
• Discuss with supervisor / peerVerified
• Update research idea or proposal (add GroupNorm vs. BatchNorm ablation note)Pending

"It is easier to optimize the residual mapping than to optimize the original, unreferenced mapping." — He et al. (2016), CVPR

Recurrence relation for layer-\(l\) receptive field \(R_l\) with kernel size \(k_l\), dilation \(d_l\), and cumulative stride \(s_{<l} = \prod_{i=1}^{l-1} s_i\):

• Related Notes: [[Note-DL-101]] · [[Note-CV-208]] · [[Note-OPT-302]]
• Related Papers: He et al. (2016) [1] · Sensoy et al. (2018) [3] · Hendrycks & Dietterich (2019) [5]
• Related Experiments: DEMO-EXP-001 (CIFAR-10-C Compact ResNet / ResNet-50 routing)
• Related Datasets / Code: torchvision.datasets.CIFAR10, models/evidential_resnet.py