09. Learning Mathematics Again: Why Understanding the Fundamentals Still Matters in AI

In university courses, we often encounter linear algebra, multivariable calculus, and probability as separate, exam-driven subjects. You invert \(3 \times 3\) matrices by hand, compute partial derivatives of polynomials, and calculate coin-flip probabilities—often without seeing how they unify inside a modern learning algorithm.

Later, when you start coding in Python with scikit-learn or PyTorch, the opposite happens: the library handles all the math silently in C++ and CUDA kernels. You can train a classifier and get a respectable score without writing a single equation.

However, as soon as you step into research or advanced model debugging, that gap catches up with you:

• Why did my covariance matrix become singular when I had more features than samples?
• Why did changing the activation function from ReLU to Softplus stabilize my evidential loss gradients?
• Why does minimizing cross-entropy loss correspond to minimizing the Kullback–Leibler (KL) divergence between the empirical data distribution and the model distribution [3, 4]?

Realizing that every one of these practical engineering questions had a clean, illuminating answer in mathematics motivated me to go back and study the fundamentals properly—not to pass an exam, but to understand the language in which machine learning is written [1].

Instead of treating mathematics as abstract theory, I constantly map each concept directly to the models I work with [1, 2]:

2.1 Linear Algebra: Geometry of Representations

Linear algebra explains how high-dimensional data is represented, projected, and compressed [2].

• When I worked on correlation-based feature extraction and PCA for improving classification accuracy, the core operation was eigendecomposition of the sample covariance matrix \(\boldsymbol{\Sigma} = \frac{1}{N}\mathbf{X}^\top \mathbf{X} = \mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^\top\). Understanding that eigenvectors \(\mathbf{v}_i\) define orthogonal directions of maximum variance—and that correlated features create redundant, collinear subspaces—made dimensionality reduction geometric rather than mechanical.
• Similarly, in LoRA fine-tuning (\(\Delta \mathbf{W} = \mathbf{B}\mathbf{A}\)) and Self-Attention (\(\mathbf{Q}\mathbf{K}^\top / \sqrt{d_k}\)), everything comes down to subspace projections, matrix rank, and inner products.

2.2 Vector Calculus: The Engine of Optimization

Calculus explains how models actually learn from errors [1].

• Backpropagation is not a mysterious deep learning magic trick; it is a computationally efficient, reverse-mode application of the multivariable chain rule over a directed acyclic graph of Jacobian matrices.
• Understanding the local curvature of a loss landscape explains why poorly scaled features create elongated elliptical contours where standard gradient descent oscillates, and why adaptive preconditioning in AdamW normalizes step sizes per coordinate.

2.3 Probability & Information Theory: Reasoning Under Uncertainty

Probability gives us a rigorous framework to reason about noise, evidence, and uncertainty [3, 4].

• Maximum Likelihood Estimation (MLE) shows that minimizing Negative Log-Likelihood (cross-entropy) is equivalent to minimizing the KL divergence \(D_{\text{KL}}(P_{\text{data}} \,\|\, P_\theta)\):

• Dirichlet Distributions provide the conjugate prior over categorical probabilities on a simplex—forming the mathematical backbone of Evidential Deep Learning that I use in my adaptive inference research.

One mistake I made early on was trying to read a 500-page math textbook from cover to cover in isolation from code. Without immediate feedback, abstract definitions fade quickly.

Now, I use an interactive five-step learning cycle that pairs pen-and-paper derivation with immediate Python/PyTorch experimentation:

1. 1
   Understand the core mathematical concept and the problem it solves (for example, Singular Value Decomposition \(\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top\) [2]).
2. 2
   Work through a tiny \(2 \times 2\) or \(3 \times 3\) example by hand on paper so the indices and dimensions are unmistakable.
3. 3
   Implement it in Python / PyTorch and verify that my manual derivation matches torch.linalg.svd or autograd.grad to machine precision.
4. 4
   Perturb the inputs (what happens when two columns are nearly collinear? what happens when logits grow huge?) and visualize the result.
5. 5
   Connect the insight directly to a real ML model (such as rank-r weight compression or calibration under shift).

import torch

# Step 3 & 4 Example: Verifying Cross-Entropy == Entropy + KL Divergence in PyTorch
p_true = torch.tensor([0.70, 0.20, 0.10])   # Ground-truth soft distribution P
q_pred = torch.tensor([0.50, 0.35, 0.15])   # Model predicted distribution Q

entropy_p = -(p_true * p_true.log()).sum()
kl_div_pq = (p_true * (p_true.log() - q_pred.log())).sum()
cross_entropy_pq = -(p_true * q_pred.log()).sum()

assert torch.allclose(cross_entropy_pq, entropy_p + kl_div_pq)

Libraries evolve every year. The API that is popular today might be deprecated eighteen months from now. New neural architectures will continue to emerge.

But mathematical reasoning does not expire. Linear algebra, optimization theory, and probability remain the permanent foundation for understanding whatever comes next in artificial intelligence. Strengthening those fundamentals is one of the best long-term investments I can make as a researcher and engineer.

• [1]
  M. P. Deisenroth, A. A. Faisal, and C. S. Ong, Mathematics for Machine Learning. Cambridge, UK: Cambridge University Press, 2020.
• [2]
  G. Strang, Linear Algebra and Learning from Data. Wellesley, MA: Wellesley-Cambridge Press, 2019.
• [3]
  C. M. Bishop, Pattern Recognition and Machine Learning. New York, NY: Springer, 2006.
• [4]
  D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms. Cambridge, UK: Cambridge University Press, 2003.