Name: TensorTonic
Author: TensorTonic

Last updated 13 October 2026

What is a covariance matrix?

A covariance matrix is a square, symmetric matrix that summarises how a set of variables vary. Entry $(i, i)$ on the diagonal is the variance of variable i, and entry $(i, j)$ is the covariance between variables i and j: positive if they rise together, negative if one rises as the other falls. Its eigenvectors give the directions PCA keeps.

Multidimensional Variance

In 1D, variance tells you how spread out data is along a single axis. But real data lives in many dimensions. Light bulb quality might be characterized by filament thickness, gas pressure, and glass clarity simultaneously.

In 2D (or n-D), data can be spread out in different directions. The Covariance Matrix captures this shape. It tells us not just how much each variable varies, but how they vary together.

The Covariance Matrix ( $\Sigma$ )

For $n$ variables, it's an $n \times n$ symmetric matrix.

\Sigma = \begin{bmatrix} Var(X_1) & Cov(X_1,X_2) & \cdots & Cov(X_1,X_n) \\ Cov(X_2,X_1) & Var(X_2) & \cdots & Cov(X_2,X_n) \\ \vdots & \vdots & \ddots & \vdots \\ Cov(X_n,X_1) & Cov(X_n,X_2) & \cdots & Var(X_n) \end{bmatrix}

Diagonal Elements

Variances of each variable. How spread out each variable is on its own.

Off-Diagonal Elements

Covariances between pairs of variables. How they move together.

For a data matrix $X$ with $m$ rows (samples), first subtract each column's mean to get $X_c$ . The sample covariance matrix is then $\Sigma = \frac{1}{m-1} X_c^T X_c$ . Dividing by $m - 1$ instead of $m$ corrects for having estimated the mean from the same data.

Reading the matrix: a worked example

Six bulbs are measured on three properties: filament resistance, luminosity and heat output. The code centers the data, builds the covariance matrix, converts it to correlations and runs PCA, which is everything this lesson covers in about fifteen lines:

Python

import numpy as np

# Bulb measurements: resistance (ohm), luminosity (lumens / 100), heat (C / 10)
X = np.array([
    [10.2, 8.1, 6.0],
    [11.5, 9.4, 6.8],
    [ 9.8, 7.6, 5.9],
    [12.1, 9.9, 7.1],
    [10.9, 8.8, 6.1],
    [11.0, 8.5, 6.6],
])

Xc = X - X.mean(axis=0)              # 1. center each column
S = Xc.T @ Xc / (len(X) - 1)         # 2. sample covariance, same as np.cov(X, rowvar=False)
print("covariance:
", S.round(3))

sd = np.sqrt(np.diag(S))             # 3. covariance -> correlation
R = S / np.outer(sd, sd)
print("correlation:
", R.round(3))

eigvals, eigvecs = np.linalg.eigh(S) # 4. PCA: eigenvectors of the covariance
order = eigvals.argsort()[::-1]
eigvals, eigvecs = eigvals[order], eigvecs[:, order]
print("variance explained:", (eigvals / eigvals.sum()).round(3))

Output

covariance:
 [[0.702 0.694 0.386]
 [0.694 0.71  0.368]
 [0.386 0.368 0.238]]
correlation:
 [[1.    0.983 0.944]
 [0.983 1.    0.895]
 [0.944 0.895 1.   ]]
variance explained: [0.974 0.023 0.003]

Diagonal: resistance and luminosity vary about equally (0.702 and 0.710), while heat varies much less (0.238) in these units.
Signs: every off-diagonal entry is positive, so all three properties rise together. A negative entry would mean one tends to fall as the other rises.
Symmetry: entry (1, 2) equals entry (2, 1), because $Cov(X, Y) = Cov(Y, X)$ .
Magnitudes: the raw size of a covariance is hard to read on its own, which the next section explains.

Covariance vs correlation

Correlation is covariance divided by the two standard deviations, which removes the units:

\rho_{ij} = \frac{\Sigma_{ij}}{\sigma_i \sigma_j}

	Covariance matrix	Correlation matrix
Diagonal	Variances, in squared units	Always 1
Off-diagonal range	Any real number	Between -1 and 1
Changes with units	Yes	No
Use it for	PCA on same-unit data, Gaussians, Mahalanobis distance	Comparing strength of relationships, PCA on mixed units

In the bulb example, the resistance and heat covariance (0.386) is about half the resistance and luminosity covariance (0.694), which suggests a weaker link. The correlations, 0.944 and 0.983, show both links are strong. The covariance was smaller only because heat has a smaller variance in the units used. Running PCA on the covariance matrix of mixed-unit data lets the largest-variance feature dominate, so standardize the features first, which amounts to running PCA on the correlation matrix.

Interactive Simulator

Manipulate the covariance matrix for 3 variables (X, Y, Z). See how changing the covariances changes the 2D projections of the data cloud.

Covariance Playground

Covariance Matrix Σ

1.0

0.80

0.50

0.80

1.0

0.20

0.50

0.20

1.0

Invalid Matrix: Impossible correlations. Reduce values.

How to use:Adjust sliders in the upper triangle. The matrix is symmetric, so changes reflect in the lower triangle automatically. Trying to set strong pairwise correlations (e.g. X-Y=0.9, Y-Z=0.9, X-Z=-0.9) will break geometry and show an error.

X vs Yr=0.80

X vs Zr=0.50

Y vs Zr=0.20

Key Properties

1. Symmetry

$Cov(X,Y) = Cov(Y,X)$ . The matrix is symmetric, so $\Sigma = \Sigma^T$ .

2. Positive Semi-Definite (PSD)

$x^T \Sigma x \geq 0$ for any vector $x$ . Why? Because variance cannot be negative. The quadratic form represents a variance.

3. Linear Transformation

If $Y = AX$ , then $\Sigma_Y = A \Sigma_X A^T$ . This is how covariance propagates through linear operations.

4. Correlation Matrix

Normalize by dividing each entry by $\sigma_i \sigma_j$ . Correlation matrix has 1s on diagonal, values between $-1$ and $1$ off-diagonal.

Geometric Intuition

Think of the Covariance Matrix as defining an ellipsoid (a stretched sphere) in n-dimensional space. The equation $(x - \mu)^T \Sigma^{-1} (x - \mu) = c$ defines an elliptical contour of constant probability.

Spherical (Identity Covariance)

Variables are uncorrelated with equal variance. Data cloud is a perfect sphere.

Ellipsoidal (General Covariance)

Variables have different variances and correlations. Data cloud is a tilted ellipsoid.

Eigen-Decomposition (Spectral Theorem)

This is the bridge between Linear Algebra and Statistics. Since $\Sigma$ is symmetric PSD, it has a special decomposition:

\Sigma = V \Lambda V^T

Eigenvectors ( $V$ ):

The columns of $V$ point in the directions of the principal axes of the ellipsoid. These are the "natural" coordinate axes of the data, where variables are uncorrelated.

Eigenvalues ( $\Lambda$ ):

The diagonal matrix where $\lambda_i$ is the variance along the i-th principal axis. Larger eigenvalue = more spread in that direction.

Why This Matters

By rotating data into the eigenvector basis, you decorrelate the variables. The covariance matrix in the new basis is diagonal. This is exactly what PCA does.

Case Study: Bulb Quality Control

The Scenario

You measure 3 properties of each bulb: Filament Resistance (Ω), Luminosity (Lumens), and Heat Output (°C). You have data from 1,000 bulbs. How do you detect anomalies?

The Covariance Matrix

Σ = [[2.5, 1.8, 0.9], [1.8, 3.2, 1.4], [0.9, 1.4, 1.8]]

The off-diagonal values (1.8, 0.9, 1.4) are positive, meaning all three properties are positively correlated. A brighter bulb is also hotter.

Mahalanobis Distance (Anomaly Detection)

D_M = \sqrt{(x - \mu)^T \Sigma^{-1} (x - \mu)}

Unlike Euclidean distance (which ignores correlations), Mahalanobis distance accounts for the covariance structure. A bulb that's hot but not bright is an outlier, even if each value alone seems normal.

Its role in PCA

PCA is literally just finding the eigenvectors of the Covariance Matrix and projecting data onto the top ones.

The Algorithm

Center the data (subtract mean from each column).
Compute Covariance Matrix $\Sigma = \frac{1}{n-1} X^T X$ .
Find Eigenvectors and Eigenvalues of $\Sigma$ .
Sort by Eigenvalues (Largest = Most Variance = PC1).
Project data: $Z = XV$ , where $V$ is the matrix of top $k$ eigenvectors.

Variance Explained

The proportion of variance explained by PC $i$ is $\lambda_i / \sum \lambda_j$ . In the bulb example the first component explains 97.4% of the variance, because all three properties move together, so a single “overall output” score keeps almost all of the information. If the first 2 PCs explain 95% of variance in 100-dimensional data, you can visualize it in 2D with little loss.

Whitening / Sphering

After PCA, scale each PC by $1/\sqrt{\lambda_i}$ . This transforms the ellipsoid back into a sphere (uncorrelated, unit variance). Common preprocessing for neural networks.

Practice this

On TensorTonic you implement this from scratch and test it in the browser. These problems follow the same path as the lesson, from computing the matrix to using it.

Compute Covariance Matrix: center the data and build $\Sigma$ without np.cov.
Compute Pearson Correlation Matrix: normalize covariances by the standard deviations.
PCA from Scratch: eigendecompose the covariance matrix and project onto the top components.
Mahalanobis Distance: measure distance with the inverse covariance, as in the case study.
Whitening Transform: turn the data ellipsoid into a sphere with unit variance.

Contents

Covariance Matrices

What is a covariance matrix?

Multidimensional Variance

The Covariance Matrix ( $\Sigma$ )

Diagonal Elements

Off-Diagonal Elements

Reading the matrix: a worked example

Covariance vs correlation

Interactive Simulator

Covariance Playground

Covariance Matrix Σ

Key Properties

1. Symmetry

2. Positive Semi-Definite (PSD)

3. Linear Transformation

4. Correlation Matrix

Geometric Intuition

Spherical (Identity Covariance)

Ellipsoidal (General Covariance)

Eigen-Decomposition (Spectral Theorem)

Why This Matters

Case Study: Bulb Quality Control

The Scenario

The Covariance Matrix

Mahalanobis Distance (Anomaly Detection)

Its role in PCA

The Algorithm

Variance Explained

Whitening / Sphering

Practice this

Contents

What is a covariance matrix?

Multidimensional Variance

The Covariance Matrix (Σ\SigmaΣ)

Diagonal Elements

Off-Diagonal Elements

Reading the matrix: a worked example

Covariance vs correlation

Interactive Simulator

Covariance Playground

Covariance Matrix Σ

Key Properties

1. Symmetry

2. Positive Semi-Definite (PSD)

3. Linear Transformation

4. Correlation Matrix

Geometric Intuition

Spherical (Identity Covariance)

Ellipsoidal (General Covariance)

Eigen-Decomposition (Spectral Theorem)

Why This Matters

Case Study: Bulb Quality Control

The Scenario

The Covariance Matrix

Mahalanobis Distance (Anomaly Detection)

Its role in PCA

The Algorithm

Variance Explained

Whitening / Sphering

Practice this

The Covariance Matrix ( $\Sigma$ )