Modules
01/11
Linear Algebra

Contents

Dot Product & Vector Norms

The atomic operations of Neural Networks and Similarity Search.

Last updated 19 October 2026

What is the dot product?

The dot product of two vectors multiplies matching entries and adds them up: a⋅b=∑iaibi\mathbf{a} \cdot \mathbf{b} = \sum_i a_i b_i. Geometrically it equals the product of their lengths times the cosine of the angle between them, so it measures how much two vectors point the same way. A norm measures a single vector's length.

At the heart are two fundamental operations: Dot Product (alignment) and Norms (magnitude).

Every neural network prediction, recommender system suggestion, and regularization penalty uses these operations. Together, they form the foundation of how machines understand similarity, distance, and direction in high-dimensional space.

Why These Matter

Understanding dot products and norms is crucial for modern ML architectures. Matrix multiplication is just batched dot products. Transformer attention computes similarity via scaled dot products. Regularization techniques like Ridge and Lasso use different norms to constrain model complexity.

The Dot Product

The dot product (also called inner product or scalar product) takes two vectors of equal length and returns a single scalar. It is the most fundamental operation in linear algebra.

a⋅b=∑i=1naibi\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i

Multiply corresponding elements, then sum.

Algorithm

a = [1, 3]
b = [4, -2]
a · b = (1)(4) + (3)(-2) = -2

Matrix Notation

aTb\mathbf{a}^T \mathbf{b}

Row vector × Column vector. This generalizes to specific rows and columns in larger matrices.

Geometric Interpretation

The dot product measures alignment between vectors. It bridges algebra and geometry.

a⋅b=∣∣a∣∣⋅∣∣b∣∣cos⁡(θ)\mathbf{a} \cdot \mathbf{b} = ||\mathbf{a}|| \cdot ||\mathbf{b}|| \cos(\theta)

Product of magnitudes times the cosine of the angle between them.

Positive

Vectors point in roughly the same direction (θ<90°\theta < 90°).

Zero

Vectors are orthogonal (perpendicular) (θ=90°\theta = 90°).

Negative

Vectors point in opposite directions (θ>90°\theta > 90°).

The Projection View

Think of it as casting a shadow of vector a onto b. The dot product tells you: "How much of a goes in the direction of b?". This "projection" concept is why dot products measure similarity.

Properties of the Dot Product

Commutative

a⋅b=b⋅a\mathbf{a} \cdot \mathbf{b} = \mathbf{b} \cdot \mathbf{a}

Order does not matter.

Distributive

a⋅(b+c)=a⋅b+a⋅c\mathbf{a} \cdot (\mathbf{b} + \mathbf{c}) = \mathbf{a} \cdot \mathbf{b} + \mathbf{a} \cdot \mathbf{c}

Distributes over vector addition.

Self Dot Product

a⋅a=∣∣a∣∣2\mathbf{a} \cdot \mathbf{a} = ||\mathbf{a}||^2

Dotting a vector with itself gives the squared norm (magnitude).

Interactive: Dot Product

Drag the sliders to change vectors. See how the dot product relates to angle, cosine, and projection.

Dot Product
ab45°6.4a · bFUSION

Vector norms: L1 vs L2 vs infinity

A norm measures the "size" or "magnitude" of a vector. Different norms measure size in different ways, each with unique geometric interpretations and ML applications.

L2 Norm (Euclidean)

∣∣x∣∣2=∑xi2||\mathbf{x}||_2 = \sqrt{\sum x_i^2}

Straight-line distance. Used in Ridge Regression, KNN.

L1 Norm (Manhattan)

∣∣x∣∣1=∑∣xi∣||\mathbf{x}||_1 = \sum |x_i|

Sum of absolute values (taxi-cab distance). Promotes sparsity (Lasso).

L-infinity Norm (Max)

∣∣x∣∣∞=max⁡∣xi∣||\mathbf{x}||_\infty = \max |x_i|

Largest single element. Used in adversarial robustness.

L1L2Infinity
Formula∑∣xi∣\sum |x_i|∑xi2\sqrt{\sum x_i^2}max⁡∣xi∣\max |x_i|
For (3,−4,1)(3, -4, 1)8about 5.104
Unit ball in 2DDiamondCircleSquare
As a penaltyLasso: pushes weights to exactly zeroRidge: shrinks all weights smoothlyCaps the largest weight

Properties of Norms

For a function to be a valid norm, it must satisfy three axioms:

  • Non-negativity: Norm is always non-negative. Zero only if the vector is zero.
  • Absolute Homogeneity: Scaling the vector scales the norm by the absolute scalar value.
  • Triangle Inequality: The direct path is always shorter than any detour. ∣∣x+y∣∣≤∣∣x∣∣+∣∣y∣∣||\mathbf{x} + \mathbf{y}|| \leq ||\mathbf{x}|| + ||\mathbf{y}||

The p-Norm Family

All common norms are special cases of the generalized p-norm (where p≥1p \geq 1).

∣∣x∣∣p=(∑i=1n∣xi∣p)1/p||\mathbf{x}||_p = \left(\sum_{i=1}^{n} |x_i|^p\right)^{1/p}

p=1p = 1

L1 Norm (Manhattan)

p=2p = 2

L2 Norm (Euclidean)

p=3,4,…p = 3, 4, \ldots

Higher-order norms

p→∞p \to \infty

L-infinity (Max)

Interactive: Unit Balls

The "unit ball" (vectors with norm = 1) has a different shape for each p-norm. This shape explains why L1 regularization promotes sparsity (sharp corners on axes).

Vector Norms & Unit Balls

Visualizing ∥x∥p=1\|x\|_p = 1 for different p-norms.

Vector Components
L2 Norm (Euclidean)
Points with constant sum of squares form a circle. Measures straight-line distance.
Unit Ball (Norm=1)
L1
7.00
L2
5.00
L∞
4.00

Normalization (Unit Vectors)

Normalization converts a vector to a unit vector (norm = 1) pointing in the same direction, isolating direction from magnitude.

x^=x∣∣x∣∣\hat{\mathbf{x}} = \frac{\mathbf{x}}{||\mathbf{x}||}

Divide by the norm.

Applications

  • Cosine Similarity: Normalized vectors reduce dot product to cosine similarity.
  • Batch Norm: Normalizes layer activations to stabilize training.
  • Word Embeddings: Often normalized so only direction (meaning) matters, not frequency.

Cosine similarity

Dividing the dot product by both lengths leaves only the angle:

cos⁡(θ)=a⋅b∣∣a∣∣2 ∣∣b∣∣2\cos(\theta) = \frac{\mathbf{a} \cdot \mathbf{b}}{||\mathbf{a}||_2 \, ||\mathbf{b}||_2}

The result runs from -1 (opposite directions) through 0 (perpendicular) to 1 (same direction), whatever the vectors' lengths. That is why embedding search and recommendation systems compare vectors with cosine similarity: a long document and a short one about the same topic still score as similar. If the vectors are already L2-normalized, cosine similarity is just their dot product.

Python implementation

NumPy computes all of these in one line each. np.linalg.norm takes the norm order as its second argument and defaults to L2:

Python
import numpy as np

a = np.array([3.0, 4.0])
b = np.array([4.0, 3.0])
x = np.array([3.0, -4.0, 1.0])

print(f"dot product       = {a @ b:.3f}")
print(f"cosine similarity = {a @ b / (np.linalg.norm(a) * np.linalg.norm(b)):.3f}")
print(f"L1 norm   = {np.linalg.norm(x, 1):.3f}")
print(f"L2 norm   = {np.linalg.norm(x):.3f}")
print(f"Linf norm = {np.linalg.norm(x, np.inf):.3f}")
print(f"unit vector L2 norm = {np.linalg.norm(x / np.linalg.norm(x)):.3f}")
Output
dot product       = 24.000
cosine similarity = 0.960
L1 norm   = 8.000
L2 norm   = 5.099
Linf norm = 4.000
unit vector L2 norm = 1.000

The two example vectors are 5 units long and almost aligned, so their cosine similarity is 0.96. The vector (3,−4,1)(3, -4, 1) gives the three different norms from the table, and dividing it by its L2 norm leaves a unit vector.

ML Applications

Regularization

Penalizing the norm of weights prevents overfitting.

  • L1 (Lasso): λ∣∣w∣∣1\lambda ||\mathbf{w}||_1 - creates sparse models (feature selection).
  • L2 (Ridge): λ∣∣w∣∣22\lambda ||\mathbf{w}||_2^2 - distributes weight values, prevents unstable large weights.

Attention Mechanism

Transformer attention is a scaled dot product. It calculates relevance scores between tokens.

Distance Metrics

KNN and K-Means rely on L2 distance ∣∣x−y∣∣2||\mathbf{x} - \mathbf{y}||_2. Using different norms here changes the algorithm's behavior drastically.

Computational Considerations

Time Complexity

Dot Product (n-dim)O(n)O(n)
Norm CalculationO(n)O(n)

Hardware

GPUs optimize these O(n) ops via massive parallelism (SIMD). Always use vectorized libraries (PyTorch/NumPy), never loops.

Practice this

On TensorTonic you implement this from scratch and test it in the browser.