MNIST template weights

If you're brand new to ML, some of these might be your first time seeing the word. Skim now, refer back as needed.

• Neural network: a function from input vectors to output vectors, defined by a collection of learnable numbers (the weights). The function is built by stacking small operations (linear maps, nonlinearities) on top of each other.
• Weights: the numbers inside the model. The thing training adjusts. Before training they're random; after training they're carefully tuned for the task. Everything we care about in interpretability lives in the weights.
• Logistic regression: the simplest kind of neural network. Literally one matrix multiply followed by a softmax. No hidden layer. The whole model is a single weight matrix W and a bias vector b.
• MLP (multilayer perceptron): logistic regression with one or more "hidden layers" stacked in between input and output, each followed by a nonlinearity. The first non-trivial neural network.
• Hidden layer: a layer of neurons between the input and the output. Called "hidden" because it's not directly observed during training - you only see input and output.
• Neuron: one scalar value (one number) inside a layer of the network. A layer is a vector of neurons. "100 hidden neurons" means the hidden layer is a 100-number vector. Each neuron's value is a weighted sum of the previous layer, optionally followed by a nonlinearity.
• Input pixel space: for MNIST, each image is 28×28 = 784 pixels. The input to the model is a length-784 vector.
• Class: one of the 10 possible labels (digit 0 through digit 9). The model predicts a probability distribution over classes.
• Softmax: a function that turns any vector of real numbers into a probability distribution (positive, sums to 1). Applied at the output of a classifier.
• Cross-entropy loss: the standard loss for classification. Penalises low predicted probability for the correct class. You don't need to understand the formula yet; just know it's what you optimise.
• Optimiser: the algorithm that adjusts the weights to reduce the loss. We'll use Adam, a sensible default. In code you'll see it spelled optimizer because PyTorch uses American spelling.
• Training loop: the cycle of (1) forward pass: model makes predictions, (2) compute loss, (3) compute gradients via backpropagation, (4) optimiser updates the weights. Repeat thousands of times.
• MNIST: a dataset of 70,000 28×28 greyscale images of handwritten digits, each labelled 0-9. Old, small, perfect for tutorials.
• Template matching: an algorithm that compares the input to a stored prototype and outputs how similar it is. As you'll see, that's essentially what logistic regression on MNIST does.

---

If you've never trained a model in PyTorch before, this is the place to learn the absolute basics. The notebook itself is heavily commented; this section gives you the conceptual scaffolding.

Tensors

A tensor is PyTorch's name for a multi-dimensional array (basically a NumPy array that can live on a GPU and track gradients).

x = torch.zeros(3, 4)         # 3×4 tensor of zeros
y = torch.randn(10, 784)      # 10×784 tensor of normal-distributed random numbers
z = x @ y.T                   # 3×10 - @ is matrix multiplication

Models as nn.Module

A model is a Python class that inherits from nn.Module. You declare the learnable parts in __init__ and the forward computation in forward:

class TinyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.W = nn.Parameter(torch.randn(10, 784) * 0.01)
        self.b = nn.Parameter(torch.zeros(10))

    def forward(self, x):
        return x @ self.W.T + self.b

nn.Parameter says "this is a learnable weight - track gradients for it, let the optimiser update it." That's the whole machinery.

We can also use nn.Linear(784, 10) which packages exactly this in one line.

The training loop

Every PyTorch training loop has the same skeleton:

for step in range(n_steps):
    logits = model(x)                       # 1. forward pass - predictions
    loss   = F.cross_entropy(logits, y)     # 2. compute loss
    optimizer.zero_grad()                   # 3. clear old gradients
    loss.backward()                         # 4. compute new gradients (backprop)
    optimizer.step()                        # 5. update weights

That's it. You don't need to understand backpropagation in detail - loss.backward() is one line of code that computes "if I nudged each weight, would the loss go up or down?" The optimiser uses that to take a step in the direction that decreases the loss.

Two helpers we'll use

• torch.optim.Adam(model.parameters(), lr=...) - the standard "smart" optimiser. Treat as a recipe.
• F.cross_entropy(logits, targets) - the standard classification loss. Compares predicted scores against integer class labels.

---