Loss Functions and the Training Loop
How a network measures its mistakes and improves: mini-batches, forward pass, loss, backward pass and update — repeated thousands of times.
📄 Illustrated notes · every chapter as a picture · printable
Quick quiz
3 questions to check your understanding.
Go deeper
University-level written lectures in The AI Lecture Hall:
Transcript
Introduction. A freshly created network makes random guesses. Training turns it into something useful through one loop, repeated thousands of times.
The loss. First we need a single number that says how wrong the network is: the loss. For classification we usually use cross-entropy: minus the log of the probability given to the correct answer. If the model gives the right class 90 percent, the loss is 0.11. If it gives only 10 percent, the loss is 2.3.
The loop. Here is the loop. Take a mini batch of examples. Run a forward pass to get predictions. Compute the loss. Run a backward pass to find how each weight affects the loss. Update the weights a little. Then take the next batch. The loss on the right falls as training goes on.
Key terms. Some vocabulary. A mini batch is a small group of examples. One iteration is one update. An epoch is one full pass through all the training data. And the loss curve tracks progress.
Watching progress. We always watch two curves: loss on the training data and loss on validation data the model does not learn from. When both fall together, the network is genuinely learning, not just memorising.
Recap. To recap. The loss measures mistakes. Cross-entropy for classes, mean squared error for numbers. The loop is batch, forward, loss, backward, update. And we watch both training and validation loss.