Skip to content

What training actually does

30 min Concepts Read alongside the maths module

"Training adjusts the model until it fits the examples" is true and it explains nothing. This page replaces it with a picture you can hold in your head. No equations, just the shape of the idea.

A model is a pile of numbers

Whatever else you hear, a trained model is a large collection of numbers called weights, plus the arithmetic that combines them. Training means finding good values for those numbers. That is the whole of it.

Start with one unit, historically called a neuron.

A single unit Three inputs each multiplied by a weight, added together, passed through a decision step, producing one output. Training means changing the weights. x1 x2 x3 inputs w1 w2 w3 add it all up, then squash the result one number out training changes w1, w2 and w3, nothing else

Each input is multiplied by its weight and the results are added. A large weight means that input matters a lot. A weight near zero means it is being ignored. Then the sum is squashed by a simple function, which is what lets the network represent curves rather than only straight lines.

One unit is not much. Stack hundreds of them into a layer, then stack layers on top of each other, and the early layers end up detecting simple things like edges while later layers combine those into shapes and eventually into "this is a hand showing scissors". Nobody programmed that progression. It falls out of training.

That stacking is what the word deep means in deep learning. Not deep as in profound. Deep as in many layers.

The model needs to know how wrong it is

Before anything can improve, you need a single number that says how badly the model is doing. That number is the loss.

Show the model an image of paper. It outputs confidence for each class, say 20 percent rock, 50 percent paper, 30 percent scissors. The right answer was paper, so it was partly right. The loss function turns that gap into one number. Confident and correct gives a small loss. Confident and wrong gives a large one.

This matters more than it looks. Choosing the loss function is choosing what the model tries to be good at. It optimises exactly what you measure, and nothing else. Every story about a system that technically achieved its target while doing something nobody wanted is a story about a loss function.

Rolling downhill

Now you have a number to make smaller. Imagine plotting the loss against one weight. The picture is a valley, and training means walking to the bottom.

Gradient descent A curve shaped like a valley. Error is on the vertical axis and the value of a weight on the horizontal axis. The model starts high on one side and takes repeated small steps downhill towards the lowest point. the value of one weight how wrong the model is start here, badly wrong as good as it gets each dot is one small step in the direction that reduces error

At each step the model works out which way is downhill for every weight at once, and nudges them all a little in that direction. Repeat a few thousand times. That is gradient descent, and it is what is happening whenever you see a progress bar during training.

Two details you will meet constantly:

Learning rate is the size of the step. Too small and training takes forever. Too large and you leap across the valley and end up higher than you started. Most training problems that look mysterious are a learning rate problem.

Epochs and batches. An epoch is one pass through all your training examples. Because looking at every example before each step is slow, the data is split into batches and a step is taken after each batch. More epochs means more passes, which is not automatically better, as the next section explains.

Real models have millions of weights, so the valley has millions of dimensions and cannot be drawn. The picture still works. The maths is the same shape.

Knowing when to stop

Here is the most useful graph in machine learning.

Training error against validation error Two curves over training time. Error on the training data keeps falling. Error on held back data falls at first, then starts rising. The point where it turns is where the model stops learning the pattern and starts memorising. training time error stop here on data it trained on on data it has never seen both improving: it is learning one improving, one worse: it is memorising

The grey line always falls. Give a model enough time and it can memorise its training data completely, which is why the grey line on its own tells you nothing.

The red line is the one that matters. It is the error on the validation set, the held back data you met earlier in this module. While both lines fall, the model is learning something general. When the red line turns upward, the model has started memorising particulars that do not transfer. Everything after that point makes it worse at the only job it has.

Stopping at the turn is called early stopping, and it is the most common cure for overfitting. This is also the answer to why you hold data back at all. Without the red line you cannot see the turn, and you will train confidently past it.

This is the same thing you saw in the first project. Your model was excellent on your own photographs and much worse on another group's. That gap is the two lines separating, measured with hands instead of a graph.

Why Teachable Machine trained in thirty seconds

Training an image model from nothing takes many hours and a lot of data. Your project took under a minute on a laptop. Something else is going on.

Teachable Machine starts from a network already trained on millions of photographs. That network has learned generally useful things: edges, textures, curves, the parts that make up objects. It knows nothing about your gestures, but it already knows how to look at images.

Training then only adjusts the last small part, the part that maps those general features onto your three classes. Almost all the weights stay as they were. This is called transfer learning, and it is why a group of students can build a working classifier in an afternoon.

It also explains a failure you may have seen. The features came from photographs of everyday scenes, mostly taken in good light by people in certain parts of the world. When your test images look unlike that, the borrowed features fit less well, and accuracy drops for reasons that have nothing to do with your fifty photographs.

Questions to sit with

  1. A model optimises exactly the number you give it. Name a task where the thing you can measure is not quite the thing you want, and say what the model would learn to do instead.
  2. Training error keeps falling while validation error rises. Your manager sees only the first graph and is delighted. What do you say?
  3. If almost all the weights in your project's model came from somebody else's training on somebody else's photographs, whose model is it, and who is responsible for what it inherited?
  4. A model with millions of weights can memorise its training set completely. Why does adding more training data make that harder, and is more data always the right answer?
  5. Nobody programmed the early layers to detect edges. It emerged from minimising a number. Does that make the model's behaviour explainable, and to whom?

Next

That completes the foundations. Everything from here is these ideas applied with more rigour, starting with the maths that makes the downhill step possible.