Skip to content

Framing a machine learning problem

25 min Concepts Read before session two

Nobody will hand you a well posed machine learning problem. They will say "can we use AI to reduce customer complaints". Turning that into something you can build is the first skill of the job, and this page is the method.

Why this matters

Most failed machine learning projects were not failures of modelling. They were projects where nobody agreed on what was being predicted, or on how anyone would know it had worked, until months of effort had already gone in.

Ten minutes of framing at the start prevents that. It is also the part of the job that never gets automated, because it is the part that requires knowing what people actually want.

The formal version

Tom Mitchell gave machine learning its standard definition in 1997, and it holds up because it is really a checklist. A system learns if its performance improves with experience, where three things are named:
T Task What it is supposed to do
E Experience What it learns from
P Performance How improvement is measured

If you cannot fill in all three, you do not yet have a machine learning problem. You have a wish.

Five questions

Mitchell's three, plus two the textbook leaves out and the real world does not.

Five questions for framing a problem One, what exactly is the task. Two, what experience will it learn from. Three, how will success be measured. Four, which learning family is this. Five, should this exist at all. 1 What exactly is the task? For one example: what goes in, and what comes out 2 What will it learn from? Which data, from whom, how much, and who labels it 3 How will you know it works? One measure, agreed before you start, not after 4 Which learning family is this? It decides what data you need and what can go wrong 5 Should this exist at all? Who benefits, who is harmed, and who gets to decide questions one to three are Mitchell's. Four is engineering. Five is yours.

Question one is harder than it looks. "Reduce customer complaints" is not a task. "Given a support message, predict whether it will become a formal complaint" is. The test is whether you can describe a single example: one input, one output.

Which family

Choosing a learning family If you have labelled examples, supervised learning, which splits into regression when the answer is a number and classification when it is a category. If you have examples with no labels, unsupervised learning, which covers clustering and dimensionality reduction. If you have an environment to act in, reinforcement learning, which produces a policy. What do you have? Supervised examples with the answers Unsupervised examples, no answers Reinforcement an environment to act in Regression the answer is a number Clustering find the natural groups A policy what to do in each situation Classification the answer is a category Dimensionality reduction squeeze detail into a simpler view binary, multi-class, multi-label your first project is multi-class classification: one hand, one of three labels

Supervised learning

You have examples and you have the right answers. The model learns the mapping from one to the other. This is most of what you will build this year, and it comes in four flavours that differ only in the shape of the answer.

Flavour One example gets Example
Binary classification One of two labels Spam or not spam
Multi-class classification One of many labels Rock, paper, or scissors
Multi-label classification Any number of labels at once A photo tagged beach, dog, friend
Regression A number A flat in Tirana costs 92,000 euro

The distinction between multi-class and multi-label catches people out. Multi class means exactly one answer per example. Multi label means an example can carry several at once, and they are not competing.

Naming the flavour tells you what to collect

Multi-label needs every applicable tag on every training example, not just the most obvious one. Regression needs examples spread across the whole range of values, not clustered at the average. The flavour is not bookkeeping, it changes the data you have to go and get.

Unsupervised learning

You have examples and no answers. The model finds structure by itself.

Clustering groups similar examples. Customers who behave alike, documents about the same topic, the transaction that looks unlike all the others. The algorithm produces the groups. Naming them is your job, and that naming is a judgment, not an output.

Dimensionality reduction compresses many measurements into a few that keep most of the information. It is how you visualise data with fifty columns, and how you strip out the parts that carry no signal.

There is a trap here worth knowing now. Unsupervised results always look like something. Give any algorithm data and it will return groups. Whether those groups mean anything is a separate question, and one the algorithm cannot answer for you.

Reinforcement learning

An agent acts in an environment, receives a reward or a penalty, and adjusts. Over many attempts it learns a policy, which is a rule for what to do in each situation rather than a single answer.

Four words cover it: agent, environment, action, reward. This is how systems learn to play Go without being shown human games, and how robots learn to walk.

Being honest about scope: you will not train a reinforcement learning agent in this programme. You need to recognise it, know when a problem is one, and know that the hard part is the reward. Define it badly and the agent will maximise exactly what you asked for rather than what you wanted, which is known as reward hacking and is a live research problem.

Two worked examples

A well posed one.

Problem as stated: stop spam reaching the inbox.

Task: given one email, predict spam or not spam. Experience: past emails that users marked as spam or left alone. Performance: of the emails we flag, how many really were spam. Losing a real email costs far more than letting one spam through. Family: supervised, binary classification. Should it exist: yes. Low stakes, visible to the user, reversible.

One where the answer is no.

Problem as stated: predict criminal behaviour from facial photographs.

Task: given a face, predict a label. Experience: photographs, labelled using past convictions. Performance: accuracy, presumably. Family: supervised, binary classification. Should it exist: no.

The first four rows fill in perfectly. That is the point. A problem can be technically well posed and still be one you refuse.

Look at the experience row. Convictions are not criminality, they are the record of who was arrested, charged and convicted, which reflects who was policed. The model would learn the pattern of past enforcement and present it as a property of a face. And there is no reason to think the target is visible in a face at all, which means any accuracy you achieve is measuring something other than what you claimed.

Question five is not a formality tacked on the end. Sometimes it is the only one that matters.

Try it yourself

Frame each of these with the five questions before the next session. Two of them are traps.

  1. Suggest which film a user should watch next.
  2. Predict how many minutes late the city bus will arrive.
  3. Sort a photo library into people, places, food and documents.
  4. Decide which job applicants get a first interview.

For each, say what one example looks like going in and coming out, where the data would come from, what you would measure, which family and flavour it is, and whether you would build it.

Questions to sit with

  1. Question three says agree the measure before you start. Why does the order matter, given that the measure is the same either way?
  2. A clustering algorithm always returns groups, even from random data. How would you tell a real pattern from an invented one?
  3. In the facial criminality example, the experience is the problem. Name another task where the data you can get is a poor stand-in for the thing you actually care about.
  4. If reward hacking means the agent maximises what you asked for rather than what you wanted, is that a failure of the agent or of the person who wrote the reward?

Next

You have a problem. Now you need the data, which is where most of the work and most of the mistakes live.

Working with data