Probability¶
100 min Foundations Needs: Plotting
Machine learning is the business of making statements you are not certain about. Probability is the language for saying how uncertain, and statistics is how you check whether the uncertainty you claimed matches reality.
Why this matters¶
A classifier does not tell you an image is a cat. It tells you it is 85 percent sure. Every number a model gives you is a claim about probability, and by the end of this programme you will be training models whose entire objective is to make those claims better.
This page is also where the loss functions of trimester two stop being magic. The thing a neural network minimises is a probability statement, and the last section of this page is that statement.
Along the way you will learn how to tell whether a pattern in your data is real or just luck, which is the question behind every hypothesis test you will run on a dataset.
The basics, quickly¶
A probability is a number between 0 and 1. Zero is impossible, one is certain, everything interesting is in between.
| Notation | Meaning |
|---|---|
| \(P(A)\) | the probability that A happens |
| \(P(A^c)\) | the probability that A does not happen, equal to \(1 - P(A)\) |
| \(P(A \cap B)\) | both A and B happen |
| \(P(A \cup B)\) | at least one of A and B happens |
| \(P(A \mid B)\) | A happens, given that B has happened |
A fair die gives \(P(6) = 1/6\) and \(P(\text{even}) = 1/2\). Nothing here is difficult. The difficulty starts when two events interact.
Disjoint is not independent¶
These two words get swapped constantly, and they mean close to opposite things.
Disjoint means the events cannot both happen. Rolling a 3 and rolling a 5 on one throw.
Independent means one event tells you nothing about the other. Two separate coin flips.
If A and B are disjoint and A happens, you now know B did not. That is a very strong piece of information, which is exactly what independence rules out.
The two rules¶
Addition, for at least one of them happening:
The subtraction is there because adding the two counts the overlap twice. When the events are disjoint the overlap is zero and the rule simplifies.
Multiplication, for both of them happening:
When the events are independent, \(P(B \mid A) = P(B)\) and this becomes a plain product.
Drawing two aces from a deck without replacement: the first is \(4/52\), and the second is \(3/51\), because the deck changed. Multiply for about 0.45 percent.
Conditional probability and Bayes¶
Conditional probability is the probability of one thing once you know another.
Bayes' theorem lets you turn a conditional around, which is the operation almost every classifier performs.
The formula is easy to apply and hard to feel. So here is the classic example counted out in people rather than symbols.
A disease affects 1 percent of the population. The test catches 95 percent of people who have it, and wrongly flags 5 percent of people who do not. You test positive. What is the chance you are ill?
Almost everyone says 95 percent. The answer is about 15.
The reason is the base rate. There are so many more healthy people that 5 percent of them outnumbers 95 percent of the sick ones, five to one. The test did nothing wrong. The population did it.
ML connection
This is class imbalance, which you met on the Working with Data page, seen from the probability side. Your fraud detector will run into exactly this: even a very accurate model, applied to a rare event, produces more false alarms than real catches. When somebody quotes you an accuracy figure, the first question is always what the base rate was.
You can check the whole thing by simulation, which is often faster than rearranging symbols.
import numpy as np
rng = np.random.default_rng(42)
n = 1_000_000
ill = rng.random(n) < 0.01
positive = np.where(ill, rng.random(n) < 0.95, rng.random(n) < 0.05)
print(ill[positive].mean())
If you can simulate it, you never have to trust your algebra alone.
Random variables and their distributions¶
A random variable is a quantity whose value depends on chance. Discrete ones take countable values, such as the number of clicks. Continuous ones take any value in a range, such as a prediction error.
A distribution describes how probability is spread across the possible values. There are three ways to write one down and they answer different questions.
The PMF gives the probability of each separate value, and only makes sense for discrete variables. The PDF describes a continuous variable, where probability lives in areas rather than points. The CDF gives the probability of being at or below a value, and it is what percentiles are built from.
The height of a PDF is not a probability
It can exceed 1. Only the area under the curve is a probability, which is why \(P(X = x) = 0\) exactly for a continuous variable. Asking for the probability that somebody is exactly 175.000 centimetres tall is asking for the area of a line.
Describing a distribution with numbers¶
Expected value is the long run average, weighting each value by how likely it is.
Variance is the average squared distance from that centre, and standard deviation is its square root, which puts it back into the units of the data.
For the scores 70, 75, 80, 85 and 90, the mean is 80, the variance is 50 and the standard deviation is about 7.07. So a typical score sits about 7 points from the average.
Squaring is what makes variance awkward to interpret and useful to optimise. It is differentiable everywhere and it punishes large errors far more than small ones.
ML connection
Mean squared error is the variance of your prediction errors. When you minimise MSE you are shrinking the spread of how wrong you are, and the squaring is why one catastrophic prediction hurts more than ten mediocre ones. That is a choice, not a law, and in trimester two you will meet losses that make a different one.
Percentiles¶
The p-th percentile is the value below which p percent of the data falls. The 25th, 50th and 75th are the quartiles, and the gap between the outer two is the interquartile range.
q1, med, q3 = np.percentile(scores, [25, 50, 75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
Those last two numbers are the whiskers of the box plot you drew last week, and anything outside them is conventionally called an outlier. Conventionally, not truly. It is a rule of thumb, not a verdict.
When the mean stops being typical¶
On a symmetric distribution the mean and median sit together. On a skewed one they separate, and the mean stops describing anybody in particular.
Report the median for skewed data, and report both when you want to show that the skew exists. A dataset where the mean is far above the median is telling you something before you have plotted anything.
The distributions worth knowing¶
Normal¶
The bell curve. Symmetric, described entirely by a mean \(\mu\) and a standard deviation \(\sigma\).
The rule worth memorising: 68, 95, 99.7. About 68 percent of values fall within one standard deviation of the mean, 95 percent within two, and 99.7 percent within three. That is how you judge in your head whether a value is unusual.
heights = rng.normal(loc=170, scale=8, size=10_000)
within_one = np.mean(np.abs(heights - 170) < 8)
print(within_one)
Bernoulli¶
One trial with two outcomes, described by a single number \(p\). A coin flip, a click, a fraud or not fraud.
This is the distribution every binary classifier outputs. When logistic regression returns 0.85, it is handing you a Bernoulli distribution.
Binomial¶
The number of successes in \(n\) independent Bernoulli trials. Mean \(np\), variance \(np(1-p)\).
Useful the moment you ask whether a result could have happened by chance. If a model gets 60 out of 100 right and guessing gives 50, the binomial tells you how often luck alone would manage 60.
Three to recognise¶
Uniform gives every value in a range the same chance, and is what you get
from rng.random(). It is used for random initialisation and for shuffling.
Poisson counts rare events in a fixed window: emails per hour, arrivals per day. One parameter, \(\lambda\), which is both the mean and the variance.
Exponential measures the waiting time between those events. If the counts are Poisson, the gaps are exponential.
You will not use these three often in this programme. Know what they model so you recognise them when they appear.
Sampling and seeing the shape¶
The fastest way to understand a distribution is to draw from it and plot it.
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
fig, axes = plt.subplots(1, 3, figsize=(14, 4))
sns.histplot(rng.normal(0, 1, 5000), bins=40, ax=axes[0])
axes[0].set_title("Normal")
sns.histplot(rng.binomial(10, 0.5, 5000), bins=11, ax=axes[1])
axes[1].set_title("Binomial, n=10, p=0.5")
sns.histplot(rng.exponential(1.0, 5000), bins=40, ax=axes[2])
axes[2].set_title("Exponential")
fig.tight_layout()
Change the parameters and run it again. Ten minutes of this teaches more than reading the formulas twice.
Why the bell curve is everywhere¶
The Central Limit Theorem says that if you take averages of samples, those averages tend towards a normal distribution as the sample grows, almost regardless of the shape you started with.
rng = np.random.default_rng(1)
population = rng.exponential(1.0, 1_000_000)
means_10 = [rng.choice(population, 10).mean() for _ in range(2000)]
means_100 = [rng.choice(population, 100).mean() for _ in range(2000)]
print(np.std(means_10), np.std(means_100))
The population is strongly skewed and the averages are not. The standard deviation of the averages falls by roughly a factor of three when the sample grows from 10 to 100, because it shrinks with the square root of \(n\).
Two consequences you will use constantly. Averages are far more trustworthy than single observations. And to halve your uncertainty you need four times the data, which is why collecting more of it has diminishing returns.
Your test score is an estimate¶
This is the practical payoff of everything above, and it is the idea that will save you from the most embarrassing mistakes in trimester two.
When you measure accuracy on a test set, you are measuring a sample. Run the same model on a different sample and you get a different number. The question is whether a difference you observe is real or is noise.
rng = np.random.default_rng(7)
true_accuracy = 0.90
for n in (100, 1000, 10_000):
trials = rng.binomial(n, true_accuracy, 2000) / n
print(n, round(trials.min(), 3), round(trials.max(), 3))
The model never changed. On 100 test rows the measured accuracy wanders between 81 and 98 percent purely by chance. So an improvement from 91 to 92 percent on a small test set is not an improvement, it is weather.
The rough rule
The uncertainty on an accuracy measured over \(n\) items is roughly \(\sqrt{p(1-p)/n}\). At 90 percent accuracy on 100 items that is about 3 percentage points, so differences smaller than about 6 points mean nothing. Compute it before you celebrate.
Is a difference real?¶
The last section asked whether a change in accuracy is real or just noise. The same question turns up all over data work. Customers who signed up on the website come back more often than customers who signed up by phone. Students who revised a new way scored higher. Is that a real pattern, or the kind of gap that luck produces all the time?
A hypothesis test answers this with one idea: assume nothing is going on, then ask how surprising your data would be.
The "nothing is going on" story has a name, the null hypothesis. For two groups it says the group makes no difference. Both groups come from the same population, and any gap you see is luck in who happened to land where.
Answer it by shuffling first¶
Here is a concrete case. One class of 12 students tried a new way of revising. Another class of 12 revised the usual way. These are their exam scores.
import numpy as np
new_way = np.array([78, 85, 69, 91, 74, 88, 82, 93, 71, 84, 79, 90])
usual = np.array([74, 82, 66, 78, 71, 86, 70, 77, 80, 64, 73, 79])
print(new_way.mean(), usual.mean())
A gap of 7 points. Now suppose the revision method made no difference at all. Then the labels "new way" and "usual" are just stickers. You could peel them off, shuffle all 24 scores, deal them back into two groups of 12, and nothing meaningful would change. So let's do exactly that, ten thousand times, and see how big the gap usually is when it is pure luck.
rng = np.random.default_rng(0)
observed = new_way.mean() - usual.mean()
pooled = np.concatenate([new_way, usual])
diffs = []
for _ in range(10_000):
rng.shuffle(pooled)
diffs.append(pooled[:12].mean() - pooled[12:].mean())
diffs = np.array(diffs)
print((np.abs(diffs) >= abs(observed)).mean())
This is a permutation test, and the picture shows what it did.
Most shuffles give a gap within about 5 points of zero. Only about 3 in 100 reach 7 points or more, in either direction. That fraction is the p-value.
The p-value is the probability of seeing a difference at least as big as yours, if nothing were going on.
A small p-value means luck rarely produces something like this, so the "nothing is going on" story becomes hard to believe. A large p-value means luck produces this all the time, so your data gives you no reason to doubt that story.
We counted gaps in both directions because we did not know in advance which class would do better, so a gap of minus 7 would have been just as surprising. This is called a two-sided test. It is SciPy's default, and the one to use unless you have a strong reason not to.
The 0.05 line¶
By convention, a result is called statistically significant when \(p < 0.05\), meaning luck alone would produce something this extreme less than 1 time in 20. There is nothing magic about 0.05. It is a habit the field settled on, and \(p = 0.049\) and \(p = 0.051\) are the same evidence. Report the number, not just which side of the line it fell on.
Two ways to be wrong¶
Whatever you conclude, you can be wrong in one of two ways.
| Nothing is really going on | Something really is | |
|---|---|---|
| You say "real difference" | False alarm | Correct |
| You say "could be luck" | Correct | Missed effect |
The 0.05 line is the false alarm rate you agree to live with: when nothing is going on, 1 test in 20 will still come out significant. Missed effects happen mostly when data is small. A real 7 point gain measured on 4 students per class can easily look like luck. These are the false positives and false negatives of the medical test again. Statisticians call them Type I and Type II errors.
Every test on this page follows the same three steps.
- Pick a number that measures the difference, such as a gap in means or a gap between counts.
- Work out how that number behaves when nothing is going on.
- See how far out your real number falls. That is the p-value.
Shuffling does step 2 by brute force. The tests below do it with a formula, which is faster and is what you will see in libraries and reports. When you are unsure about a formula, you can always check it against shuffling.
Two categories: the chi-square test¶
Often both things you care about are categories. Did the customer come back, yes or no? How did they sign up: by phone, in a store, or on the website? The question is whether the two are related.
Start by counting. pd.crosstab builds a contingency table, with one row
for each value of one column, one column for each value of the other, and a
count in every cell.
Each row is a conditional probability waiting to be read.
\(P(\text{came back} \mid \text{web}) = 72/120 = 60\) percent, against 48 percent
for phone and 37.5 percent for store. Passing normalize="index" to
pd.crosstab gives you these rates directly. They are clearly different. The
question is the same as before: is that more than luck would produce among 300
people?
What "no relationship" would look like¶
If the sign-up route and coming back were independent, the multiplication rule from the top of this page would hold: \(P(\text{web and came back}) = P(\text{web}) \cdot P(\text{came back})\). Multiply by \(n\) to turn that probability into a count, and you get the expected count for each cell.
For web and came back that is \(120 \times 150 / 300 = 60\). In plain words: half of everyone came back, so if the route made no difference, half of the 120 website customers would have come back too. We saw 72.
Measuring the gap¶
The chi-square statistic adds up how far each cell is from its expected count.
The difference is squared so that too many and too few both count. It is divided by \(E\) so that a gap of 12 matters more in a cell expecting 60 than in a cell expecting 6000. Here the total is 9.96, and almost all of it comes from the store and web rows.
Is 9.96 big? Here is a handy rule: when nothing is going on, \(\chi^2\) averages about its degrees of freedom, which for a table is
The name makes sense once you see it: with the row and column totals fixed, you can choose only 2 cells freely, and the rest are forced. So luck typically gives a value around 2, and 9.96 is far out in the tail. SciPy does the whole calculation in one call.
from scipy import stats
table = pd.crosstab(df["signup"], df["came_back"])
chi2, p, dof, expected = stats.chi2_contingency(table)
print(round(chi2, 2), round(p, 4), dof)
print(expected)
Notice that the table passed to the test has no margins=True. The totals are
not data, and counting them as cells would give a wrong answer.
A p-value of 0.007 means luck alone would produce a gap this large about 7 times in 1000. So the sign-up route and coming back look genuinely related.
What to keep in mind¶
- Feed it counts, never percentages. 60 percent of 10 people and 60 percent of 10,000 people are very different amounts of evidence, and only the counts carry that difference.
- Each individual is counted once, in exactly one cell, and individuals do not influence each other.
- Expected counts should be at least about 5 in every cell. With smaller
ones the formula stops being reliable. Merge rare categories, or for a 2 by 2
table use
stats.fisher_exact. - For a 2 by 2 table, SciPy applies the Yates correction by default. It
shrinks each \(|O - E|\) by 0.5 before squaring, which makes the test a little
more cautious with small counts. Your hand calculation will not match SciPy
unless you pass
correction=False. Both are acceptable; just know which one you used. - It tells you whether, not how much. The finding people care about is "60 percent against 37.5 percent". The p-value only tells you that gap is unlikely to be luck.
Two groups of numbers: Welch's t-test¶
When the thing you measure is a number, such as a score, a bill or a number of months, you compare means. Back to the two revision classes and their 7 point gap.
Whether 7 points is a lot depends on how noisy the scores are. A 7 point gap between groups whose scores barely vary is striking. A 7 point gap between groups whose scores swing by 30 points is nothing.
The t statistic puts signal and noise into one number.
The top is the signal, the gap between the two means. The bottom is the noise, called the standard error of the gap: how much that gap would wobble if you ran the whole study again with new students. Each group contributes its own variance \(s^2\) divided by its own size \(n\). This is the same square root of \(n\) you met in the rough rule for accuracy. More data means less wobble, so the same gap produces a bigger \(t\).
gap = new_way.mean() - usual.mean()
se = np.sqrt(new_way.var(ddof=1) / 12 + usual.var(ddof=1) / 12)
print(round(se, 2), round(gap / se, 2))
The gap is about 2.4 times the size of its own wobble. When nothing is going on, \(t\) mostly lands between about \(-2\) and \(2\), for the same reason averages form a bell curve. So 2.36 sits just outside the usual range. SciPy turns that into a p-value.
result = stats.ttest_ind(new_way, usual, equal_var=False)
print(round(result.statistic, 2), round(result.pvalue, 3))
Shuffling gave 0.032. The formula gives 0.028. Two completely different routes, nearly the same answer. That agreement is the whole point: the formula is a fast shortcut for the shuffling idea.
Why Welch¶
The original version, Student's t-test, assumes both groups have the same
spread. Welch's version does not. Each group keeps its own variance in the
standard error, exactly as in the formula above. Real groups rarely have
identical spreads, and when they do, Welch gives practically the same answer
anyway. So pass equal_var=False every time. SciPy's default is still the old
version, so remembering it is your job.
Welch also adjusts the degrees of freedom with a slightly messy formula that
SciPy handles for you (result.df is about 21.2 here). You do not need to
compute it by hand.
What to keep in mind¶
- It compares means, so outliers push it around. One customer with a bill of 10,000 can create a difference or hide one. Look at a box plot of each group before you trust the number.
- It leans on the central limit theorem. With about 30 or more in each group, the means behave like a bell curve even when the data is skewed, and the p-value is trustworthy. With small, skewed groups, also compare the medians, or run the permutation test, which makes no assumption about shape.
- Observations must be independent. If the same people are measured twice,
before and after, the groups are not separate. Use the paired version,
stats.ttest_rel. - It tells you whether, not how much. Always report the gap in means next to the p-value.
Reading a p-value without fooling yourself¶
Running a test takes one line of code. Reading the result well is the real skill, and five mistakes catch almost everyone.
1. The p-value is not the probability that nothing is going on. A p-value of 0.03 says: if nothing were going on, data like this would turn up 3 percent of the time. That is \(P(\text{data} \mid \text{null})\). People hear \(P(\text{null} \mid \text{data})\). You have seen this swap before, in the medical test, where \(P(\text{positive} \mid \text{healthy})\) was 5 percent and \(P(\text{healthy} \mid \text{positive})\) was 85 percent. It is the same trap in a new place.
2. Significant does not mean important. With enough data, even tiny differences become significant. So always report the effect size, the difference itself in units people understand: percentage points, euros, exam points. Then ask whether a difference of that size matters to anyone.
3. Not significant does not mean no effect. A large p-value means you could not tell, which is not the same as there being nothing to tell. Small samples miss real effects all the time. "No evidence of a difference" is not "evidence of no difference".
The size of the difference and the p-value answer two separate questions, and you need both.
4. Run enough tests and something will come out significant. At 0.05, 1 test in 20 is significant by luck alone. Here are 40 comparisons where both groups come from exactly the same distribution, so nothing is going on in any of them.
rng = np.random.default_rng(4)
false_alarms = 0
for _ in range(40):
a, b = rng.normal(0, 1, 50), rng.normal(0, 1, 50)
false_alarms += stats.ttest_ind(a, b, equal_var=False).pvalue < 0.05
print(false_alarms)
Two "discoveries", both pure luck. The simplest protection is the Bonferroni correction: divide 0.05 by the number of tests you ran. With 40 tests, only trust results with \(p < 0.05 / 40 = 0.00125\). It is strict, but it is honest.
5. One column at a time misses combinations. A test looks at one column against the target, alone. A column that seems useless on its own can matter in combination with another. Two columns that both look strong may be telling you the same thing twice. A model sees all the columns together; a test sees one.
ML connection
In a project you will often run a chi-square test on every categorical column and a t-test on every numeric column against the target, and then think about dropping the columns with large p-values. Treat the results as a screen, not a verdict. A small p-value says the column is related to the target, so a model may find it useful. A large p-value says you could not see a relationship in that column on its own, not that it is useless. Before you drop anything, look at the effect size, remember how many tests you ran, and remember that the model will see the columns together. When in doubt, keep the column and let the model's test score decide.
Likelihood, and where loss functions come from¶
Here is the idea that connects this page to every model you will train.
A model that outputs probabilities can be scored by asking one question: what probability did it assign to the answer that actually happened? High is good. That number is the likelihood.
Multiply those probabilities across thousands of examples and you get a number so small that a computer cannot represent it. So we take logarithms, which turn products into sums, and then flip the sign so that smaller is better.
where \(p_i\) is the probability the model gave to the correct answer for example \(i\). That is cross-entropy, and it is the loss function behind almost every classifier you will build.
The shape of the negative logarithm is the whole story. Give the right answer 90 percent and the cost is 0.105. Give it 5 percent and the cost is 3.0, thirty times more. As the probability approaches zero the cost grows without limit.
So the function does not merely punish being wrong. It punishes being confident and wrong, and it rewards honest uncertainty. A model that says 60 percent and is occasionally wrong scores better than one that says 99 percent and is occasionally wrong.
p_correct = np.array([0.9, 0.8, 0.6, 0.05])
print(-np.log(p_correct))
print(-np.log(p_correct).mean())
One bad prediction dominates the average. That single line explains most of what you will see when you start watching training curves.
Correlation, and its limits¶
Covariance measures whether two variables move together. Correlation is covariance rescaled to sit between -1 and 1, which makes it comparable across different units.
Two warnings, both of which will cost somebody marks this trimester.
Correlation only measures straight line relationships. A perfect U shape can have a correlation near zero. This is why you plot before you trust the number, which is the same lesson as Anscombe's quartet from the Plotting page.
Correlation is not causation. Ice cream sales correlate with drownings. Both are caused by summer. A model does not care, because a correlated feature predicts perfectly well until the underlying cause shifts and the relationship quietly stops holding.
Common confusions¶
Disjoint and independent
Disjoint means they cannot both happen. Independent means one tells you nothing about the other. Disjoint events with non-zero probability are always dependent.
\(P(A \mid B)\) and \(P(B \mid A)\)
These are not the same number and are often wildly different. The probability of testing positive given illness was 95 percent. The probability of illness given a positive test was 15. Swapping these two is the single most common probability error in the wild.
The PDF height looks like a probability
It is not, and it can exceed 1. Probability is the area under the curve between two values.
A big correlation means one causes the other
It means they move together. It says nothing at all about why.
My model improved from 91 to 92 percent
On how many test items? Work out the uncertainty first. On a few hundred rows that difference is indistinguishable from noise.
Dividing by n or by n minus 1
Use \(n\) when your data is the whole population, and \(n-1\) when it is a sample you are using to estimate a wider population. NumPy defaults to \(n\), and pandas defaults to \(n-1\), which is why the two sometimes disagree by a little. For large datasets the difference is negligible.
The p-value is the chance my result is a fluke
Close, but backwards. It is the chance of data at least this extreme assuming it is a fluke. Whether it actually is a fluke also depends on how plausible a real effect was to begin with, just as in the medical test.
The p-value is tiny, so the difference must be big
The p-value mixes the size of the difference with the amount of data. With a million rows, a difference nobody would notice can still give \(p < 0.001\). Look at the difference itself to judge whether it matters.
The p-value is 0.4, so the groups are the same
It means the data could not tell them apart. With small groups, real differences often give large p-values. Absence of evidence is not evidence of absence.
Check yourself¶
Use NumPy rather than algebra wherever you can. Simulating is a legitimate way to answer these.
- Simulate 100,000 rolls of two dice. Estimate the probability that the sum is 7, and compare it with the exact answer.
- Change the disease prevalence in the medical test simulation from 1 percent to 30 percent. Recompute the chance of being ill given a positive test, and explain in one sentence why it moved so much.
- Draw 10,000 values from a normal distribution of your choice and verify the 68, 95, 99.7 rule yourself.
- Take an exponential population, then plot the distribution of the mean of 5 samples and of 50 samples. Describe what changes.
- A model is 85 percent accurate. How large must your test set be before a 2 percentage point improvement is distinguishable from noise? Answer by simulation.
- Compute the cross-entropy loss for a model that gives the right answer 0.99, 0.99, 0.99 and 0.01, and for one that gives 0.7 four times. Which is the better model, and does the number agree with your instinct?
- Run the permutation test on
new_wayandusualwith 1,000, 10,000 and 100,000 shuffles and compare each p-value withstats.ttest_ind(new_way, usual, equal_var=False). Then change one score inusualfrom 80 to 40 and run both again. Which method moves more, and why? - Put the sign-up table
[[52, 48], [50, 30], [48, 72]]in a NumPy array. Compute the expected counts yourself from the row totals, column totals and \(n\) (hint:np.outer), then the chi-square statistic. Check both againststats.chi2_contingency. Repeat with a 2 by 2 table of your own and explain why your answer and SciPy's no longer agree.
Quick reference¶
| Idea | Formula or code |
|---|---|
| Complement | \(P(A^c) = 1 - P(A)\) |
| Addition | \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\) |
| Multiplication | \(P(A \cap B) = P(A) P(B \mid A)\) |
| Independence | \(P(A \cap B) = P(A) P(B)\) |
| Conditional | \(P(A \mid B) = P(A \cap B) / P(B)\) |
| Bayes | \(P(A \mid B) = P(B \mid A) P(A) / P(B)\) |
| Mean, variance, sd | a.mean(), a.var(), a.std() |
| Percentiles | np.percentile(a, [25, 50, 75]) |
| Correlation | np.corrcoef(x, y)[0, 1] |
| Seeded generator | rng = np.random.default_rng(42) |
| Normal sample | rng.normal(loc, scale, size) |
| Binomial sample | rng.binomial(n, p, size) |
| Uniform sample | rng.random(size) |
| Cross-entropy | -np.log(p_correct).mean() |
| Accuracy uncertainty | \(\sqrt{p(1-p)/n}\) |
| Contingency table | pd.crosstab(df["a"], df["b"]) |
| Rates per row | pd.crosstab(df["a"], df["b"], normalize="index") |
| Expected count | row total \(\times\) column total \(/ n\) |
| Chi-square statistic | \(\sum (O - E)^2 / E\), df \(= (r-1)(c-1)\) |
| Chi-square test | chi2, p, dof, expected = stats.chi2_contingency(table) |
| Standard error of a gap | \(\sqrt{s_1^2/n_1 + s_2^2/n_2}\) |
| Welch's t-test | stats.ttest_ind(a, b, equal_var=False) |
| Permutation p-value | (np.abs(diffs) >= abs(observed)).mean() |
| Bonferroni | trust only \(p < 0.05 / \text{number of tests}\) |
Questions to sit with¶
- The test in the Bayes example was 95 percent accurate and a positive result still meant an 85 percent chance of being healthy. If you were designing a national screening programme, what would you do with that fact?
- Cross-entropy punishes confident wrong answers far more than uncertain ones. What behaviour does that encourage in a model, and can you think of a situation where you would not want it?
- Minimising mean squared error means treating one error of size 10 as worse than ten errors of size 1. Name a problem where that is right, and one where it is badly wrong.
- To halve your uncertainty you need four times the data. What does that imply about projects that promise big gains from collecting a bit more?
- A feature correlates strongly with your target and you have no idea why. Do you use it? Does your answer change if the model decides who gets a loan?
- A chi-square test of one column against your target gives \(p = 0.3\). Do you drop the column? What would you want to know before you decide?
Next¶
That completes Maths and tools. You can represent data, shape it, see it, and now describe how certain you are about it.
The next group moves from clean examples to real data, where nothing arrives in the shape you want.
Worth your time: Seeing Theory is an interactive visual introduction to everything on this page, and 3Blue1Brown on Bayes' theorem explains the medical test example better than any text can.