Skip to content

Data Preprocessing

85 min Real data The capstone of this group

Everything in this trimester so far has been one stage of a single process. This page puts the stages in order, names them properly, and covers the two things that turn a working notebook into a pipeline somebody else can trust.

Why this matters

The next module is about models, and you will discover quickly that the model is rarely the interesting part. Choosing between a random forest and a logistic regression might move your score a couple of points. Fixing how your data was prepared moves it far more, and it is the difference between a result that holds up and one that quietly does not.

This page is also your project brief. The questions at the end are the ones you answer about your own dataset, in writing, before you build anything.

The whole pipeline

The whole data pipeline Collect, store the raw data, clean it, explore it, engineer features, split it, transform it, and store the result. Splitting comes before transforming. collect scraping, APIs, files store raw never edited again clean types, missing, duplicates explore EDA, plots, tests engineer new features, binning split before you transform transform encode, scale, impute store ready versioned, reproducible a model, next module everything in this trimester, in the order it happens The two red boxes are the ones people get wrong. Keep the raw data untouched, and split before you transform.

Two boxes are marked because they are the ones people get wrong.

Store the raw data and never edit it. Your cleaning script reads the raw file and writes a new one. When you discover in week three that a decision was wrong, you change the script and rerun it. If you edited the original, you cannot.

Split before you transform. Everything that learns something from the data, a scaler, an imputer, an encoder, must learn it from the training set alone. This has a name and it gets its own section.

The vocabulary, untangled

These words appear in job adverts as if they were interchangeable. They are not, and knowing the difference is worth something in an interview.

How the words overlap Data engineering is the widest term and covers moving and storing data. Wrangling covers cleansing and transformation. Feature engineering creates new columns. EDA runs alongside all of them rather than being a stage. data engineering getting data from where it is to where you need it, and keeping it there wrangling, or munging cleansing fix what is wrong: types, duplicates, missing, errors transformation change the shape: encode, scale, reshape, aggregate feature engineering invent columns that were not there: ratios, counts, bands EDA not a stage. It runs beside all of them, the whole time These words are used loosely and interchangeably in job adverts. The distinctions that matter: cleansing fixes what is wrong, transformation changes the form, feature engineering adds what was not there.

Data engineering is the widest. Moving data from where it is to where you need it, and keeping it there: pipelines, storage, scheduling, reliability. It is a job title as well as an activity.

Data wrangling, also called munging, is getting a messy dataset into a usable shape. It covers the next two.

Data cleansing fixes what is wrong. Wrong types, duplicates, impossible values, missing entries. You did this on the churn file.

Data transformation changes the form without changing the meaning. Encoding, scaling, reshaping, aggregating.

Feature engineering adds what was not there. A ratio, a count, a band, a flag. This is the part with the most room for skill, because it is where domain knowledge enters.

EDA is not a stage at all. It runs alongside every one of the others, whenever you want to know what you are actually holding.

One sentence that separates them

Cleansing fixes errors. Transformation changes form. Feature engineering creates information that was implicit in the data and is now explicit.

Where your data comes from

The first questions in your brief are about sources, and they matter more than they look.

One source or several? Combining sources means joining them, and every join is a chance to silently lose or duplicate rows. You saw this in the Pandas page: check the row count before and after, every time.

Do the sources agree? Two systems will spell the same city three ways, store dates in two formats, and disagree about the same customer's details. Decide which one wins, and write that decision down.

How fresh is each one? Joining a table refreshed daily to one refreshed quarterly means three months of your rows are wrong in a way nothing will warn you about.

merged = customers.merge(usage, on="customer_id", how="left", validate="one_to_one")
print(len(customers), len(merged), merged["usage_minutes"].isna().sum())

validate raises if the relationship is not what you claimed, which is far better than discovering it three steps later.

Formats and storage

Where you keep data is part of the design, not an afterthought.

Format Good for Watch out for
CSV Sharing, reading by eye, universal Loses dtypes, slow, no compression
JSON Nested structures, API responses Awkward as a table, large
Parquet Working storage for anything sizeable Not human readable
SQL Data that is queried and updated Needs a server, needs SQL 
df.to_parquet("data/processed/churn_clean.parquet")
df = pd.read_parquet("data/processed/churn_clean.parquet")

Parquet keeps your dtypes, compresses well and loads far faster. After the TotalCharges episode, a format that remembers a column was numeric is worth having.

A layout worth adopting now:

data/
  raw/           never edited, exactly as collected
  interim/       partly processed, safe to delete
  processed/     model ready, produced by a script

With data/raw/ in .gitignore if the files are large, and a note in the README saying where they came from.

Hypotheses before you look

Your brief asks whether you have preexisting hypotheses. Write them down before you start exploring, and the reason is not tidiness.

If you explore first and form hypotheses afterwards, you will find patterns, because with enough columns some will look interesting by chance. You met this on the Probability page. Twenty features tested at the usual threshold gives you about one false finding for free.

H1: month-to-month customers churn more than contract customers
H2: churn is highest in the first year and falls after that
H3: customers with more services churn less

Three sentences, written first, and then tested. Anything else you find during exploration is still worth having, but label it as exploratory rather than confirmed, because it was not a prediction.

Sparse, dense, missing and extreme

Sparse data is mostly zeros or mostly empty. One-hot encoding a column with two hundred categories produces a sparse matrix, and that is normal. Text data is extremely sparse. If your matrix is mostly zeros, some models handle it gracefully and others fall over.

Missing you have covered twice. The rule that carries forward: ask why before you ask how, and record what you chose.

Outliers are new, and the mistake is treating them as noise by default.

q1, q3 = df["MonthlyCharges"].quantile([0.25, 0.75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
outliers = df[(df["MonthlyCharges"] < low) | (df["MonthlyCharges"] > high)]
print(len(outliers))

That rule is the box plot whiskers you already know. What to do with what it finds depends entirely on which of three things you are looking at.

An error. An age of 200, a negative price. Fix it or remove it.

A rare but real value. A genuinely enormous customer. Keep it. It is the kind of case your model most needs to handle, and deleting it is choosing to be wrong about the customers who matter most.

A different population. Business accounts mixed in with domestic ones. The right answer is usually a separate model, or a column saying which is which.

Deleting outliers improves your metrics and may ruin your model

Removing the hard cases always makes the score go up. That is exactly why it is tempting, and exactly why it needs justifying in writing every time.

Splitting, properly

You know stratified splitting. There are two more cases, and using the wrong one is leakage that your test score will not reveal.

Four ways to split, and when each is right Random splitting suits independent rows. Stratified splitting preserves class balance. Time based splitting trains on the past and tests on the future. Group splitting keeps all rows from one person on the same side. random rows are independent of each other red is the minority class stratified one class is rare red is the minority class by time you will predict the future past, train future, test by group several rows share a person A B C D E Splitting randomly when time or grouping matters is leakage, and it is invisible in your score until deployment.

Random is fine when rows are independent of each other.

Stratified preserves class balance, which matters whenever one class is rare.

By time is required whenever you will predict the future. Train on the past, test on what came after. Split a time series randomly and your model learns from next month to predict this month, which it will never be able to do in production.

from sklearn.model_selection import TimeSeriesSplit

for train_idx, test_idx in TimeSeriesSplit(n_splits=5).split(X):
    pass

By group is required when several rows belong to the same entity. If one customer has twelve monthly records and six land in train and six in test, your model has effectively seen the test customers.

from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(X, y, groups=df["customer_id"]))

Ask one question of every dataset: could a row in my test set have been influenced by a row in my training set? If yes, a random split is wrong.

Leakage

This is the most important idea on the page.

Leakage is any information reaching your training process that would not be available when the model is used for real. It produces excellent scores and useless models, and it is hard to spot precisely because nothing goes wrong until it matters.

Where leakage comes from Fitting a transformer on the whole dataset lets statistics from the test set reach the training data. Fitting on the training set alone keeps the test set genuinely unseen. fit on everything: leaked the whole dataset scaler learns mean and sd train test, already seen the test set helped choose the numbers used to scale the training set split first: clean the whole dataset scaler fits on train train test, untouched the test set is transformed using numbers it had no part in choosing The score on the left is better and means less. That is what makes leakage dangerous rather than merely wrong.

Four common sources:

Fitting a transformer before splitting. The scaler learns the mean of the whole dataset, including the test set.

A feature recorded after the outcome. cancellation_reason predicts churn perfectly and is only filled in for people who already left.

A proxy for the answer. Not the label itself, but something created because of it. An internal risk flag set by the team who also decided the outcome.

Time travel. Any feature computed from data that did not exist yet at the moment you are predicting.

The test that catches most of it

For every feature, ask: would I have this value, with this content, at the moment I need to make the prediction?

If the answer is no, or if you have to think about it, it is a leak.

The other signal is a model that is too good. If your accuracy jumps to 99 percent on a hard problem, do not celebrate. Go and find out which column did it.

Making leakage structurally impossible

You can remember to fit on the training set only. You will forget once, and once is enough. The better answer is to build the preprocessing so that forgetting is not possible.

A preprocessing pipeline as one object A column transformer sends numeric columns through imputation and scaling, and categorical columns through imputation and one-hot encoding. The results join and feed the model. The whole thing is fitted once, on the training data. one Pipeline object, fitted once on the training set raw columns numeric columns impute median, then scale categorical columns impute most frequent, one-hot joined matrix model Because the whole thing is one object, fitting it on the training fold and applying it to the test fold is automatic. Cross-validation refits it on every fold. Leakage stops being something you have to remember to avoid. 

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

numeric = ["tenure", "MonthlyCharges", "TotalCharges"]
categorical = ["Contract", "InternetService", "PaymentMethod"]
numeric_steps = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
categorical_steps = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="ignore")),
])
preprocess = ColumnTransformer([
    ("num", numeric_steps, numeric),
    ("cat", categorical_steps, categorical),
])
model = Pipeline([
    ("prep", preprocess),
    ("clf", LogisticRegression(max_iter=1000)),
])
model.fit(X_train, y_train)
print(model.score(X_test, y_test))

The whole thing is one object. fit learns every statistic from the training data, and score applies those same statistics to the test data. There is no line you could forget.

It also makes cross-validation correct for free, because the pipeline is refitted inside every fold.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc")
print(scores.mean(), scores.std())

handle_unknown="ignore" matters. A category that appears only in the test set would otherwise crash the encoder, and in production it would crash your service.

ML connection

In the next module you will compare several models. Doing that fairly means every one of them sees identically prepared data, which is exactly what a pipeline guarantees. Swap the last step and everything before it stays fixed.

Bias

Your brief asks how you will ensure the dataset is unbiased. The honest answer is that you cannot, so the real question is which biases you have and whether you have looked.

Sampling bias. Who is in your data and who is not. A churn model built on customers who answered a survey describes people who answer surveys.

Historical bias. The data faithfully records a world that was unfair. Nothing is wrong with the collection and the model still learns the unfairness.

Measurement bias. Your proxy is not the thing you care about. Using support tickets as a measure of dissatisfaction measures who complains.

Aggregation bias. One model for groups that behave differently. It ends up fitting the largest group and serving the others badly.

The practical check is to stop looking at the average.

One overall score hiding four different ones A model with 88 percent accuracy overall scores 93, 90, 78 and 61 percent on four subgroups. The average conceals the group it serves worst. the headline number 88 percent accurate the same model, by group group A 93 group B 90 group C 78 group D 61 Nothing in the headline number tells you group D exists. You only find it if you go looking. 

for group, rows in df.groupby("Contract"):
    print(group, len(rows), rows["Churn"].mean())

Do this for every group you can identify, on the data before modelling and on the predictions afterwards. You are looking for groups that are small, groups whose rates are very different, and groups your model gets wrong more often.

This is the fourth time bias has appeared in this handbook. That is deliberate. It is not a topic you cover once.

Reproducibility

If you cannot rerun it and get the same answer, you do not have a result, you have an anecdote.

Seed everything. random_state=42 on every split and every model.

Script it. A notebook where cells were run out of order is not reproducible even by you. When something works, move it into a .py file that runs top to bottom.

Version the data, not just the code. At minimum, a filename with a date and a line in your README saying where it came from and when.

Write a data card. Half a page: source, collection date, row and column counts, known problems, every cleaning decision, who is represented and who is not.

That last one is the data statement from your first project, grown up. It is also the document that makes your portfolio project look professional rather than academic, because it is what a real team would require.

Your project brief

These are the questions to answer in writing, before you build. Most of them have a section above them.

Question Where to look
What are my sources, and do I need to combine them? Where your data comes from
What format is it in, and what will I store it as? Formats and storage
What features exist, and how will I explore them? EDA, and the previous page
What do I expect to find, and how will I test it? Hypotheses before you look
Is it sparse or dense? What about missing values and outliers? Sparse, dense, missing and extreme
How will I split it? Splitting, properly
How will I check for bias? Bias
Which features do I expect to use? Feature engineering, previous page
What types am I handling, and how will I transform them? Encoding and scaling, previous page
Where and how will I store the result? Formats and storage, Reproducibility

Answer them in a document, not in your head. The answers will change as you work, and the changes are worth recording too.

When it goes wrong

My cross-validation score is much better than my test score

Something leaked into the cross-validation. The usual cause is preprocessing done before the split rather than inside a pipeline.

Accuracy is 99 percent on a hard problem

Look for a leaking feature. Print the feature importances or coefficients, find the one doing all the work, and ask whether you would have it at prediction time.

ValueError: Found unknown categories at prediction time

A category appeared that was not in the training data. Use handle_unknown="ignore" in the encoder, and expect this to happen in production.

The model works in the notebook and fails on new data

Either the new data has a different shape, which a pipeline prevents, or you transformed the training data in a way you cannot reproduce, which a pipeline also prevents.

Rerunning gives a different answer

An unseeded split, an unseeded model, or a notebook run out of order. Set random_state everywhere and rerun from a clean kernel.

My joined table has more rows than either input

The key is not unique on one side. Check with duplicated().sum() before merging, and use validate= to make pandas complain.

Check yourself

Use the churn data, or your own project data if you have it.

  1. Write the three-folder data layout and move a dataset into it. Nothing should ever write to raw/.
  2. Write three hypotheses about your data before exploring it. Then test them and report which survived.
  3. Find the outliers in a numeric column with the IQR rule. Classify each one as an error, a rare real value, or a different population, and say what you did.
  4. Build a full ColumnTransformer and Pipeline for your dataset and fit it in one call.
  5. Deliberately introduce leakage by fitting a scaler before splitting. Record both scores and explain the gap.
  6. Report your target rate for every subgroup you can identify. Name the smallest group and say what that means for any model you build.
  7. Write the data card. Half a page, and no longer.

Quick reference

Task Code
Safe join a.merge(b, on="k", how="left", validate="one_to_one")
Fast typed storage df.to_parquet(path), pd.read_parquet(path)
Outlier bounds q1 - 1.5 * iqr, q3 + 1.5 * iqr
Stratified split train_test_split(X, y, stratify=y, random_state=42)
Time split TimeSeriesSplit(n_splits=5)
Group split GroupShuffleSplit(test_size=0.2)
Column routing ColumnTransformer([("num", steps, cols), ...])
Whole pipeline Pipeline([("prep", preprocess), ("clf", model)])
Honest validation cross_val_score(pipeline, X_train, y_train, cv=5)
Unseen categories OneHotEncoder(handle_unknown="ignore")
Subgroup rates df.groupby("col")["target"].mean()

Questions to sit with

  1. Leakage produces a better score and a worse model. What does that say about using the score as your only guide to whether the work is going well?
  2. You can remember to fit on the training set only, or you can use a pipeline so that forgetting is impossible. Where else in your work would that distinction be worth applying?
  3. Deleting outliers raises your accuracy. Under what circumstances is that an improvement, and how would you tell the difference?
  4. You cannot make a dataset unbiased. Given that, what is the honest thing to put in a report, and what would you refuse to claim?
  5. Somebody else has your code, your data and your seed. Can they reproduce your result exactly? If not, what is missing, and does it matter?

Next

That completes Real data, and with it the data half of the trimester. You can collect it, clean it, understand it, prepare it and store it so somebody else can rerun your work.

The next module is where a model finally sees the table you built. Almost everything that will go wrong there is something on this page.

Back to the group overview

Worth your time: the scikit-learn pipeline guide is the reference for the pipeline section, and Datasheets for Datasets is the paper that introduced the idea of the data card.