Skip to content

Data Preparation and Visualization

85 min Real data Needs: Pandas, Plotting, Probability

You have a file of 7043 telephone customers and a column saying which of them left. This page is what you do between opening that file and handing a table to a model.

Why this matters

You already know the commands. dropna, astype, groupby, histplot. What you have not done is decide which ones to run, in what order, and why, on a dataset nobody has cleaned for you.

That decision sequence is the job. Two analysts given the same file will produce different tables, and the difference is judgment rather than syntax. This page is about the judgment, and the churn data is here to make it concrete.

Exploratory data analysis

EDA is a name for the loop you run when you meet a dataset.

The exploratory loop Look at the data, ask what could be wrong with it, check that suspicion, then fix it or write it down. Each answer produces the next question. look head, info, describe ask what would break this? check plot it, count it, test it fix or record and write down why every answer produces the next question EDA is not a phase you complete before modelling. It is what you keep doing whenever a result surprises you.

It is not a checklist you complete. Every answer raises the next question: the target is imbalanced, so which metric will you use; total charges is text, so why; eleven values are missing, so what do those eleven rows have in common.

Start with the three commands you already know.

import pandas as pd

df = pd.read_csv("Telco-Customer-Churn.csv")
print(df.shape)
df.info()
df.describe()
df.describe(include="object")

describe on its own only reports numeric columns, which on this file is three of twenty-one. Pass include="object" and you get counts, unique values and the most frequent value for the rest. Two commands, and you already know the shape of the problem.

Missing data, and what it means

Finding missing values is a line of code. Deciding what they mean is the work, and this dataset has a perfect example.

print(df.isna().sum().sum())

0

Zero missing values. Except TotalCharges has dtype object, which for a column of money is suspicious. Eleven of its entries are a single space character, which pandas read as text rather than as missing.

df["TotalCharges"] = pd.to_numeric(df["TotalCharges"], errors="coerce")
print(df["TotalCharges"].isna().sum())

11

Now look at what those eleven rows have in common.

Missing values that are not missing at random Every row with a blank total charge has a tenure of zero. These are brand new customers who have not been billed yet, so the blank means zero rather than unknown. a few rows from the churn file customerID tenure MonthlyCharges TotalCharges 7590-VHVEG 1 29.85 29.85 5575-GNVDE 34 56.95 1889.5 4472-LVYGI 0 52.55 blank 3115-CZMFE 0 20.25 blank 9237-HQITU 2 70.70 151.65 5709-LVOEQ 0 80.85 blank every blank has tenure 0 All 11 blanks belong to customers who joined this month and have not been billed yet. The value is not unknown. It is zero. Filling it with the median would invent a billing history.

Every one of them has tenure equal to zero. These are customers who signed up this month and have not been billed yet. The value is not unknown, it is zero.

Always ask why before you ask how

Fill those eleven with the column median, around 1400 euro, and you have invented two years of billing history for customers who joined last week. Every model you build afterwards learns from that invention.

The three options are drop, replace and impute, and they are not interchangeable:

Drop the rows when they are few and you cannot justify a value. Eleven out of 7043 is nothing, so this is defensible here.

Replace with a value you can justify from the meaning of the data. Filling these with 0 is correct, because zero is what a customer with no billing cycle has been charged.

Impute with a statistic, or with a model, when the value genuinely exists but was not recorded. Median for skewed columns, mean for symmetric ones, and a flag column either way.

Here, replacing with 0 is right and imputing is wrong, and no amount of looking at the isna count would have told you that.

Types

TotalCharges was text because of eleven spaces. That is the common case: one bad value drags a whole column to object and every numeric operation you try afterwards either fails or silently does something strange.

SeniorCitizen has the opposite problem. It is stored as 0 and 1, so pandas reports it as an integer and describe cheerfully tells you the mean senior citizen is 0.16. It is a category wearing a number's clothes.

df["SeniorCitizen"] = df["SeniorCitizen"].astype("category")

The test is not what the values look like. It is whether arithmetic on the column means anything.

Duplicates and columns to drop

print(df.duplicated().sum())
print(df["customerID"].duplicated().sum())

Two different questions. The first asks whether an entire row repeats. The second asks whether the identifier repeats, which would mean the same customer appears twice with different details, and that is a data collection problem rather than a copy and paste one.

Then drop what cannot help.

Identifiers. customerID is unique for every row. A model given it can memorise the training set perfectly and learn nothing that transfers. Drop it.

Constant columns. If every row has the same value, the column carries no information.

Columns that leak the answer. Anything recorded at the same time as the outcome, or after it. There is no such column in this file, but in a real churn dataset a field like cancellation_reason would be one, and it would give you a model with 99 percent accuracy and no use whatsoever.

Look at the target first

Before any feature, plot the thing you are predicting.

import seaborn as sns

print(df["Churn"].value_counts(normalize=True))
sns.countplot(data=df, x="Churn")

No     0.735
Yes    0.265

About one customer in four left. That single number changes several later decisions:

  • A model that predicts "No" for everybody scores 73.5 percent accuracy. That is your baseline, and anything you build must beat it before it is interesting.
  • Accuracy is the wrong headline metric here, for the reason you saw on the Probability page.
  • Your splits need to be stratified, which is the last section of this page.

ML connection

You met this as class imbalance in Working with Data and as base rates in Probability. This is the same idea for the third time, now with a number attached to a real dataset. If a later page quotes you an accuracy for this problem, 73.5 percent is the number to compare it against.

Distributions

Categorical features first, because two thirds of this file is categorical.

categorical = df.select_dtypes(include=["object", "category"]).columns
fig, axes = plt.subplots(4, 4, figsize=(18, 14))
for ax, col in zip(axes.flatten(), categorical):
    sns.countplot(data=df, x=col, ax=ax)
    ax.tick_params(axis="x", rotation=30)
fig.tight_layout()

What you are looking for: categories with almost no members, which will be unreliable no matter what the model does, and values that should be the same category but are spelled differently.

This dataset has a nice example. Several service columns contain No internet service as well as No. Those are two different statements and it is your choice whether they stay separate.

Then the continuous ones. There are only three: tenure, MonthlyCharges, TotalCharges.

for ax, col in zip(axes, ["tenure", "MonthlyCharges", "TotalCharges"]):
    sns.histplot(data=df, x=col, bins=30, ax=ax)

tenure is U-shaped: many customers are brand new, many have been there for years, fewer in between. That shape is a finding, not noise, and it is the argument for the binning section later.

Relationships with the target

A distribution tells you what a column looks like. What you need is whether it relates to churn.

sns.heatmap(df.corr(numeric_only=True), annot=True, cmap="coolwarm", center=0)

With three numeric columns this is a small heatmap, and it says one useful thing: tenure and TotalCharges are strongly correlated, because total charges is roughly tenure times monthly charges. Two columns carrying the same information is worth knowing before you feed both to a model.

For categorical features against churn, proportions beat counts.

pd.crosstab(df["Contract"], df["Churn"], normalize="index")

Churn                 No       Yes
Contract
Month-to-month     0.573     0.427
One year           0.887     0.113
Two year           0.972     0.028

normalize="index" is what makes this readable. Raw counts would mostly tell you that month-to-month is the largest group. Percentages tell you that its churn rate is fifteen times the two year rate, which is the actual finding.

For numeric features, compare the distributions across the two groups.

sns.boxplot(data=df, x="Churn", y="tenure")

Churners have visibly shorter tenure. You can see it. The next section is about whether you can defend it.

Is the pattern real?

Your eyes will find patterns in noise. A statistical test asks whether the pattern you are looking at could plausibly have arisen by chance.

Choosing a test for whether a feature relates to churn For a categorical feature against a categorical target use a chi-square test of independence. For a numeric feature compared across two groups use Welch t-test. does this feature relate to churn? the feature is categorical the feature is numeric Chi-square build a contingency table of counts, compare it with what independence would predict Welch t-test compare the mean for churners with the mean for the rest, without assuming equal spread Both answer the same question: could this pattern plausibly have appeared by chance alone? Neither tells you the effect is large enough to care about. With 7043 rows, tiny differences come back significant.

Chi-square, for two categorical variables

Build a table of counts and compare it with the counts you would expect if the two variables were unrelated.

from scipy.stats import chi2_contingency

table = pd.crosstab(df["Contract"], df["Churn"])
chi2, p, dof, expected = chi2_contingency(table)
print(chi2, p)

A small p-value means the observed counts would be unlikely if contract type and churn were independent.

Welch's t-test, for a numeric variable across two groups

from scipy.stats import ttest_ind

churned = df.loc[df["Churn"] == "Yes", "tenure"]
stayed = df.loc[df["Churn"] == "No", "tenure"]
stat, p = ttest_ind(churned, stayed, equal_var=False)
print(stat, p)

equal_var=False is what makes it Welch's version rather than the standard one. The standard test assumes both groups have the same spread, which here they do not: there are far more stayers than churners and their tenure is spread differently. Welch's does not assume it, costs nothing, and is the safer default in almost every real comparison.

Significant does not mean important

With 7043 rows almost everything comes back significant, because a large sample can detect a tiny difference reliably. A p-value answers "is this difference real". It does not answer "is this difference big enough to care about".

Always look at the effect alongside the p-value. A two month difference in average tenure with a p-value of 0.001 is real and probably useful. A difference of 0.3 months with the same p-value is real and useless.

Making better features

Two moves, and both are about helping a model express something it otherwise could not.

Binning

Binning a continuous feature Tenure in months is cut into four bands, new, growing, established and loyal, so the model can express a jump between groups rather than a straight line. new 0 to 12 months growing 13 to 24 months established 25 to 48 months loyal 49 plus months tenure in months Binning throws away detail on purpose. Do it when the effect is a jump between groups, not a steady slope. 

df["tenure_band"] = pd.cut(
    df["tenure"],
    bins=[-1, 12, 24, 48, 100],
    labels=["new", "growing", "established", "loyal"],
)

Binning throws away information deliberately. That is worth it when the effect is a step rather than a slope: the difference between month 2 and month 14 is enormous here, while the difference between month 50 and month 62 is small. A linear model cannot say that about a raw tenure column, and it can about bands.

Use pd.cut for boundaries you choose and pd.qcut for equal sized groups. Choosing your own is usually better, because the boundaries then mean something to the business.

New columns

df["avg_monthly"] = df["TotalCharges"] / df["tenure"].replace(0, 1)
df["services"] = (df[service_cols] == "Yes").sum(axis=1)

The second one is the interesting kind. Counting how many services a customer has turns sixteen separate columns into one number with an obvious meaning, and a model can use it directly.

Every feature you create should have a sentence attached explaining what it means. If you cannot write the sentence, you are generating columns rather than engineering features.

Encoding

Models take numbers. Most of this dataset is text.

Three ways to turn a category into numbers Label encoding assigns 0, 1 and 2, which invents an order. Ordinal encoding does the same but is correct when the categories really are ordered. One-hot encoding creates one column per category and invents nothing. InternetService DSL Fiber None label 0 1 2 says None is twice Fiber wrong here one-hot DSL Fiber None 1 0 0 0 1 0 0 0 1 no order implied, one column each correct here Label encoding is only safe when the categories genuinely have an order, such as low, medium, high. Used on unordered categories it tells a linear model that the gap from DSL to None is twice the gap from DSL to Fiber. Tree models survive it. Linear models and anything distance based do not. 

df = pd.get_dummies(df, columns=["InternetService", "Contract", "PaymentMethod"], drop_first=True)
df["Churn"] = (df["Churn"] == "Yes").astype(int)

For binary columns like Partner, a simple 0 and 1 is fine and there is no ordering to invent. For unordered categories with more than two values, one-hot is the default. For genuinely ordered categories, such as your tenure_band, ordinal encoding preserves information that one-hot throws away.

drop_first=True removes one column per feature. The information is still there, because "not DSL and not Fiber" means None, and it avoids a redundancy that upsets linear models.

Scaling

Two features before and after scaling Before scaling, total charges run to thousands while tenure runs to seventy, so any distance calculation is dominated by total charges. After scaling both features sit on the same range. before after standardising tenure, 0 to 72 8000 tenure, about -2 to 2 2 Distance based models treat one unit of tenure as one unit of euros. Scaling makes that comparison meaningful. Tree models do not care at all, because they only ever ask whether a value is above a threshold.

tenure runs to 72 and TotalCharges runs past 8000. Any model that measures distance, or that uses gradient descent, treats those numbers as if they were on the same scale. They are not.

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Standardisation subtracts the mean and divides by the standard deviation, giving a column centred on zero. Min-max squeezes into a 0 to 1 range, which preserves shape but is more affected by outliers.

Which models care:

Model Needs scaling?
Linear and logistic regression Yes, especially with regularisation
Anything distance based, such as k-nearest neighbours Yes, critically
Neural networks Yes, training is much slower without it
Decision trees and random forests No, thresholds are unaffected

Fit on the training data only

Look at the two lines above. fit_transform on the training set, transform on the test set. The scaler learns the mean and standard deviation from training data alone.

Call fit on the whole dataset and the test set's statistics have leaked into your training. Your score comes out better than it should, and the model performs worse than promised on anything new. This is the single most common way a beginner produces a result that is not real, and the next page is largely about it.

Splitting

A stratified split keeps the class balance The full dataset is about 73 percent stayed and 27 percent churned. A stratified split keeps that ratio in the training, validation and test sets. A plain random split can distort it, especially in the smallest set. the whole dataset, 7043 customers stayed, 73 percent churned, 27 percent stratified: every part keeps 73 to 27 train, 60% validation, 20% test, 20% unstratified: the smallest set drifts 25 percent churned 29 percent churned 36 percent churned If your test set has a different balance from reality, its score answers a question nobody asked. 

from sklearn.model_selection import train_test_split

X = df.drop(columns=["Churn"])
y = df["Churn"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
print(y_train.mean(), y_test.mean())

stratify=y keeps the 73 to 27 balance in both halves. Without it the split is random, and with a minority class the smaller set can drift several percentage points, which makes your test score measure a slightly different problem from the one you trained on.

random_state makes the split reproducible, so that a change in your score comes from a change you made rather than from a different shuffle.

Split before you scale, before you impute, and before anything else that learns from the data.

The whole thing in one screen

df = pd.read_csv("Telco-Customer-Churn.csv")
df["TotalCharges"] = pd.to_numeric(df["TotalCharges"], errors="coerce").fillna(0)
df["SeniorCitizen"] = df["SeniorCitizen"].astype("category")
df = df.drop_duplicates().drop(columns=["customerID"])
df["tenure_band"] = pd.cut(df["tenure"], [-1, 12, 24, 48, 100], labels=["new", "growing", "established", "loyal"])
df["Churn"] = (df["Churn"] == "Yes").astype(int)
X = pd.get_dummies(df.drop(columns=["Churn"]), drop_first=True)
y = df["Churn"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)

Eleven lines. Every one of them is a decision somebody could disagree with, which is why the last step of preparation is writing them down.

Keep a decisions file

One short note per choice: what you did, and why. Filled TotalCharges blanks with 0 because all eleven have tenure 0. Dropped customerID as an identifier. Binned tenure at 12, 24 and 48 months because those are contract boundaries.

In three months you will not remember, and neither will the person reviewing your work. This is the same habit as the data statement from your first project.

When it goes wrong

describe() shows only three columns

It reports numeric columns by default. Use df.describe(include="object") or include="all" for the rest.

A money column has dtype object

Something in it is not a number. pd.to_numeric(col, errors="coerce") then isna() shows you exactly which rows, and those rows usually explain themselves.

isna().sum() says zero but values are clearly missing

Empty strings, single spaces and the words unknown or n/a are not missing values as far as pandas is concerned. Pass na_values to read_csv or convert afterwards.

My test accuracy is suspiciously high

Check for an identifier column, for a feature recorded after the outcome, and for a scaler fitted before the split. All three produce excellent scores that mean nothing.

Every feature is statistically significant

With 7043 rows that is expected. Look at the size of the difference, not only the p-value.

get_dummies produced hundreds of columns

One of your categorical columns has very many distinct values, often an identifier that should have been dropped. Check df[col].nunique() before encoding.

Check yourself

Use the churn file for all of these.

  1. Report the shape, the dtype of every column, and which columns are categorical wearing numeric clothes.
  2. Convert TotalCharges properly, show that exactly eleven values become missing, and demonstrate what those rows have in common.
  3. Justify in one sentence why 0 is the right fill for those rows and the median is not.
  4. Produce the churn rate by Contract as proportions, and state the baseline accuracy any model must beat.
  5. Run a chi-square test for two categorical features and a Welch's t-test for MonthlyCharges. Report both the p-value and the size of the difference.
  6. Bin tenure into four bands of your own choosing and defend the boundaries.
  7. Encode, scale and split, in the correct order, and prove the class balance is preserved in both halves.
  8. Write your decisions file. Aim for one line per decision.

Quick reference

Task Code
Full summary df.describe(include="all")
Force numeric pd.to_numeric(col, errors="coerce")
Category from number df["SeniorCitizen"].astype("category")
Missing map sns.heatmap(df.isna(), cbar=False)
Duplicate rows, duplicate ids df.duplicated().sum(), df["id"].duplicated().sum()
Target balance df["Churn"].value_counts(normalize=True)
Rates by category pd.crosstab(df["a"], df["b"], normalize="index")
Chi-square chi2_contingency(pd.crosstab(a, b))
Welch t-test ttest_ind(g1, g2, equal_var=False)
Bin by boundaries pd.cut(col, bins=[...], labels=[...])
Bin by quantiles pd.qcut(col, q=4)
One-hot pd.get_dummies(df, columns=[...], drop_first=True)
Scale StandardScaler().fit(X_train) then .transform(X_test)
Stratified split train_test_split(X, y, stratify=y, random_state=42)

Questions to sit with

  1. Eleven rows had a blank total charge and you decided it meant zero. What evidence supported that, and what would have changed your mind?
  2. A model that predicts nobody churns is 73.5 percent accurate. Is that model useless, and does your answer depend on what the company does with the prediction?
  3. Binning tenure throws away information on purpose. What exactly did you give up, and what did you buy with it?
  4. Every choice on this page could have gone differently. If two analysts produce different churn rates from the same file, is one of them wrong?
  5. Scaling must be fitted on training data only. Why is fitting on everything so tempting, and what does that tell you about how leakage usually happens?

Next

You have a clean, encoded, scaled, split dataset. The next page is about the ways that process quietly goes wrong, and how to build it so it cannot.

Data Preprocessing

Worth your time: the scikit-learn preprocessing guide is the reference for the last three sections, and the Telco churn dataset page has community notebooks worth reading after you have formed your own view.