In my previous post (Intro to ML #4), I wrote about thresholds, the recall-precision tradeoff, and what it looks like when a model actually changes business outcomes.
Posts #1 through #4 all revolved around the same type of problem: predicting a category. Will this loan default? Did this passenger survive? Yes or no. This is called classification.
This time: the other major type of supervised machine learning — regression, where you predict a number instead of a category.
- 1 Classification vs. Regression: The Shape of the Answer
- 2 The Auto Insurance Problem
- 3 How Regression Models Are Evaluated: RMSE and R²
- 4 Turning Predictions Into Business Actions
- 5 AutoML: When the Model Selection Happens Automatically
- 6 What I Learned: Start with the Shape of the Answer
- 7 Books to Go Deeper
Classification vs. Regression: The Shape of the Answer
The distinction between classification and regression comes down to what you’re trying to predict:
- Classification: The answer is a category. “Will this borrower default?” → yes or no. Evaluated with AUC, recall, precision.
- Regression: The answer is a continuous number. “How much will this insurance claim cost?” → $4,200, $11,800, etc. Evaluated with RMSE and R².
Which one to use isn’t a choice you make based on preference — it’s determined by the shape of the answer the business problem requires. That framing (“what shape is the answer?”) was a simple but useful reframe.
The Auto Insurance Problem
The exercise used data from an auto insurance company. Here’s the situation:
- Around 50,000 policyholders
- Premium income: flat $5,000 per customer
- Average claim payout: approximately $5,000 per customer, totaling nearly $250 million
- Business model built on being the cheapest option — but recent high-value claims are killing the margin
In short: revenue roughly equals cost. There’s no room for profit. If you could predict how much each customer was likely to claim, you could actually do something about it.
How Regression Models Are Evaluated: RMSE and R²
Just as classification models have AUC, regression models have their own evaluation metrics.
- RMSE (Root Mean Square Error): Measures the average size of the prediction error. Lower is better. If your model predicts insurance claims with an RMSE of $1,200, it means predictions are off by roughly $1,200 on average.
- R² (Coefficient of Determination): Measures how well the model explains the variance in the data. Ranges from 0 to 1. An R² of 0.7 means the model explains 70% of the variability in claim amounts. Higher is better.
Where AUC tells you whether the model ranks things correctly, RMSE tells you how far off the actual dollar amounts are. Both are measuring prediction quality — just from different angles.
Turning Predictions Into Business Actions
Once you have a model that predicts claim amounts, the interesting question becomes: what do you actually do with it?
Several ideas came up in the exercise:
- AI-based underwriting screening: Reject applicants whose predicted claim amount significantly exceeds the premium. Don’t take on customers who will cost more than they pay.
- Dynamic pricing: Set premiums individually based on predicted risk. High predicted claims → higher premium. Low predicted claims → lower premium.
- Targeted marketing: Focus acquisition efforts on demographic segments or geographies with consistently low predicted claim amounts.
The same model, applied in three different directions — reducing costs, increasing revenue, and optimizing customer acquisition. What’s interesting is that each approach has different tradeoffs.
Dynamic pricing, for example, means customers who get a lower rate will be happy. But customers who are quoted a higher rate may walk away — which means you lose revenue while keeping only your riskiest existing customers. The ability to predict something doesn’t automatically tell you the right thing to do with the prediction. That’s still a business judgment.
AutoML: When the Model Selection Happens Automatically
During this session, I got a brief introduction to AutoML (Automated Machine Learning).
Normally, choosing which algorithm to use (random forest, XGBoost, linear regression, etc.) requires judgment and experimentation. AutoML tools like PyCaret automate this: they try multiple algorithms on your dataset simultaneously and rank them by performance.
If you just need the best-performing model and aren’t constrained by interpretability requirements, AutoML dramatically lowers the barrier. The democratization of machine learning is often discussed in abstract terms, but watching this run in real time made it feel concrete.
What I Learned: Start with the Shape of the Answer
Classification and regression are both forms of prediction. The difference is whether the answer is a category or a number.
Framing the business problem correctly — before choosing any algorithm — comes down to asking: “What exactly am I trying to predict? And what form does that answer take?”
Will/won’t → classification. How much/how many/how likely a specific value → regression. That question, asked at the start, determines everything downstream: which algorithms apply, which metrics matter, and how the output gets used.
It sounds simple. But working through it with a real business problem — an insurance company that’s losing money — made it stick in a way that a textbook definition wouldn’t have.
→ [Intro to ML #6 — coming soon]
Books to Go Deeper
① For a Rigorous but Accessible Introduction to Regression and Classification
An Introduction to Statistical Learning — Gareth James, Daniela Witten, Trevor Hastie & Robert Tibshirani (Springer, free PDF available)
The gold standard introduction to statistical learning methods, covering linear regression, classification, resampling, and more. The authors deliberately kept the math accessible without sacrificing rigor. The free PDF makes it easy to keep as a reference. If you want to understand what’s actually happening inside regression models, this is the book to reach for.
② For Hands-On Practice with Both Regression and Classification in Python
Introduction to Machine Learning with Python — Andreas Müller & Sarah Guido (O’Reilly)
Practical and approachable, this book walks through supervised learning (classification and regression alike), model evaluation, pipelines, and more using scikit-learn. The conceptual explanations are clear even if you skip the code. A solid bridge between “understanding the ideas” and “actually building models.”
