This is Part 3 of the Python project walkthrough with L2D. Here we dive into building and testing models, using classical machine learning techniques.
Classical Machine Learning for Breast Cancer Prediction
This notebook demonstrates a streamlined classical machine learning (ML) approach for breast cancer diagnosis. We will build a selection of models, using multiple algorithms and techniques, and compare their performance. As in the previous episode, the dataset we will be using is the Wisconsin Breast Cancer Database.
Key Objectives
- Apply multiple classical ML algorithms
- Perform feature selection and preprocessing
- Use cross-validation and hyperparameter tuning
- Compare model performance and interpretability
1. Importing Packages and Loading Data
First off, let’s import the main functions we’ll be using. As a Python user, you have a powerful toolkit for machine learning at your disposal.
- Numpy is a numerical computing library with an efficient backend implemented in C.
- Pandas provides DataFrame functionality, which is a convenient way of storing and manipulating labelled data.
- Matplotlib is for building visualisations, and Seaborn extends its functionality with high-level statistical plots.
- Most importantly for our purposes, Scikit-learn implements a huge number of algorithms with a consistent API.
Scikit-learn is used by everyone from scientists to multinational companies. It is built on Numpy, SciPy, and Matplotlib, and mostly integrates well with Pandas. Because it has so many capabilities, it contains a lot of packages, so here we explicitly import the functions we want to use to avoid having to name them with long references.
# Core libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# Scikit-learn essentials
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, classification_report
# Classical ML algorithms
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
import warnings
warnings.filterwarnings('ignore')
plt.style.use('seaborn-v0_8')
Next we load the data. We can read it directly into a Pandas dataframe as before, and print out a list of the column headings.
data = pd.read_csv('data/breast_cancer.csv')
print(data.columns)
# In the diagnosis column, 0 = benign, 1 = malignant
Index(['diagnosis', 'radius_mean', 'texture_mean', 'perimeter_mean',
'area_mean', 'smoothness_mean', 'compactness_mean', 'concavity_mean',
'concave points_mean', 'symmetry_mean', 'fractal_dimension_mean',
'radius_se', 'texture_se', 'perimeter_se', 'area_se', 'smoothness_se',
'compactness_se', 'concavity_se', 'concave points_se', 'symmetry_se',
'fractal_dimension_se', 'radius_worst', 'texture_worst',
'perimeter_worst', 'area_worst', 'smoothness_worst',
'compactness_worst', 'concavity_worst', 'concave points_worst',
'symmetry_worst', 'fractal_dimension_worst'],
dtype='object')
Data Exploration
Here we take the loaded data and check the shape, to see how many rows and columns are present, and look for missing values as a basic integrity check.
Then we check the distribution of the target variable – the thing we want to predict, which in our case is the diagnosis. We want to know the proportion of benign and malignant cases in our dataset. This is important, because some algorithms perform poorly when one class is better represented than another.
# Load and explore the data:
print(f"Data shape: {data.shape}")
# Missing values (if any):
print(f"\nMissing values: {data.isnull().sum().sum()}")
print("\nClasses:\n0: Benign\n1: Malignant")
# Target distribution:
print(f"\nTarget distribution:")
print(data['diagnosis'].value_counts())
print(f"\nTarget proportions:")
print(data['diagnosis'].value_counts(normalize=True))
# Visualise target distribution with a Pandas Plot:
plt.figure(figsize=(5, 5))
data['diagnosis'].value_counts().plot(kind='bar', color=['skyblue', 'lightcoral']) # Pandas plot function.
plt.title('Diagnosis Distribution')
# value_counts returns classes in descending order and x-tick locations relate to the position.
# Hence, Malignant (1) is at x=0, and Benign (0) is at x=1.
plt.xticks([0, 1], ['Malignant', 'Benign'], rotation=0)
plt.tight_layout()
plt.show()
Data shape: (569, 31) Missing values: 0 Classes: 0: Benign 1: Malignant Target distribution: diagnosis 1 357 0 212 Name: count, dtype: int64 Target proportions: diagnosis 1 0.627417 0 0.372583 Name: proportion, dtype: float64
2. Feature Analysis & Selection
This section first separates the dataset into predictors (X) and the outcome variable (y) so that the features can be analysed independently of the label. The diagnosis column (0 = benign, 1 = malignant) is isolated as y, while the remaining clinical measurements are stored as X.
Next, a simple univariate correlation is computed between each feature and the binary target; taking the absolute value of each correlation provides a ranking by strength regardless of direction (positive or negative), which makes it easy to identify the features most strongly associated with malignancy. The top ten features by absolute correlation are printed so you can inspect the strongest individual signals, and the top fifteen are plotted as a bar chart to give a compact visual comparison of relative importance.
Note that this step is exploratory: correlation highlights linear associations and does not prove causation or capture non-linear relationships. The actual model pipeline will use a formal selection method. Importantly, this will be applied after splitting the data into training and testing partitions, to avoid information leakage.
# Create the feature matrix X by dropping the 'diagnosis' column from the dataset
X = data.drop('diagnosis', axis=1)
# Create the target vector y by selecting the 'diagnosis' column
y = data['diagnosis']
# Calculate the correlation between each feature and the target,
# take the absolute values, and sort from highest to lowest
correlations = X.corrwith(y).abs().sort_values(ascending=False)
print("Top 10 Features by Correlation with Target:")
print("-" * 45)
# Loop through the top 10 correlations (with index i starting from 1)
for i, (feature, corr) in enumerate(correlations.head(10).items(), 1):
print(f"{i:2d}. {feature:<25} | {corr:.3f}")
# Set up a new figure for the bar plot with size 10 by 6 inches
plt.figure(figsize=(10, 6))
correlations.head(15).plot(kind='bar')
plt.title('Feature Correlation with Target (Top 15)')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()
Top 10 Features by Correlation with Target: --------------------------------------------- 1. concave points_worst | 0.794 2. perimeter_worst | 0.783 3. concave points_mean | 0.777 4. radius_worst | 0.776 5. perimeter_mean | 0.743 6. area_worst | 0.734 7. radius_mean | 0.730 8. area_mean | 0.709 9. concavity_mean | 0.696 10. concavity_worst | 0.660
2a. Clinical Features
This table gives a description of the features in the dataset, and their typical relationship with the diagnosis.
| Feature | Meaning | Malignant tendency |
|---|---|---|
| concave points_worst | Number of concave (indented) parts of the nuclei contour, worst-case value | Strongest correlation – malignant tumours have more concave points |
| perimeter_worst | Perimeter of nuclei contour, worst-case value | Larger perimeter → more likely malignant (bigger/irregular cells) |
| concave points_mean | Average number of concave points across nuclei | Malignant tumours show greater irregularity |
| radius_worst | Largest radius of nuclei (worst case) | Malignant cells tend to be larger |
| perimeter_mean | Average perimeter across nuclei | Larger in malignant |
| area_worst | Largest area measurement | Malignant nuclei typically cover more area |
| radius_mean | Average radius of nuclei | Larger on average in malignant |
| area_mean | Average area across nuclei | Malignant = bigger cell nuclei |
| concavity_mean | Average severity of concave portions of nuclei | More irregular shapes → malignancy |
| concavity_worst | Worst-case concavity | Malignant cells tend to have deep irregular indentations |
| compactness_mean | Roundness/tightness (perimeter² / area), mean | Malignant tumours less compact |
| compactness_worst | Roundness/tightness (worst case) | Higher = more irregular |
| radius_se | Standard error of radius | Higher = more variability in nucleus size (malignant) |
| perimeter_se | Standard error of perimeter | Indicates irregularity/variation in cell boundary |
| area_se | Standard error of area | Malignant nuclei vary more in size |
3. Data Preprocessing & Splitting
We prepare the data for modelling by selecting the most informative predictors, splitting into train/test sets, and standardising features to ensure fair treatment by scale-sensitive algorithms. Concretely, the dataset is split 80/20 with stratification on the diagnosis to preserve class balance. The split is random, but the randomiser uses a fixed seed for reproducibility.
A StandardScaler is fit on the training set and used to transform both training and test features. Scaling helps algorithms like SVM, KNN and logistic regression converge and behave sensibly – without it, variables that have a larger range tend to dominate predictions. Tree-based methods are generally more robust to differences in scale. Fitting the scaler only on the training set avoids information leakage.
Feature selection is carried out on the scaled training data using a univariate ANOVA F-test (SelectKBest with f_classif). Compared with simple correlation, the ANOVA F-test explicitly measures how well each feature discriminates between the two diagnostic classes by comparing between-class and within-class variance — a more principled way to rank predictive power. We keep the top 15 features to reduce dimensionality while retaining the strongest signals (a choice informed by the prior correlation analysis). The selector is fit only on the training set and then applied to the test , again to avoid leakage. The names of the selected features are printed for reference before proceeding to model training.
# Split the data:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
# Feature scaling:
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Feature selection - top 15 features:
selector = SelectKBest(f_classif, k=15)
X_train_selected = selector.fit_transform(X_train_scaled, y_train)
X_test_selected = selector.transform(X_test_scaled)
# Get selected feature names:
selected_features = X.columns[selector.get_support()].tolist()
print(f"Selected features ({len(selected_features)}):")
for i, feature in enumerate(selected_features, 1):
print(f"{i:2d}. {feature}")
Selected features (15): 1. radius_mean 2. perimeter_mean 3. area_mean 4. compactness_mean 5. concavity_mean 6. concave points_mean 7. radius_se 8. perimeter_se 9. area_se 10. radius_worst 11. perimeter_worst 12. area_worst 13. compactness_worst 14. concavity_worst 15. concave points_worst
# Print the ANOVA f-value returned by f_classif for each feature, sorted in descending order:
f_values, p_values = f_classif(X_train_scaled, y_train)
feature_scores = pd.DataFrame({'Feature': X.columns, 'F-value': f_values, 'p-value': p_values})
feature_scores = feature_scores.sort_values(by='F-value', ascending=False).reset_index(drop=True)
print("\nFeature Scores (sorted by F-value):")
print(feature_scores.head(15))
# Set up a new figure for the bar plot with size 10 by 6 inches
plt.figure(figsize=(10, 6))
feature_scores.head(15).plot(kind='bar', x='Feature', y='F-value', legend=False)
plt.title('Feature ANOVA F-values')
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()
Feature Scores (sorted by F-value):
Feature F-value p-value
0 concave points_worst 795.223341 9.269318e-102
1 perimeter_worst 719.598823 1.327313e-95
2 concave points_mean 703.364031 3.135578e-94
3 radius_worst 687.430367 7.296427e-93
4 perimeter_mean 565.485755 1.011815e-81
5 area_worst 541.713694 2.150031e-79
6 radius_mean 523.613907 1.387363e-77
7 area_mean 469.516453 5.741182e-72
8 concavity_mean 416.374874 4.058059e-66
9 concavity_worst 365.747389 3.351548e-60
10 compactness_mean 269.466042 7.418408e-48
11 compactness_worst 257.926057 2.885121e-46
12 area_se 214.858763 4.296371e-40
13 radius_se 212.499767 9.611165e-40
14 perimeter_se 198.422440 1.247305e-37
<Figure size 1000x600 with 0 Axes>
4. Model Training & Evaluation
In this section we built and evaluate our models. We define a small suite of classical classifiers (logistic regression, random forest, SVM, k‑nearest neighbours and gradient boosting) and train each one on the preprocessed training data. Using the same selected features and scaling avoids data leakage and ensures each algorithm is compared on an identical input representation, so differences in performance reflect the model behaviour rather than data preprocessing choices.
For each model we fit to the training split, and generate predictions and predicted probabilities on the held‑out test set. Because not all models produce probabilities, we check that probabilities are available – even though the models we use here do all produce them.
We also compute a set of complementary evaluation metrics (accuracy, precision, recall and F1) to capture different aspects of predictive performance, and use ROC‑AUC to assess ranking ability independent of a threshold. To estimate generalisation more robustly we also compute 5‑fold cross‑validation accuracy on the training data. All results are collected into a dictionary and converted to a Pandas DataFrame so that models can be compared side‑by‑side in a clear tabular format.
# Define models in a dictionary:
models = {
'Logistic Regression': LogisticRegression(random_state=42),
'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
'SVM': SVC(random_state=42, probability=True),
'K-Nearest Neighbors': KNeighborsClassifier(),
'Gradient Boosting': GradientBoostingClassifier(random_state=42)
}
# Iterate through, train and evaluate each model:
results = {}
for name, model in models.items():
# Train model
model.fit(X_train_selected, y_train)
# Make predictions
y_pred = model.predict(X_test_selected)
y_pred_proba = model.predict_proba(X_test_selected)[:, 1] if hasattr(model, 'predict_proba') else None
# Calculate metrics
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)
roc_auc = roc_auc_score(y_test, y_pred_proba) if y_pred_proba is not None else None
# Cross-validation score
cv_score = cross_val_score(model, X_train_selected, y_train, cv=5, scoring='accuracy').mean()
results[name] = {
'Accuracy': accuracy,
'Precision': precision,
'Recall': recall,
'F1-Score': f1,
'ROC-AUC': roc_auc,
'CV Score': cv_score
}
# Display results
results_df = pd.DataFrame(results).T
print("Model Performance Comparison:")
print("=" * 60)
print(results_df.round(3))
Model Performance Comparison:
============================================================
Accuracy Precision Recall F1-Score ROC-AUC CV Score
Logistic Regression 0.965 0.972 0.972 0.972 0.991 0.941
Random Forest 0.956 0.959 0.972 0.966 0.991 0.947
SVM 0.947 0.958 0.958 0.958 0.991 0.947
K-Nearest Neighbors 0.956 0.972 0.958 0.965 0.980 0.936
Gradient Boosting 0.939 0.945 0.958 0.952 0.990 0.927
4b. Model Comparison & Visualisation
This section builds a compact 2×2 figure that summarises model performance from several complementary perspectives, so we can quickly compare the strengths and weaknesses of each model. The top-left and top-right panels show simple bar charts of Accuracy and F1‑Score across the candidate classifiers — these give a straightforward view of overall correctness and the balance between precision and recall, respectively. The lower-left panel displays ROC‑AUC, which measures ranking ability independent of any chosen decision threshold and is therefore useful when you care about relative confidence rather than a hard class label.
The lower-right panel is a heatmap of key metrics (Accuracy, Precision, Recall, F1‑Score) that places multiple performance dimensions side‑by‑side; this makes it easy to spot models that trade a bit of accuracy for substantially better precision or recall, for example. We format the graphs to make them more readable, so that the relatively small differences between high‑performing models are easier to interpret.
Finally, the visualisation also highlights the best model by a chosen criterion (here accuracy) and prints its name and score so you have a clear, reproducible pick for subsequent hyperparameter tuning and final evaluation.
# Visualise model performance:
fig, axes = plt.subplots(2, 2, figsize=(15, 10))
# Accuracy comparison:
axes[0, 0].bar(results_df.index, results_df['Accuracy'], color='skyblue')
axes[0, 0].set_title('Model Accuracy Comparison')
axes[0, 0].set_ylabel('Accuracy')
axes[0, 0].tick_params(axis='x', rotation=45)
axes[0, 0].set_ylim(0.8, 1.0)
# F1-Score comparison:
axes[0, 1].bar(results_df.index, results_df['F1-Score'], color='lightgreen')
axes[0, 1].set_title('F1-Score Comparison')
axes[0, 1].set_ylabel('F1-Score')
axes[0, 1].tick_params(axis='x', rotation=45)
axes[0, 1].set_ylim(0.8, 1.0)
# ROC-AUC comparison (excluding models without probability):
roc_data = results_df.dropna(subset=['ROC-AUC'])
axes[1, 0].bar(roc_data.index, roc_data['ROC-AUC'], color='lightcoral')
axes[1, 0].set_title('ROC-AUC Comparison')
axes[1, 0].set_ylabel('ROC-AUC')
axes[1, 0].tick_params(axis='x', rotation=45)
axes[1, 0].set_ylim(0.8, 1.0)
# Overall metrics heatmap:
metrics_heatmap = results_df[['Accuracy', 'Precision', 'Recall', 'F1-Score']]
sns.heatmap(metrics_heatmap, annot=True, cmap='Blues', fmt='.3f', ax=axes[1, 1])
axes[1, 1].set_title('Performance Metrics Heatmap')
plt.tight_layout()
plt.show()
# Find the best model:
best_model_name = results_df['Accuracy'].idxmax()
print(f"\nBest performing model: {best_model_name}")
print(f"Accuracy: {results_df.loc[best_model_name, 'Accuracy']:.3f}")
Best performing model: Logistic Regression Accuracy: 0.965
6. Hyperparameter Tuning (Best Model)
In this step we perform targeted hyperparameter tuning to see if we can improve the accuracy of the model that performed best out-of-the-box, using a method called GridSearchCV. We define an appropriate parameter grid for each algorithm (for example: number and depth of trees for Random Forest, kernel/C/gamma combinations for SVM, or regularisation strength and penalty for Logistic Regression) and run GridSearchCV with 5‑fold cross‑validation to find the combination that maximises accuracy on the training set. GridSearchCV is fitted only on the training data (using the previously selected and scaled features), again to avoid data leakage. Once the best parameters are found, the corresponding estimator is applied to the held‑out test set. We print the optimal hyperparameters and the best cross‑validation score, then report final test performance — overall accuracy and a detailed classification report (precision, recall and F1 for each class) — so you can assess how well the tuned model generalises to unseen samples.
# Hyperparameter tuning for the best model:
if best_model_name == 'Random Forest':
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [None, 10, 20],
'min_samples_split': [2, 5, 10]
}
best_model = RandomForestClassifier(random_state=42)
elif best_model_name == 'SVM':
param_grid = {
'C': [0.1, 1, 10],
'gamma': ['scale', 'auto', 0.1, 1],
'kernel': ['rbf', 'linear']
}
best_model = SVC(random_state=42, probability=True)
elif best_model_name == 'Logistic Regression':
param_grid = {
'C': [0.1, 1, 10, 100],
'penalty': ['l1', 'l2'],
'solver': ['liblinear', 'saga']
}
best_model = LogisticRegression(random_state=42)
else:
# Default to Random Forest if other models are best
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [None, 10, 20]
}
best_model = RandomForestClassifier(random_state=42)
# Grid search:
grid_search = GridSearchCV(best_model, param_grid, cv=5, scoring='accuracy', n_jobs=-1)
grid_search.fit(X_train_selected, y_train)
print(f"Best parameters for {best_model_name}:")
print(grid_search.best_params_)
print(f"Best cross-validation score: {grid_search.best_score_:.3f}")
# Final evaluation:
final_model = grid_search.best_estimator_
final_predictions = final_model.predict(X_test_selected)
final_accuracy = accuracy_score(y_test, final_predictions)
print(f"\nFinal optimised model accuracy: {final_accuracy:.3f}")
print("\nDetailed Classification Report:")
print(classification_report(y_test, final_predictions, target_names=['Benign', 'Malignant']))
Best parameters for Logistic Regression:
{'C': 0.1, 'penalty': 'l1', 'solver': 'liblinear'}
Best cross-validation score: 0.947
Final optimised model accuracy: 0.965
Detailed Classification Report:
precision recall f1-score support
Benign 0.93 0.98 0.95 42
Malignant 0.99 0.96 0.97 72
accuracy 0.96 114
macro avg 0.96 0.97 0.96 114
weighted avg 0.97 0.96 0.97 114
7. Models explained
This table summarises the models we built here.
| Model | How it Works | Pros | Cons | Why it performed as shown |
|---|---|---|---|---|
| Logistic Regression | Fits a linear equation to the features and applies a sigmoid to predict probability of malignancy vs benign. Learns coefficients that describe how each feature affects log-odds of outcome. | – Very fast and interpretable – Produces probabilities – Works well on linearly separable problems |
– Only linear boundaries – Sensitive to multicollinearity – Limited with complex feature interactions |
Breast cancer features (like radius, texture, concavity) are reasonably linearly separable, so logistic regression already achieves high accuracy and ROC-AUC. Its simplicity means little overfitting, hence stable performance. |
| Random Forest | An ensemble of decision trees trained on bootstrapped subsets of the data and features. Prediction = majority vote (classification). | – Handles non-linearities – Robust to noise/outliers – Works well without scaling – Provides feature importance |
– Can be slower on very large datasets – Less interpretable – May overfit if trees are deep |
Performs similarly to logistic regression here because the dataset is relatively low-dimensional (30 features) and well-structured. Gains from capturing non-linearities are modest because many features are individually strong predictors. |
| Support Vector Machine (SVM) | Finds a hyperplane that maximises the margin between classes. With kernels, can model non-linear decision boundaries. | – Effective in high-dimensional spaces – Works well with clear margin separation – Flexible with kernels |
– Sensitive to parameter choice – Can be slower on larger datasets – Harder to interpret |
Slightly lower scores than logistic regression and RF — probably because the data already separates well linearly, so the SVM’s extra complexity does not add much benefit and may even slightly reduce generalisation. |
| K-Nearest Neighbours (KNN) | Classifies a new point by looking at the majority class of its k nearest neighbours in feature space. | – Simple, intuitive – No training phase – Captures local structure |
– Slow at prediction on larger datasets – Sensitive to scaling and irrelevant features – Struggles in high dimensions |
Works well here because malignant and benign tumours cluster separately in feature space. However, noise and overlapping regions reduce its stability compared to LR and RF. |
| Gradient Boosting | Builds an ensemble of weak learners (shallow trees) sequentially, where each tree corrects errors of the previous. | – High accuracy – Captures complex patterns – Robust with tuning |
– Computationally intensive – Sensitive to hyperparameters – Can overfit small datasets |
Slightly lower in your chart because the dataset is not very complex — boosting adds unnecessary complexity and may overfit noise. With tuning, performance could rise, but here simplicity wins. |
8. Summary
This streamlined classical ML workflow demonstrates:
- Efficient data loading and exploration with key statistics and visualisations
- Feature analysis using correlation analysis and feature selection
- Multiple classical ML algorithms comparison with standardised evaluation
- Hyperparameter optimization for the best-performing model
- Model interpretability through feature importance analysis
The workflow provides a solid foundation for breast cancer diagnosis using classical machine learning techniques, with clear performance metrics and model comparison.