This is the fourth and final episode of the series showcasing Python for data exploration and analysis. In this episode we introduce how to use Pytorch to build, train, and assess a simple neural network. These are the foundations of cutting edge AI!
Neural Networks with PyTorch
A Brief Introduction to Neural Networks
Artificial Neural Networks (ANNs) are computational models inspired by the structure and function of biological neural networks found in the human brain. These algorithms consist of interconnected processing units called neurons or nodes, arranged in layers that work together to learn patterns from data.
A typical neural network architecture includes:
- An Input layer: Neurons that receive information from dataset features
- One or more Hidden layers: Intermediate processing layers that extract increasingly complex patterns
- An Output layer: Final layer that produces the network’s predictions
As data flows through the network, each neuron applies a weighted transformation to its inputs, adds a bias term, and passes the result through an activation function that determines what is passed onto the next layer. This activation function introduces non-linearity, enabling the network to model complex relationships that linear models cannot capture.
The learning process occurs through back-propagation (Hecht-Nielsen, 1989), where prediction errors are calculated and propagated backwards through the network to adjust the weights and biases. This iterative process continues until the network minimises its prediction errors on the training data.
Deep Neural Networks (DNNs) extend this concept by incorporating multiple hidden layers, allowing for hierarchical feature learning. Deep learning has achieved remarkable success in complex, high-dimensional tasks such as image recognition, natural language processing, and medical diagnosis, including melanoma detection and radiological imaging analysis.
Potential Issues: Due to these multiple layers, DNNs are heavly parametrised and as a result they are prone to over-fitting. This means your model can be poor at predicting on new data that may contain previously unseen patterns. There are strategies to prevent this such as dropout regularisation, which we will include below.
Over-fitting also means any biases in your training dataset will be captured and utilised. These inherent biases cannot be accounted for by other techniques, other than properly understanding your dataset and its flaws. Remember an ANN is good at predicting the result on new data given previous data; it’s up to you to ensure that the data you collected represents your question.
In this notebook we will run through how to use PyTorch to build a neural net to train and predict on the same Wisconsin Breast Cancer Database we explored previously in notebooks 1 and 2.
Key Objectives
- Understand some basic concepts of neural network architecture
- Set up and verify the PyTorch environment
- Load and preprocess data for deep learning
- Build a simple neural network for binary classification
- Train and evaluate the model using appropriate classification metrics
1. Environment Setup and Imports
First we import all the packages and functions we need.
With PyTorch and sklearn there are lots of submodules, so it is common to import individual functions from these modules rather than the whole package as an alias, like with Numpy, Pandas, etc.
Here we load PyTorch, and checl to see if a GPU device is available.
# Standard imports
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# PyTorch imports
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
# Sklearn for preprocessing and metrics
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, classification_report, roc_curve, auc
# Check PyTorch version and CUDA availability
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"Device: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU'}")
# Set device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"Using device: {device}")
PyTorch version: 2.7.1 CUDA available: False Device: CPU Using device: cpu
GPUs massively accelerate the kind of parrallel computation that is required to train and use neural networks. The model we build here is very small, so it’s still totally feasible to run the this notebook on a CPU-only machine. Alternatively, you can also open the notebook in Google Colab to get free, limited access to a GPU.
Click the button below, then click Runtime, Change runtime type, and under Hardware accelerator toggle the T4 GPU and click save.
2. Load Breast Cancer Dataset
We’ve already explored the dataset in the previous episodes, so we won’t repeat those steps. However, let’s confirm the target class distribution again.
# Load breast cancer dataset from local CSV file
cancer_data = pd.read_csv('./Data/breast_cancer.csv')
# Analyse the distribution of the target variable
class_dist = cancer_data['diagnosis'].value_counts()
print("Distribution of classes:")
print("-" * 50)
print(f"Benign samples: {class_dist[0]} ({class_dist[0] / len(cancer_data) * 100:.1f}%)")
print(f"Malignant samples: {class_dist[1]} ({class_dist[1] / len(cancer_data) * 100:.1f}%)")
print(f"\nClass ratio (benign:malignant): {class_dist[0]/class_dist[1]:.2f}:1")
Distribution of classes: -------------------------------------------------- Benign samples: 212 (37.3%) Malignant samples: 357 (62.7%) Class ratio (benign:malignant): 0.59:1
3. Data Preprocessing
Preprocessing is crucial for neural network training. We separate our features (cellular characteristics) from our target variable (diagnosis), split the data into training and testing sets to evaluate model performance, and apply standardisation to normalise all features to have zero mean and unit variance. This standardisation ensures that no single feature dominates the learning process due to its scale. Finally, we convert our data to PyTorch tensors and create DataLoaders that will efficiently batch and shuffle our data during training.
# Separate features and target
X = cancer_data.drop('diagnosis', axis=1).values
y = cancer_data['diagnosis'].values
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
# Standardise features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Convert to PyTorch tensors
X_train_tensor = torch.FloatTensor(X_train_scaled)
y_train_tensor = torch.FloatTensor(y_train)
X_test_tensor = torch.FloatTensor(X_test_scaled)
y_test_tensor = torch.FloatTensor(y_test)
# Create PyTorch datasets and dataloaders
train_dataset = TensorDataset(X_train_tensor, y_train_tensor)
test_dataset = TensorDataset(X_test_tensor, y_test_tensor)
batch_size = 32
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size)
print(f"Training samples: {len(train_dataset)}")
print(f"Test samples: {len(test_dataset)}")
print(f"Number of features: {X_train.shape[1]}")
print(f"Training set class distribution: {np.bincount(y_train.astype(int))}")
print(f"Test set class distribution: {np.bincount(y_test.astype(int))}")
Training samples: 455 Test samples: 114 Number of features: 30 Training set class distribution: [170 285] Test set class distribution: [42 72]
What is a tensor?
A tensor is a mathematical object that generalises familiar concepts like numbers (scalars), vectors, and matrices. Think of it as a container for data that can have different dimensions – a single number (0D), a list of numbers (1D), a table of numbers (2D), or even higher-dimensional arrangements. In machine learning, tensors are fundamental because they provide a unified way to represent and manipulate the complex, multi-dimensional data that neural networks process, from simple input features to the intricate weight matrices that define how networks learn and make predictions.
4. Define the Neural Network Model
Here we create a simple neural network for breast cancer diagnosis prediction, which we name BreastCancerNet.
Our BreastCancerNet is a feedforward neural network with four fully connected (linear) layers that progressively reduce the dimensionality: input → 64 → 32 → 16 → 1 neuron. This architecture creates a funnel-like structure that learns to extract increasingly abstract features from the cellular characteristics to predict diagnosis. We use ReLU (Rectified Linear Unit) activation functions between layers to introduce non-linearity, allowing the network to learn complex patterns. Dropout layers (20% probability) are included to prevent overfitting by randomly setting some neurons to zero during training, forcing the network to be more robust and generalisable. The final layer uses a sigmoid activation function to output a single probability between 0 and 1 for binary classification.
class BreastCancerNet(nn.Module):
def __init__(self, input_dim):
super(BreastCancerNet, self).__init__()
# Define layers with progressively decreasing dimensions
self.fc1 = nn.Linear(input_dim, 64) # First hidden layer
self.fc2 = nn.Linear(64, 32) # Second hidden layer
self.fc3 = nn.Linear(32, 16) # Third hidden layer
self.fc4 = nn.Linear(16, 1) # Output layer
# Activation and regularisation
self.relu = nn.ReLU()
self.dropout = nn.Dropout(0.2)
self.sigmoid = nn.Sigmoid() # For binary classification
def forward(self, x):
# Forward pass through the network
x = self.relu(self.fc1(x)) # Apply ReLU activation to first layer
x = self.dropout(x) # Apply dropout for regularization
x = self.relu(self.fc2(x)) # Apply ReLU activation to second layer
x = self.dropout(x) # Apply dropout for regularization
x = self.relu(self.fc3(x)) # Apply ReLU activation to third layer
x = self.fc4(x) # Final linear transformation
x = self.sigmoid(x) # Sigmoid activation for binary classification
return x
# Initialise model
input_dim = X_train.shape[1] # Number of features (30 in our case)
model = BreastCancerNet(input_dim=input_dim).to(device)
# Display model architecture
print(model)
print(f"\nTotal parameters: {sum(p.numel() for p in model.parameters())}")
BreastCancerNet( (fc1): Linear(in_features=30, out_features=64, bias=True) (fc2): Linear(in_features=64, out_features=32, bias=True) (fc3): Linear(in_features=32, out_features=16, bias=True) (fc4): Linear(in_features=16, out_features=1, bias=True) (relu): ReLU() (dropout): Dropout(p=0.2, inplace=False) (sigmoid): Sigmoid() ) Total parameters: 4609
5. Training Setup
In this section we configure the loss function, optimiser, and other parameters that control how training is performed.
The loss function essentially quantifies how “wrong” a prediction is during training. Here we use BCE (Binary Cross-Entropy). BCE measures the difference between predicted probabilities and actual binary labels. BCE is ideal for binary classification problems as it penalises confident wrong predictions more heavily and provides smooth gradients for optimisation.
The optimiser determines how to adjust the model’s weights based on each set of training examples. We select Adam Optimiser, an adaptive learning rate optimiser that combines the benefits of momentum and RMSprop. Adam automatically adjusts the learning rate for each parameter individually, making it very effective for training neural networks with minimal hyperparameter tuning.
Finally, we define two functions, train_epoch and validate. The train_epoch function performs forward propagation (calculating predictions), computes the loss, and uses backpropagation to update model weights. The validate function evaluates model performance on unseen data without updating weights, helping us monitor for overfitting.
# Loss function and optimizer
criterion = nn.BCELoss() # Binary Cross-Entropy for binary classification
optimiser = optim.Adam(model.parameters(), lr=0.001)
# Training parameters
num_epochs = 50
train_losses = []
val_losses = []
# Training function
def train_epoch(model, dataloader, criterion, optimiser, device):
model.train()
total_loss = 0
for batch_X, batch_y in dataloader:
batch_X, batch_y = batch_X.to(device), batch_y.to(device)
# Forward pass
outputs = model(batch_X).squeeze()
loss = criterion(outputs, batch_y)
# Backward pass
optimiser.zero_grad()
loss.backward()
optimiser.step()
total_loss += loss.item()
return total_loss / len(dataloader)
# Validation function
def validate(model, dataloader, criterion, device):
model.eval()
total_loss = 0
with torch.no_grad():
for batch_X, batch_y in dataloader:
batch_X, batch_y = batch_X.to(device), batch_y.to(device)
outputs = model(batch_X).squeeze()
loss = criterion(outputs, batch_y)
total_loss += loss.item()
return total_loss / len(dataloader)
6. Train the Model
Here we use what we’ve assembled so far to train the neural network, monitoring performance by calling the previous functions for each training epoch.
# Training loop
print("Starting training...")
print("-" * 50)
for epoch in range(num_epochs):
train_loss = train_epoch(model, train_loader, criterion, optimiser, device)
val_loss = validate(model, test_loader, criterion, device)
train_losses.append(train_loss)
val_losses.append(val_loss)
if (epoch + 1) % 10 == 0:
print(f"Epoch [{epoch+1}/{num_epochs}] - Train Loss: {train_loss:.4f}, Val Loss: {val_loss:.4f}")
print("-" * 50)
print("Training completed!")
Starting training... -------------------------------------------------- Epoch [10/50] - Train Loss: 0.0582, Val Loss: 0.0806 Epoch [20/50] - Train Loss: 0.0309, Val Loss: 0.0891 Epoch [30/50] - Train Loss: 0.0203, Val Loss: 0.0846 Epoch [40/50] - Train Loss: 0.0147, Val Loss: 0.0951 Epoch [50/50] - Train Loss: 0.0098, Val Loss: 0.1212 -------------------------------------------------- Training completed!
7. Visualise Training Progress
We can build some useful visualisations to show how the model improved as training progressed.
We expect to see both training and validation loss decrease with each each epoch, indicating that the model is learning. Ideally, both curves should converge to a similar low value. If the training loss continues to decrease while validation loss increases, this indicates overfitting. If both curves plateau at a high value, the model may be underfitting and need more complexity or different hyperparameters.
# Plot training history
plt.figure(figsize=(10, 6))
plt.plot(train_losses, label='Training Loss')
plt.plot(val_losses, label='Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('BCE Loss')
plt.title('Training and Validation Loss')
plt.legend()
plt.grid(True)
plt.show()
8. Model Evaluation
In this section we evaluate the model’s performance on the test set using a range of metrics:
- Accuracy: The proportion of correct predictions out of total predictions. For medical diagnosis, high accuracy is important but not the only consideration.
- Precision: The proportion of positive predictions that were actually correct. In medical context, this represents how often we correctly identify cancer when we predict it.
- Recall (Sensitivity): The proportion of actual positive cases that were correctly identified. This is crucial in medical diagnosis as we want to catch as many cancer cases as possible, minmising false benign diagnosis’s (false negatives).
- F1-Score: The harmonic mean of precision and recall, providing a balanced measure when both are important.
- ROC Curve and AUC: The ROC curve plots the true positive rate against the false positive rate at various threshold settings. AUC (Area Under the Curve) provides a single measure of performance across all classification thresholds.
We also build a Confusion Matrix, which shows the breakdown of correct and incorrect predictions for each class, helping identify if the model has bias towards certain predictions.
# Make predictions
model.eval()
with torch.no_grad():
X_test_device = X_test_tensor.to(device)
prediction_probs = model(X_test_device).squeeze().cpu().numpy()
predictions = (prediction_probs > 0.5).astype(int)
# Calculate classification metrics
accuracy = accuracy_score(y_test, predictions)
precision = precision_score(y_test, predictions)
recall = recall_score(y_test, predictions)
f1 = f1_score(y_test, predictions)
print(f"Test Set Performance:")
print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1-Score: {f1:.4f}")
print("\nDetailed Classification Report:")
print(classification_report(y_test, predictions, target_names=['Benign', 'Malignant']))
# Confusion Matrix
cm = confusion_matrix(y_test, predictions)
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
xticklabels=['Benign', 'Malignant'],
yticklabels=['Benign', 'Malignant'])
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.show()
# ROC Curve
fpr, tpr, thresholds = roc_curve(y_test, prediction_probs)
roc_auc = auc(fpr, tpr)
plt.figure(figsize=(10, 6))
plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.3f})')
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--', label='Random classifier')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic (ROC) Curve')
plt.legend(loc="lower right")
plt.grid(True, alpha=0.3)
plt.show()
print(f"\nROC AUC Score: {roc_auc:.4f}")
Test Set Performance:
Accuracy: 0.9561
Precision: 0.9855
Recall: 0.9444
F1-Score: 0.9645
Detailed Classification Report:
precision recall f1-score support
Benign 0.91 0.98 0.94 42
Malignant 0.99 0.94 0.96 72
accuracy 0.96 114
macro avg 0.95 0.96 0.95 114
weighted avg 0.96 0.96 0.96 114
ROC AUC Score: 0.9924
9. Decreasing False Negative Rates
In cancer diagnosis, false negatives (missing actual cancer cases) are extremely dangerous as they delay critical treatment. Here we implement two approaches to reduce false negatives. We might want to tune our classifier to be more cautious, flagging cases where there is any suspicion for further investigation.
We’ll compare two approaches here:
- Threshold Adjustment: Using ROC analysis to find the optimal decision threshold
- Class-Weighted Loss: Retraining with a penalty for false negatives
9a. Threshold Adjustment
The simplest approach is to adjust the decision threshold. Instead of using 0.5, we find the optimal threshold from our ROC curve that minimises false negatives (maximises sensitivity).
### Method 1: Threshold Adjustment
print("=" * 50)
print("METHOD 1: THRESHOLD ADJUSTMENT")
print("=" * 50)
# Find threshold that minimises false negatives using ROC data
fpr, tpr, roc_thresholds = roc_curve(y_test, prediction_probs)
# Find threshold where TPR is maximised (minimises false negatives)
# False Negative Rate = 1 - TPR, so maximising TPR minimises FNR
max_tpr_idx = np.argmax(tpr)
optimal_threshold = roc_thresholds[max_tpr_idx]
print(f"Optimal threshold to minimise FN: {optimal_threshold:.3f}")
print(f"This gives TPR of: {tpr[max_tpr_idx]:.3f} (sensitivity)")
print(f"This gives FPR of: {fpr[max_tpr_idx]:.3f}")
# Make new predictions with optimal threshold
predictions_adjusted = (prediction_probs >= optimal_threshold).astype(int)
# Show confusion matrix
cm_adjusted = confusion_matrix(y_test, predictions_adjusted)
plt.figure(figsize=(6, 4))
sns.heatmap(cm_adjusted, annot=True, fmt='d', cmap='Greens',
xticklabels=['Benign', 'Malignant'],
yticklabels=['Benign', 'Malignant'])
plt.title(f'Threshold Adjusted (τ={optimal_threshold:.2f}) - Confusion Matrix')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.show()
print(f"Threshold Adjusted False Negatives: {cm_adjusted[1,0]} (was {cm[1,0]})")
print(f"Threshold Adjusted False Positives: {cm_adjusted[0,1]} (was {cm[0,1]})")
print()
================================================== METHOD 1: THRESHOLD ADJUSTMENT ================================================== Optimal threshold to minimise FN: 0.065 This gives TPR of: 1.000 (sensitivity) This gives FPR of: 0.024
Threshold Adjusted False Negatives: 0 (was 4) Threshold Adjusted False Positives: 1 (was 1)
9b. Class-Weighted Loss Function
The second approach retrains the model with a loss function that penalises false negatives more heavily. We use BCEWithLogitsLoss with a pos_weight > 1 to encourage the model to predict more positive cases, reducing false negatives.
### Method 2: Class-Weighted Loss Function
print("=" * 50)
print("METHOD 2: CLASS-WEIGHTED LOSS RETRAINING")
print("=" * 50)
# Create new model with class-weighted loss
model_weighted = BreastCancerNet(input_dim=input_dim).to(device)
# Calculate class weights - penalise false negatives more heavily
# pos_weight > 1 means we encourage positive predictions more (reduces FN)
pos_weight = torch.tensor([100.0]).to(device) # Higher = more penalty for FN
criterion_weighted = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
optimizer_weighted = optim.Adam(model_weighted.parameters(), lr=0.001)
print(f"Training with pos_weight = {pos_weight.item():.1f} (higher = more FN penalty)")
# Quick retraining (fewer epochs for demo)
num_epochs_weighted = 30
for epoch in range(num_epochs_weighted):
model_weighted.train()
for batch_X, batch_y in train_loader:
batch_X, batch_y = batch_X.to(device), batch_y.to(device)
# Forward pass (note: using raw logits for BCEWithLogitsLoss)
outputs = model_weighted(batch_X).squeeze()
# Remove sigmoid from outputs since BCEWithLogitsLoss applies it internally
outputs = torch.logit(outputs.clamp(1e-7, 1-1e-7)) # Convert back to logits
loss = criterion_weighted(outputs, batch_y)
optimizer_weighted.zero_grad()
loss.backward()
optimizer_weighted.step()
if (epoch + 1) % 10 == 0:
print(f"Epoch [{epoch+1}/{num_epochs_weighted}] completed")
# Make predictions with weighted model
model_weighted.eval()
with torch.no_grad():
prediction_probs_weighted = model_weighted(X_test_tensor.to(device)).squeeze().cpu().numpy()
predictions_weighted = (prediction_probs_weighted > 0.5).astype(int)
# Show confusion matrix
cm_weighted = confusion_matrix(y_test, predictions_weighted)
plt.figure(figsize=(6, 4))
sns.heatmap(cm_weighted, annot=True, fmt='d', cmap='Oranges',
xticklabels=['Benign', 'Malignant'],
yticklabels=['Benign', 'Malignant'])
plt.title('Class-Weighted Loss Model - Confusion Matrix')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.show()
print(f"Class-Weighted False Negatives: {cm_weighted[1,0]} (was {cm[1,0]})")
print(f"Class-Weighted False Positives: {cm_weighted[0,1]} (was {cm[0,1]})")
print()
================================================== METHOD 2: CLASS-WEIGHTED LOSS RETRAINING ================================================== Training with pos_weight = 100.0 (higher = more FN penalty) Epoch [10/30] completed Epoch [20/30] completed Epoch [30/30] completed
Class-Weighted False Negatives: 0 (was 4) Class-Weighted False Positives: 4 (was 1)
10. Save the Model to the local folder
To avoid repeating the training process, which can be computationally expensive, you would typically save the model to a local file. Then it can be reloaded later for further training or to make predictions on new data.
Pytorch uses a Python-based binary format for saving models. The conventional file extension is .pt
# Save the trained model as a .pt file in the current directory
model_path = 'breast_cancer_model.pt'
torch.save({
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimiser.state_dict(),
'scaler': scaler,
'input_dim': X_train.shape[1],
'train_losses': train_losses,
'val_losses': val_losses
}, model_path)
print(f"Model saved to {model_path}")
Model saved to breast_cancer_model.pt
You can load the model like this:
# load in the models parameters
checkpoint = torch.load('./breast_cancer_model.pth', weights_only=False)
# Grab the input dimensions and initialise the model
input_dim = checkpoint['input_dim']
model = BreastCancerNet(input_dim=input_dim)
# Load the model state dictionary
model.load_state_dict(checkpoint['model_state_dict'])
# Set to evaluation mode for predictions (it switches off the drop-out layers so all neurons are active)
# If you are training further, forgo this step
model.eval()
# Move to appropriate device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)