Classification Model Evaluation Metrics

The Confusion Matrix

These four counts form the foundation for every standard classification metric.

Point-Based Metrics

These metrics apply to a specific classification threshold (typically 0.5).

The most common trap: on a 99:1 imbalanced dataset, a model predicting "negative" for every sample achieves 99% accuracy while being useless.

Threshold-Independent Metrics

These evaluate performance across all possible thresholds.

Probability Quality Metrics

These assess the predicted probabilities themselves, not the final class labels.

Multi-Class Evaluation

When averaging metrics across multiple classes:

• Micro Average: Aggregates counts across all classes; weights each sample equally. Best for overall performance.
• Macro Average: Calculates the metric independently per class, then averages. Weights each class equally; highlights rare-class performance.
• Weighted Average: Macro average weighted by class sample counts. A compromise between the two.

If Macro-F1 is much lower than Micro-F1, your model struggles with minority classes. Always plot the class distribution first.

Common Issues

Confusing Precision and Recall. Recall measures what you catch (false negatives); Precision measures false alarms (false positives).

F1 is not universal. If your business cares more about Recall (e.g., cancer screening), use F2. If Precision matters most (e.g., legal document review), use F0.5.

Incomparable AUC scores across datasets. Different class distributions, sample difficulty, and data size affect curve shape. AUC values should not be compared directly across different test sets.

Separation vs. calibration. High ROC-AUC means the model ranks classes well but does not guarantee accurate predicted probabilities. Before production deployment, verify both with AUC and LogLoss or ECE.

Python Template

This template computes multiple metrics for an imbalanced multi-class problem using scikit-learn.

import numpy as np
from sklearn.metrics import (accuracy_score, precision_recall_fscore_support,
                             roc_auc_score, average_precision_score,
                             log_loss, brier_score_loss, confusion_matrix, matthews_corrcoef)
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# 1. Generate an imbalanced multi-class dataset
X, y = make_classification(n_samples=3000, n_classes=3,
                           weights=[0.7, 0.25, 0.05], # Imbalanced class weights
                           n_informative=5, n_redundant=2,
                           flip_y=0.03, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)

# 2. Train a simple Logistic Regression model
model = LogisticRegression(max_iter=1000, multi_class="ovr", random_state=42)
model.fit(X_train, y_train)
y_proba = model.predict_proba(X_test)
y_pred = model.predict(X_test)

# 3. Calculate various metrics
print("--- Classification Metrics ---")

# Point-based metrics (weighted average)
acc = accuracy_score(y_test, y_pred)
prec, rec, f1, _ = precision_recall_fscore_support(y_test, y_pred, average='weighted')
mcc = matthews_corrcoef(y_test, y_pred)

print(f"Accuracy (Weighted) : {acc:.4f}")
print(f"Precision (Weighted): {prec:.4f}")
print(f"Recall (Weighted)   : {rec:.4f}")
print(f"F1-Score (Weighted) : {f1:.4f}")
print(f"Matthews Corr Coef  : {mcc:.4f}\n")

# Macro-averaged metrics (better for seeing performance on rare classes)
macro_prec, macro_rec, macro_f1, _ = precision_recall_fscore_support(y_test, y_pred, average='macro')
print(f"Precision (Macro)   : {macro_prec:.4f}")
print(f"Recall (Macro)      : {macro_rec:.4f}")
print(f"F1-Score (Macro)    : {macro_f1:.4f}\n")

# Probability quality metrics
logloss = log_loss(y_test, y_proba)
# Brier Score needs to be averaged across classes
brier = np.mean([brier_score_loss((y_test == k), y_proba[:, k]) for k in np.unique(y_test)])

print(f"LogLoss             : {logloss:.4f}")
print(f"Brier Score (Avg)   : {brier:.4f}")

Example Output:

--- Classification Metrics ---
Accuracy (Weighted) : 0.8944
Precision (Weighted): 0.8861
Recall (Weighted)   : 0.8944
F1-Score (Weighted) : 0.8862
Matthews Corr Coef  : 0.7183

Precision (Macro)   : 0.7989
Recall (Macro)      : 0.7311
F1-Score (Macro)    : 0.7552

LogLoss             : 0.2819
Brier Score (Avg)   : 0.0542

The Macro-F1 (0.755) is notably lower than Weighted-F1 (0.886). This gap reveals that the model fails on rare classes—a critical insight hidden by overall accuracy.

Choosing Metrics

1. Define business cost: What does each error type cost?
2. Select point metrics: With known costs, optimize the threshold and report Precision/Recall/Fβ. Without known costs, F1-Score is a reasonable default.
3. Check probability calibration: Before deployment, verify LogLoss and ECE or Brier Score.
4. Assess ranking power: Use AUPRC (imbalanced) or ROC-AUC (balanced) to evaluate overall discriminative ability.
5. Address imbalance: If Macro-F1 ≪ Micro-F1, analyze which minority classes underperform. Consider resampling, cost-sensitive learning, or Focal Loss.