A Case Study in Failed Fine-Tuning Training: Analyzing Problems and Optimization from Qwen2.5's Loss Curve
Causes of Training Failure
Loss divergence during training stems from two broad sources: problems in the optimization process itself—parameter settings or algorithmic issues—and problems in the data. Both require careful inspection.
Optimization-Related Causes
Learning rate too high: A learning rate set too large is one of the most common causes of training divergence. When parameters shift too drastically with each update, the optimizer may "overshoot" the loss basin entirely, causing loss to rise instead of fall. In extreme cases, activation values can overflow to infinity, causing exponentials to produce INF in subsequent calculations; backpropagation then propagates NaN to weights, after which loss becomes irretrievable. If loss spikes abruptly early in training (within the first 100 iterations), high learning rate is the likely culprit.
Gradient explosion: In deep networks, backpropagated gradients can grow exponentially, making parameter updates too large for stable learning. Even with moderate learning rate, poor model initialization or structure can trigger gradient explosion and cause loss to spike dramatically. If gradients are not clipped, a single large update can break the entire model. Gradient explosion often correlates with high learning rate—which amplifies already-large gradients—but can also arise from the model's own numerical instability.
Batch Normalization instability: Batch Normalization typically stabilizes training, but can introduce instability under certain conditions. If batch size is too small, or if data distribution shifts sharply, BatchNorm outputs can fluctuate excessively and disrupt convergence. Research has shown that BatchNorm can destabilize training in sensitive tasks; in adversarial networks and similar distribution-sensitive scenarios, BatchNorm can distort the data distribution enough that the model cannot converge. If large BatchNorm layers coincide with divergence, suspect unstable running mean and variance estimates.
Data-Related Causes
Data format errors: When input data does not match model expectations, the model learns incorrect patterns, and loss computation can become corrupted. Illegal values (NaN/Inf), incorrect feature dimensions, improper normalization, missing or invalid labels—any of these will disrupt training. If labels fall outside valid range or if a computation like log(0) occurs, loss becomes Inf or NaN.
Label misalignment: If labels do not correspond correctly to inputs, the model cannot learn the intended relationship and loss will not decrease. Misalignment can occur during data preprocessing or loading. In sequence tasks, input and target sequences can become offset; in supervised learning, samples and labels can be confused. Incorrect labels effectively inject noise into training and cause severe damage. If even a single batch cannot converge when trained in isolation, label-to-sample correspondence is suspect.
Class imbalance: When different classes appear in very different frequencies, the majority class dominates gradient direction. When the optimizer encounters minority-class samples, loss can spike because the model has not yet learned their features well. Imbalance harms training stability and can cause loss oscillation or rise. In extreme cases, rare classes with persistently high loss drag down overall progress. Balancing the dataset helps smooth the loss curve.
Engineering Response Strategies
Given these possible causes, we can respond with engineering measures on both the optimization and data fronts:
Reduce learning rate: Consider dropping the learning rate by one order of magnitude or more. If NaN or large loss spikes appear within the first 100 iterations, high learning rate is likely responsible; reduce it immediately. Slower updates reduce the risk of overshooting. It may take multiple trials across different learning rate scales to find a stable range. Alternatively, switch to an adaptive optimizer (Adam, AdaGrad) to automatically tune the effective learning rate and improve stability.
Apply gradient clipping: Gradient clipping limits the maximum norm or value of gradients, preventing a single catastrophic update from breaking the model. Before each parameter step, clip gradients (e.g., cap their L₂ norm to a threshold). This is especially valuable when training large models or deep networks. Once gradient magnitudes begin to grow abnormally, clipping pulls them back into a safe range, avoiding divergence.
Use learning rate warmup: Warmup keeps the learning rate small during initial training steps, then increases it gradually to the target value. This allows the model to begin learning with safe step sizes, avoiding early divergence. Over the first few epochs or steps, linearly or exponentially increase the learning rate from a tiny value to the target, then proceed normally. Warmup smoothly guides the model into training and is especially helpful for very deep or large models—the warmup phase limits how severely parameters can diverge at the start. For fine-tuning large models like Qwen2.5-1.5B, appropriate warmup helps stabilize training onset.
Inspect the data pipeline: Thoroughly audit data loading and preprocessing to ensure inputs and labels match model expectations. Verify that the dataset contains no illegal values or misaligned labels: check that images are correctly normalized, text tokens align with labels, classification labels fall within valid ranges. Any dirty data (NaN/Inf) or format mismatch disrupts training. Before training, consider a full data cleaning and validation pass to remove obvious anomalies. Add assertions and logging to the training code to catch unreasonable values immediately.
Verify label alignment: To address label misalignment, manually inspect a small batch of training samples and their labels to confirm one-to-one correspondence. Print a few examples and verify they match. In sequence tasks, confirm that alignment strategy matches expectations (offset, padding, etc.). Correct any labeling errors or remove affected samples immediately; corrupted labels are highly destructive. A practical test: train the model on a single batch and observe whether loss decreases. If not, label-to-sample correspondence is the problem.
Balance class distribution: For imbalance-induced instability, oversample minority classes, undersample majority classes, or weight classes in the loss function so the model attends more closely to underrepresented categories. Focal loss and similar objectives give more weight to hard examples, reducing majority-class domination of gradients. Direct data augmentation can increase minority-class sample count. If feasible, aim for similar class frequencies to avoid excessive model bias. These measures smooth the training loss curve.
Monitor for anomalies and stop early: Add monitoring of loss and gradients during training. Periodically check gradient norms and loss values; if loss spikes dramatically or becomes NaN/Inf, trigger early stopping or pause to investigate. Many deep learning frameworks can auto-stop on NaN; developers can also manually monitor in the training loop. Upon divergence, save the model checkpoint and optimizer state for later diagnosis. Check whether weights contain NaN or infinity—if so, gradient explosion has occurred and the model should be restored from the last good checkpoint with the above fixes applied (reduced learning rate, etc.). Stopping early prevents wasted computation on a diverged trajectory.
Adjust model architecture and other hyperparameters: If the above steps do not stabilize training, review the model design and other hyperparameters. Inspect network depth (does a very deep network use residual connections to mitigate gradient issues?), select activations and initializations that preserve numerical stability. If BatchNorm is the problem, try larger batch sizes for more stable mean/variance estimates, reduce its momentum, or freeze pretrained BN parameters during fine-tuning. If issues persist, consider replacements like LayerNorm or GroupNorm that are less sensitive to batch statistics. Reduce gradient accumulation steps, check optimizer momentum factors, disable overly strong regularization. LLM fine-tuning often requires coordinated adjustment of multiple hyperparameters to reach stable convergence.
Summary
Analyzing the loss curve of the Qwen2.5-1.5B fine-tuning case reveals training failure: loss not only failed to decrease but rose exponentially over iterations, a sign of severe training malfunction. Possible causes include inappropriate learning rate, unchecked gradient explosion, BatchNorm-induced instability, and data format or label problems. We have outlined multiple response strategies: at the optimization level, reduce learning rate and apply warmup and gradient clipping for robust training; at the data level, validate the pipeline, verify labels, and mitigate class imbalance; and via monitoring, detect anomalies promptly. Engineering experience shows that most training divergence can be resolved by carefully diagnosing the cause and adjusting the corresponding configuration. In future model tuning, engineers should watch loss curves closely; upon detecting patterns like those in this case, promptly apply these strategies. Through continued experimentation and hyperparameter tuning, fine-tuning runs like Qwen2.5 can be restored to stable convergence and improved performance.