From Traditional Image Processing to Deep Learning: Evolution of a License Plate Recognition System
2. First Release: Traditional Image Processing Route
Our earliest version didn't use deep learning at all. We "brute-forced" it with image processing and logic rules. The accuracy wasn't high, but the prototype ran quickly.
2.1 Sliding window + OpenCV for plate region detection
We used OpenCV to preprocess the image in a standard pipeline: grayscale conversion → Gaussian blur → Canny edge detection. Then we slid a window across the entire image, looking for rectangular regions with aspect ratios resembling license plates.
To filter noise, we added heuristics based on color and shape. Common blue-and-white Chinese plates are easy to segment in HSV color space. With these rules, we could roughly locate plate regions, though we occasionally misidentified advertisements or taillights.
2.2 Character segmentation + template matching for recognition
After framing the region, we split the plate characters one by one and performed template matching—comparing each character against a collection of reference letter and digit images, then picking the closest match.
Frankly, this step relies purely on pixel comparison. Any style variation breaks it: thicker strokes, tilted characters, background glare—all lead to errors.
2.3 Problems we encountered in practice
Internal testing seemed acceptable, but real deployment failed quickly:
• Rain leaves water marks on plates, recognition accuracy plummets;
• Poor lighting at night prevents edge detection;
• When vehicles enter at an angle, character segmentation breaks;
• Template matching is too sensitive to character shape variations; new energy vehicle plates basically failed;
Speed was also crippling. Sliding-window processing across a single image took too long, especially at higher resolution. CPU couldn't keep pace. We realized further optimization had limits—we needed a different approach: deep learning.
3. Upgrade Stage One: Introducing CTPN for Character Region Detection
Traditional methods were already struggling to localize plate regions, but the real bottleneck was character segmentation. Once a plate tilted even slightly, or characters had inconsistent spacing, the standard projection-based segmentation method broke. Our team spent nearly half our time tuning character segmentation, and it never stabilized.
So we decided to replace the character localization module with a more robust detection model: CTPN (Connectionist Text Proposal Network).
3.1 Why CTPN?
At that time, lightweight OCR models weren't as abundant as today. CTPN wasn't cutting-edge, but it was designed specifically for text detection in natural scenes, and it handled tilted text and background clutter well. Its mechanism was elegant: instead of framing an entire character line as a single box, it split the line into segments of small proposals, then linked them together to form the final text region.
This horizontal-connectivity concept fit license plates perfectly—a single row of characters.
3.2 Model workflow
We reused an open-source CTPN TensorFlow implementation, validated it on public datasets first, then fine-tuned it with our own collected plate images.
Training had few surprises; the bottleneck was data annotation. Each image required manual markup of character bounding boxes. We built a semi-automatic tool: traditional edge detection pre-selected regions, then we manually adjusted—much more efficient.
After training, the improvement was immediate:
• Even plates tilted slightly were framed correctly;
• Partially occluded or blurry plates still produced roughly complete character regions;
• Most importantly, once we had the character row, no further manual character splitting was needed.
This was a qualitative leap for downstream recognition.
3.3 Performance
CTPN itself wasn't lightweight, especially on edge-deployment hardware. We optimized in several ways:
• Cropped input to only regions near detected plates (using traditional methods for coarse localization first);
• Reduced input image size and used smaller anchor dimensions;
• Quantized the model and applied TensorRT optimization (done later, significant gains);
After optimization, the CTPN module averaged 300–500ms per image, with much higher accuracy than the traditional approach.
4. Upgrade Stage Two: CRNN + CTC for Character Recognition
With plate regions localized, we needed to read the characters. We first tried lightweight character classification networks—extracting each character and classifying independently with Softmax + CNN—but results were unstable. Character-level misalignment broke recognition, and this approach had no sense of character sequence.
We settled on a classic combination: CRNN + CTC.
4.1 What is CRNN?
Simply put, CRNN (Convolutional Recurrent Neural Network) has three stages:
Convolutional layers: extract image features;
RNN structure (we used bidirectional LSTM): process feature sequences and capture context;
Fully connected layer + CTC output: generate character predictions.
This structure excels at processing "an entire line of characters"—you feed in a character region and it outputs the full string directly, without requiring per-character segmentation. Perfect for license plates.
4.2 Why CTC?
CTC (Connectionist Temporal Classification) is remarkable because it doesn't require character-position alignment. During training, you only provide the full image and the text label:
Label: 鲁N Y97L0
CTC learns to align characters and positions automatically. No need to pre-mark where each character sits. For a high-annotation-cost project like ours, this was invaluable.
4.3 Training experience
Our training data came mainly from surveillance video screenshots. We wrote scripts to periodically extract frames, then manually labeled the plate text. Cleaning was tedious:
• Removed blurry and overexposed images;
• Standardized aspect ratios;
• Applied augmentation: rotation, blur, noise, color shifts to improve robustness.
We hit several pitfalls:
• Sequence length too short caused CTC to collapse (repeated characters in output);
• Incomplete character set—missed "D" and "F" prefixes for new energy vehicles;
• Imbalanced data distribution led to poor recognition on certain characters;
Overall, validation accuracy reached ~95% quickly. Crucially, the model recognized objects in scenarios where traditional methods failed completely. Even blurry images humans could barely read—it recognized them. That gave us real confidence.
5. Data Preparation and Augmentation in Practice
Model training isn't about fancy architectures. It's about data quality. Our license plate model works well because 80% of the effort went into data handling.
5.1 Data sources
Initially, we used parking lot surveillance screenshots. Every vehicle entry/exit was captured; we wrote scripts to extract frames with timestamps.
Raw data required heavy cleaning:
• Image cropping: kept only the plate region; discarded images with excessive background;
• Quality filtering: removed blurry, overexposed, and overly reflective images;
• Manual annotation: used a simple annotation tool to manually enter the true license plate number for each image;
• Character inventory: collected all possible characters and built a dictionary (including new energy prefixes, military plates, province abbreviations, etc.);
Annotation was tedious but worthwhile. Once the initial batch was clean, we used the model for pre-annotation and manual correction—much faster.
5.2 Image augmentation (actually useful)
To handle all kinds of "plate oddities," we applied extensive augmentation:
• Rotation: ±15° to simulate vehicles entering at an angle;
• Brightness variation: randomly brighten/darken to simulate day and night;
• Blur: add slight Gaussian blur to simulate focus loss;
• Noise: simulate rain and dirt;
• Color shift: HSV perturbation to simulate white-balance errors;
These weren't cosmetic. The model initially performed poorly on night scenes. We added a batch of "night-style" augmented images, retrained, and saw clear improvement.
Some problems augmentation couldn't solve—occlusion, glare—so we collected those problem images and trained them separately as specialized reinforcement.
5.3 Data distribution matters
Early on, we made an embarrassing mistake: the dataset was dominated by Shanghai plates (沪A, 沪B), so the model loved predicting "沪". We rebalanced the data to include different provinces and plate types more uniformly. That solved it.
6. Model Deployment and System Integration
Training was only half the work. For production, the model had to become a service that responds instantly on demand. This took significant effort.
6.1 Flask wrapper for the API
We chose the simplest approach: Flask + TensorFlow (1.x) native Session. No fancy frameworks—we wanted lightness and debuggability. The interface looked roughly like:
• /predict: accepts an image (base64 or multipart), runs inference, returns recognized text;
• /ping: health check;
• /reload (internal): reloads the model for hot updates.
Each request follows: image preprocessing → CTPN → CRNN+CTC → post-processing. Primitive, but simple and reliable. Stable in offline deployment.
6.2 Model loading and inference optimization
TensorFlow 1.x wasn't friendly. Initially, we created a Session per request, leading to 2–3 second inference times—unusable. We switched to global model loading with session locking. Inference time dropped to under 500ms.
Other optimization tips:
• Freeze the graph: convert trained checkpoints to .pb files for faster loading;
• Fixed batch size: always 1 during inference to avoid unnecessary memory overhead;
• Aligned image resize: standardized input dimensions to avoid dynamic reshaping;
• Preprocessing and post-processing on main thread, only protect pure model inference with locks to improve concurrency;
Minor changes, each one meaningful.
6.3 System integration: recognition + billing + gate control
The model doesn't operate in isolation. It integrates with the parking management system:
• Entry recognition: camera captures image, calls the API to recognize the plate, records entry time;
• Exit recognition + billing: recognize again, query database for duration and fee;
• Gate control: after successful recognition and payment, send the gate-open signal;
• Fallback: if recognition fails or results are inconsistent across multiple attempts, escalate to manual review;
To prevent queue buildup, the entire pipeline runs in under 1.5 seconds, with model inference capped at 700ms, leaving time for database queries, business logic, and network calls.
6.4 Deployment environment
Early locations used a small edge server (i5 + 8GB RAM, no GPU). Stress tests barely passed. As we expanded, we containerized with lightweight Docker and deployed on local-network gateway devices—more stable response times and easier maintenance.
7. Results and Lessons Learned
We were anxious on launch day, especially at the first location. Equipment connected, and immediately three or four vehicles lined up. The model had to perform.
7.1 Production performance
Our system exceeded expectations in real scenarios:
• Recognition accuracy: stable at 96%+ in daylight; ~93% at night;
• Average latency: single-image recognition in 600–800ms (full pipeline);
• Daily volume: single location ~500–1000 recognitions, peaking at 2000+;
• Deployed locations: 30+ parking lots, no major incidents.
More importantly, peak-hour queues at the entrance diminished significantly. Overall throughput improved roughly 40%. Staff said they were vastly relieved—gate keepers used to get frustrated writing down plates and calculating fees; now each day is easy.
7.2 Pitfalls we hit (lessons from experience)
Results were good, but we hit several traps:
• No character validity checks: early on, the model recognized nonsense like "1A2B3C4". We added regex rules and confidence thresholds;
• Training data too "clean": we trained on high-quality images, then reality hit hard. We added deliberately dirty data for robustness;
• Multi-threaded Session crashes: without locks in Flask, concurrent Session access caused frequent errors. We fixed it with proper locking;
• Inconsistent character format: some annotations were "沪A12345", others "沪 A12345". Training didn't account for this, and output format was inconsistent, breaking downstream logic;
These weren't model problems—they were gaps in production engineering. The system isn't a demo; it has to actually work.
7.3 Key takeaway:
The biggest lesson from this project:
License plate recognition is never a pure model problem. It's about
data, business logic, deployment, and system integration.
The jump from traditional to deep learning looks like "technology upgrade," but it's more a shift in perspective: you start caring about system-level concerns—service performance, API design, user experience, operations, and observability.
Of course, our solution isn't perfect. Later versions explored:
• Lighter models (MobileNet+CRNN) for edge deployment;
• ONNX or TensorRT for cross-platform deployment;
• Logging and auto-alerting for maintainability;
Those are stories for another time.