Yuxu Ge
Article · 2023-04-27

Data Preprocessing in Machine Learning

Data preprocessing addresses fundamental data quality issues: removing invalid or erroneous records, filling missing values, and normalizing the scale, units, and format so that raw data becomes suitable for model training. Strong preprocessing improves both training efficiency and model performance—and typically consumes more effort than hyperparameter tuning. It is one of the most critical foundations of machine learning engineering.

We cover common preprocessing steps and methods below, with explanations of why each step matters for model performance.

Handling Missing Values: Completing Incomplete Data

Real-world data is rarely complete. Missing values force difficult choices: many algorithms fail or silently treat empty cells as zero, introducing systematic bias.

Two primary strategies exist:

Consider a product pricing feature with some missing prices: impute using the feature's mean price, allowing the model to train on most samples without halting on null values. For user-submitted forms with missing age fields, either impute using the mean age of respondents who provided it, or add a boolean feature "age missing" so the model learns whether missingness correlates with outcomes.

A brief code example imputes missing values using the mean:

import numpy as np

# 示例数据，包含一个缺失值 np.nan
data = np.array([1.0, 2.5, np.nan, 4.0])
print("原始数据：", data)  # 输出原始数组，其中第三个元素为 nan

# 计算非缺失元素的平均值
mean_val = np.nanmean(data)  
print("非缺失值平均值：", mean_val)  # 输出计算得到的均值

# 用平均值填充缺失位置
data[np.isnan(data)] = mean_val  
print("填充缺失值后：", data)  # 输出填补缺失值后的数组

output

原始数据： [1.  2.5 nan 4. ]
非缺失值平均值： 2.5
填充缺失值后： [1.  2.5 2.5 4. ]

Here, np.nanmean automatically ignores NaN when computing the mean. After imputation, NaN values are replaced with the mean, and the data has no gaps.

Why does handling missing values matter? Untreated missing values cause most algorithms (such as scikit-learn implementations) to fail outright or refuse to train. Imputation with unsuitable values—such as filling everything with zero—introduces systematic bias and teaches the model incorrect statistical patterns. Choose strategies grounded in domain understanding, aiming to preserve the data's true distribution or provide interpretable signals.

In practice, different features often need different strategies. Age missingness might use mean age imputation; missing review text might use an empty string or special token. Always record which values were imputed, so the model can distinguish inferred from observed data if needed. Scikit-Learn's SimpleImputer and similar tools handle this conveniently, though understanding the logic remains essential.

Scaling Numerical Features: Making Ranges Comparable

Numerical features often span different units and ranges. Feeding raw values directly to a model allows features with larger magnitude to dominate predictions disproportionately. For example, predicting health from height (cm: ~150–180) and weight (kg: ~50–80) risks the model relying more on height simply because its numbers are larger. Distance-sensitive algorithms like KNN and gradient-based methods like linear regression or neural networks amplify this imbalance. To prevent such "unit bias," scale features to a common range.

Two main approaches follow.

Standardization (Mean Centering)

Standardization transforms features to zero mean and unit variance using:

$$X' = \frac{X - \mu}{\sigma}$$

where $\mu$ is the feature's mean and $\sigma$ its standard deviation. After this linear transformation, data oscillates around zero, with most values falling within ±3 for normally distributed data. Standardization ensures every feature has comparable "baseline" and "spread," preventing any single feature from dominating the model.

Example: Consider samples with three features each:

import numpy as np
from sklearn.preprocessing import scale

# 样本数据：每列代表一个特征
raw_samples = np.array([
    [3.0,  -1.0,  2.0],
    [0.0,   4.0,  3.0],
    [1.0,  -4.0,  2.0]
])
print("原始数据：\n", raw_samples)
print("每列特征的均值：", raw_samples.mean(axis=0))
print("每列特征的标准差：", raw_samples.std(axis=0))

# 使用 sklearn 的 scale 函数进行标准化（均值归零，方差归一）
std_samples = scale(raw_samples)
print("标准化后的数据：\n", std_samples)
print("标准化后每列特征的均值：", std_samples.mean(axis=0))
print("标准化后每列特征的标准差：", std_samples.std(axis=0))

output

原始数据：
 [[ 3. -1.  2.]
 [ 0.  4.  3.]
 [ 1. -4.  2.]]
每列特征的均值： [ 1.33333333 -0.33333333  2.33333333]
每列特征的标准差： [1.24721913 3.29983165 0.47140452]
标准化后的数据：
 [[ 1.33630621 -0.20203051 -0.70710678]
 [-1.06904497  1.31319831  1.41421356]
 [-0.26726124 -1.1111678  -0.70710678]]
标准化后每列特征的均值： [ 5.55111512e-17  0.00000000e+00 -2.96059473e-16]
标准化后每列特征的标准差： [1. 1. 1.]

Running this code shows:

Standardization levels the playing field. This is especially important for linear regression, logistic regression, and neural networks: standardized features make gradient descent more stable and convergent; model weight updates remain fair across features instead of biasing toward arbitrarily large values.

Intuition: Standardization resembles converting measurements to a common standard. Comparing two people's wealth becomes misleading if one uses Chinese yuan and the other Japanese yen—1 yen is much smaller than 1 yuan. Only after converting both to the same currency can you fairly compare wealth. Standardizing features gives the model a fair basis to evaluate each feature's contribution.

Min-Max Normalization (Range Scaling)

Normalization typically rescales data to a fixed interval, commonly [0, 1]:

$$X' = \frac{X - X_{\min}}{X_{\max} - X_{\min}}$$

where $X_{\min}$ and $X_{\max}$ are the feature's minimum and maximum. After this mapping, the minimum becomes 0, the maximum becomes 1, and intermediate values scale proportionally.

Normalization also makes feature ranges comparable. It is particularly useful for distance-based methods: if one feature spans a much wider range, it dominates distance calculations; normalization prevents this. Scaling inputs to [0, 1] can also accelerate convergence in some models, such as neural network gradient descent.

A simple example of Min-Max normalization:

import numpy as np
from sklearn.preprocessing import MinMaxScaler

# 样本数据：3个特征，每列数值差异较大
raw_samples = np.array([
    [ 1.0,  2.0,  300.0],
    [ 4.0,  5.0,  600.0],
    [ 7.0,  8.0,  900.0]
])
print("原始数据：\n", raw_samples)

# 初始化一个MinMax缩放器，将范围缩放到[0,1]
mms = MinMaxScaler(feature_range=(0, 1))
scaled_samples = mms.fit_transform(raw_samples)
print("Min-Max归一化后的数据：\n", scaled_samples)

output

原始数据：
 [[  1.   2. 300.]
 [  4.   5. 600.]
 [  7.   8. 900.]]
Min-Max归一化后的数据：
 [[0.  0.  0. ]
 [0.5 0.5 0.5]
 [1.  1.  1. ]]

If the original data's third column is much larger (hundreds) than the first two (single digits), the output shows:

Caveat: Min-Max normalization compresses the value distribution and is extremely sensitive to outliers. A single extreme value squeezes normal data toward 0, destroying resolution. Preprocess outliers or use a more robust scaling method.

Engineering note: Scikit-Learn's MinMaxScaler handles this conveniently. To scale to a range other than [0, 1], pass feature_range=(min, max) when creating the scaler.

Sample-wise Normalization

The above two methods scale across feature columns. Sometimes you need to normalize each sample individually so that all features in a single row sum to 1 (or have unit norm). This is useful when you care about proportions rather than absolute magnitudes.

For example, consider programming language usage across two years: 2017 had Python 100k users, Java 200k, PHP 50k; 2018 had Python 80k, Java 100k, PHP 10k. Year-to-year totals differ. Comparing raw counts, Python appears to shrink (80k < 100k), but its share of total developers actually grew. Row-wise normalization converts each year to a proportion, making relative shifts clear.

Scikit-Learn's preprocessing.normalize enables sample-wise normalization. Set norm='l1' to scale each sample so feature values sum to 1 (absolute values); set norm='l2' to scale so the feature vector has unit length. Here is l1 normalization:

from sklearn.preprocessing import normalize
import numpy as np

raw_samples = np.array([
    [10.0, 20.0, 5.0],
    [ 8.0, 10.0, 1.0]
])
# 使用 L1 范数归一化每个样本（行）
norm_samples = normalize(raw_samples, norm='l1')
print("按样本归一化后的数据：\n", norm_samples)

output

按样本归一化后的数据：
 [[0.28571429 0.57142857 0.14285714]
 [0.42105263 0.52631579 0.05263158]]

Each output row's features sum to 1. For example, [10, 20, 5] becomes [0.2857, 0.5714, 0.1429]—each element is its fraction of the total (verify: 0.2857 + 0.5714 + 0.1429 = 1). This preprocessing is invaluable when comparing component composition rather than raw magnitude, such as normalizing term-frequency vectors in text.

Binarization: Thresholding for Simplicity

Sometimes you care only whether a value exceeds a threshold, not its exact magnitude. Binarization converts numerical features to binary (0 or 1): below threshold becomes 0, above becomes 1. This simplifies the model while retaining key information. Image processing often binarizes grayscale images to highlight edges while ignoring subtle shading.

Scikit-Learn makes binarization simple:

import sklearn.preprocessing as sp
import numpy as np

raw_samples = np.array([[65.5, 89.0, 73.0],
                        [55.0, 99.0, 98.5],
                        [45.0, 22.5, 60.0]])
binarizer = sp.Binarizer(threshold=60)  # 定义阈值为60
bin_samples = binarizer.fit_transform(raw_samples)
print("二值化处理后的数据：\n", bin_samples)

output

二值化处理后的数据：
 [[1. 1. 1.]
 [0. 1. 1.]
 [0. 0. 0.]]

With threshold 60, output values ≤ 60 become 0 and values > 60 become 1. Caution: binarization discards numerical granularity (65.5 and 89.0 both become 1) and is irreversible (you cannot recover the original values from 0/1). Use binarization only if your model truly needs only threshold information. For reversible encoding that represents categories, consider one-hot encoding below.

Binarization suits specific scenarios: converting continuous audio to silence/sound (0/1), or marking pass/fail on an exam. Threshold selection is critical and domain-dependent.

Categorical Encoding: One-Hot and Label Encoding

Most machine learning models cannot accept categorical text features directly. Attributes like gender, color, or brand need numerical encoding. Two common methods are one-hot encoding and label encoding, suited to different scenarios.

One-Hot Encoding

import numpy as np
from sklearn.preprocessing import OneHotEncoder

# 原始数据：每行一个样本，包含三个类别特征
raw_samples = np.array([
    [1,  3,  2],
    [7,  5,  4],
    [1,  8,  6],
    [7,  3,  9]
])
# 定义 OneHotEncoder，sparse=False 表示输出稠密NumPy数组
one_hot = OneHotEncoder(sparse_output=False)
oh_samples = one_hot.fit_transform(raw_samples)
print("独热编码后的结果：\n", oh_samples)
print("编码后矩阵形状：", oh_samples.shape)
# 可以通过 inverse_transform 将编码结果还原回原始类别
print("还原回原始数据：\n", one_hot.inverse_transform(oh_samples))

output:

独热编码后的结果：
 [[1. 0. 1. 0. 0. 1. 0. 0. 0.]
 [0. 1. 0. 1. 0. 0. 1. 0. 0.]
 [1. 0. 0. 0. 1. 0. 0. 1. 0.]
 [0. 1. 1. 0. 0. 0. 0. 0. 1.]]
编码后矩阵形状： (4, 9)
还原回原始数据：
 [[1 3 2]
 [7 5 4]
 [1 8 6]
 [7 3 9]]

Here we have three categorical features. OneHotEncoder detects distinct values in each column and creates binary indicators. The output oh_samples is a 4×9 matrix: the first three columns encode the first feature's values {1, 7}, the next three encode the second feature's values {3, 5, 8}, and the final three encode the third feature's values {2, 4, 6, 9} (only as many columns as categories actually present). Each row has exactly a few 1s and the rest 0s. For example, the original row [1, 3, 2] might encode as [1, 0, 1, 0, 0, 1, 0, 0, 0], meaning: first feature is 1 (columns 1–2), second is 3 (columns 3–5), third is 2 (columns 6–9). inverse_transform confirms correctness by recovering the original categories.

One-hot encoding is lossless and reversible. The downside is dimensionality explosion, especially with many categories, creating sparse 0/1 features and increasing computation and storage. For most linear and tree models, however, this is standard practice. If a feature has many categories, alternative methods like target encoding or dimensionality reduction may apply, but that is beyond this article's scope.

Label Encoding

Label encoding maps each category to a single integer—for example, "Beijing, Shanghai, Guangzhou" become 0, 1, 2. This avoids dimensionality increase and converts categories to numbers directly. However, the encoded integers have no inherent ordering. Without care, algorithms sensitive to magnitude—such as linear regression or SVM—may mistakenly treat category codes as ordered information. For unordered categories like colors or cities, one-hot encoding is usually safer than raw label codes.

Label encoding suits ordinal features with intrinsic order (education: high school=0, associate=1, bachelor=2, master=3) or encoding target labels (binary classification: positive=1, negative=0). In preprocessing pipelines, label encoding often converts text categories to numbers before passing them to one-hot encoding or other algorithms.

This example demonstrates encoding and decoding with LabelEncoder:

import numpy as np
from sklearn.preprocessing import LabelEncoder

raw_labels = np.array(['lv', 'ee', 'lth', 'ee', 'tt', 'lv'])
label_encoder = LabelEncoder()
encoded_labels = label_encoder.fit_transform(raw_labels)
print("标签编码结果：", encoded_labels)
print("还原回原始标签：", label_encoder.inverse_transform(encoded_labels))

output

标签编码结果： [2 0 1 0 3 2]
还原回原始标签： ['lv' 'ee' 'lth' 'ee' 'tt' 'lv']

The output encoded_labels might be [0 2 0 1 2 1], mapping 'audi' → 0, 'bmw' → 1, 'ford' → 2 (order depends on LabelEncoder's sorting or encounter order). inverse_transform confirms correctness and recovers the original strings.

Label encoding is straightforward but risky for unordered categories—it invents fake ordering. One-hot encoding is more universally safe but increases dimensions. Choose based on your specific use case.

Summary

Data preprocessing is indispensable. Removing invalid data, filling gaps thoughtfully, normalizing scales, and encoding categories correctly give your model clean, consistent input. Like laying a strong foundation before building a house, thorough preprocessing makes subsequent model training far more efficient and reliable. Different datasets and tasks demand different strategies, but the goal is always to make data faithfully express the problem and satisfy the model's assumptions.

© 2026 Yuxu Ge ·