Yuxu Ge
Article · 2022-04-25

State Synchronization Defect: Technical Reflection

Root Cause

The problem was straightforward in hindsight: batch operations updated state without recording the corresponding log entries. The fundamental issue was code that ignored consistency with single-item operations. When the bulk freeze feature was added, the developer updated the main table directly in bulk rather than reusing the single-freeze logic. Since single-item freezing included a separate logging step, that step was skipped in the batch path.

This is a classic instance of the double-write inconsistency problem in distributed systems. When state changes must be written to two places—here, the main state table and the operation log table—without atomic guarantees or strict consistency procedures, one side can update while the other doesn't. Different components reading from different sources then see different state.

The system relied on a log table rather than direct table scans for performance and loose coupling. Scanning the main table for changes at scale is expensive compared to consuming incremental log entries. The log table also serves an audit function, recording who modified which resources when. This dual role makes the log a source of truth in many system designs. But that design imposes a requirement: every state change must update both main data and log consistently.

In this case, a specific oversight caused the problem: batch operation code omitted log writes. State changed in the main table but went unrecorded in the log, so components depending on the log never learned of the freeze.

Repair

Once the root cause was identified, the fix had two parts:

We validated with multiple test rounds. First, we verified bulk operations updated both tables correctly. Next, we simulated sync tasks to confirm other modules properly received the frozen state. Once tests passed, we deployed the patch and resolved the production issue.

Common Risks in Log-Driven Sync

This case highlights typical problems and precautions in log-driven synchronization. In distributed systems, a common pattern is: one module produces a log (or event stream), others consume it to stay synchronized. Key considerations:

The log-driven pattern is a form of eventual consistency: components may diverge temporarily but reach agreement through log propagation. It's common in event sourcing, change subscriptions, and distributed cache refresh. But it demands rigor from developers: ensure every event is logged exactly once. A flawed log mechanism directly breaks state synchronization. Designing such systems requires thorough thinking and extensive testing.

Batch Operation Consistency Design

Batch operations introduce their own consistency challenges. Bulk updates affect many rows; without guaranteeing consistent success or failure, partial success becomes a risk. Combining this experience with common practice, batch operations should consider:

These principles help anticipate consistency problems during batch operation design, reducing incidents like this one.

Example: Python Batch State Update with Logging

To illustrate the problem and solution concretely, here's a simplified Python example. Two data structures: state_table holds current resource state, log_table records operations. States are "active" or "frozen".

First, a single freeze function and a naive batch freeze:

# 定义示例数据表
state_table = {
    "A": "active",
    "B": "active",
    "C": "active",
}
log_table = []  # 日志表开始时为空

# 单个资源冻结
def freeze_single(resource_id, operator):
    # 更新主状态表
    state_table[resource_id] = "frozen"
    # 记录操作日志
    log_entry = {
        "resource": resource_id,
        "action": "freeze",
        "operator": operator
    }
    log_table.append(log_entry)

# 批量冻结（初始版本，存在缺陷：没有记录日志）
def freeze_batch(resources, operator):
    for res in resources:
        state_table[res] = "frozen"
    # 忘记记录每个资源的冻结日志

Using these, we freeze resource A and batch-freeze B and C:

# 冻结单个资源A
freeze_single("A", operator="User1")
# 批量冻结资源B和C
freeze_batch(["B", "C"], operator="User1")

print(state_table)  # 查看主状态表
print(log_table)    # 查看日志表

Output:

{'A': 'frozen', 'B': 'frozen', 'C': 'frozen'}
[{'resource': 'A', 'action': 'freeze', 'operator': 'User1'}]

The main table shows A, B, C as frozen, but the log only records A. If other components consume the log to detect state changes, they'll know A froze but miss B and C. Other modules will treat B and C as still active—a consistency violation.

Now an improved batch freeze that logs each update and uses exception handling to simulate transaction rollback:

# 改进的批量冻结，增加日志记录和简单的事务机制
def freeze_batch_with_log(resources, operator):
    # 备份原始状态，以备回滚
    original_states = {}
    try:
        for res in resources:
            # 备份状态
            original_states[res] = state_table[res]
            # 更新状态
            state_table[res] = "frozen"
            # 写入日志
            log_entry = {
                "resource": res,
                "action": "freeze",
                "operator": operator
            }
            log_table.append(log_entry)
            # 模拟某种可能的错误，例如操作某个特殊资源出问题
            if res == "C":
                raise Exception(f"Failed to freeze {res}")  # 模拟错误
    except Exception as e:
        # 发生错误，回滚之前的更新
        for res in resources:
            if res in original_states:
                state_table[res] = original_states[res]  # 恢复原状态
        # 回滚日志（简单起见，将此次批量中的日志移除）
        log_table[:] = [entry for entry in log_table if entry["resource"] not in resources]
        print("Error during batch operation:", e)

This function freezes each resource and logs it. If an error occurs, the exception handler restores already-processed resources to their original state and removes their log entries, making the entire operation "never happened" (atomicity). Here we throw an exception during C's freeze to simulate mid-operation failure.

Reset and try the improved function on B and C:

# 重置状态
state_table = {"A": "frozen", "B": "active", "C": "active"}
log_table = [{"resource": "A", "action": "freeze", "operator": "User1"}]

# 尝试批量冻结B和C，期间模拟出错
freeze_batch_with_log(["B", "C"], operator="User1")

print(state_table)  # 查看主状态表
print(log_table)    # 查看日志表

Output might be:

Error during batch operation: Failed to freeze C
{'A': 'frozen', 'B': 'active', 'C': 'active'}
[{'resource': 'A', 'action': 'freeze', 'operator': 'User1'}]

Despite the exception during C's freeze, the rollback restores B and C to "active" and removes any log residue. The batch operation has no final effect—either all items succeed or nothing changes. Production systems would use database transactions or more robust mechanisms; this Python version illustrates the principle.

This example shows why state and log must update together and why batch operations must be atomic. In distributed systems, these details determine consistency across components—small oversights create lasting problems.

Closing

State synchronization defects remind us that details matter. In distributed systems, data consistency is often harder than features. When adding new functionality (batch operations) or architectures (log-driven sync), think globally about how data flows and state updates, ensuring each step is sound.

Concretely: strengthen code review and testing, paying special attention to changes affecting consistency. In design, avoid scattering system state across multiple sources without unified validation. If you must separate data and logs for performance and decoupling, guarantee strong consistency or provide self-healing—periodic audits that catch and fix divergence.

Distributed systems trade consistency for availability and performance, but as this incident showed, investing thought upfront beats firefighting later.

© 2026 Yuxu Ge ·