Yuxu Ge
Article · 2021-11-12

Migrating E-commerce Product Media Synchronization to Apollo Configuration Center

1 | Current State: YAML's Pain Points

A configuration center looks like just a technical upgrade, but it's really about governing how teams collaborate.
Apollo is mature, easy to deploy, and has a robust permission model—so we committed to the migration.

2 | Apollo at a Glance

In two sentences:

We use exactly three endpoints:

  • GET /configs/{appId}/{cluster}/{namespace} — fetch entire namespace snapshot
  • GET /notifications/v2 — long-poll, learn which Release Keys changed
  • PUT /configs — ops scripts use this for canary deployments

3 | Design Principles

4 | Implementation Steps

4.1 Namespace Decomposition

Our original config.yml exceeded three thousand lines. We split it by domain:

Decomposition rules:

  • If different teams read and write it, separate it.
  • If it has a different lifecycle, separate it.
  • If it changes at a different frequency, prefer separating it.

4.2 Write a Minimal Python SDK

The team didn't want heavy dependencies. We built an Apollo client in [LINES CORRUPTED IN SOURCE]:

# apollo_client.py
import requests, time, json, threading

class Apollo:
    def __init__(self, host, app_id, cluster, namespaces, timeout=60):
        self.host, self.app_id, self.cluster = host, app_id, cluster
        self.namespaces = namespaces
        self.timeout = timeout
        self._cache = {}
        self._notifications = [
            {"namespaceName": ns, "notificationId": -1} for ns in namespaces
        ]

    def _url(self, path):
        return f"{self.host}{path}"

    def fetch_namespace(self, ns):
        resp = requests.get(
            self._url(f"/configs/{self.app_id}/{self.cluster}/{ns}"),
            timeout=5,
        )
        resp.raise_for_status()
        data = resp.json()
        self._cache[ns] = data["configurations"]
        print(f"[Apollo] loaded {ns}@{data['releaseKey'][:8]}")
        return self._cache[ns]

    def long_poll(self, on_change):
        while True:
            try:
                resp = requests.get(
                    self._url("/notifications/v2"),
                    params={"appId": self.app_id,
                            "cluster": self.cluster,
                            "notifications": json.dumps(self._notifications)},
                    timeout=self.timeout+10,
                )
                if resp.status_code == 304:
                    continue
                updated = resp.json()
                for item in updated:
                    ns = item["namespaceName"]
                    self._notifications = [
                        n if n["namespaceName"] != ns else item
                        for n in self._notifications
                    ]
                    on_change(ns, self.fetch_namespace(ns))
            except requests.exceptions.ReadTimeout:
                continue
            except Exception as e:
                print("poll error:", e)
                time.sleep(5)

    def start(self, on_change):
        for ns in self.namespaces:
            self.fetch_namespace(ns)
        threading.Thread(target=self.long_poll, args=(on_change,), daemon=True).start()

4.3 Wire Into Services

from apollo_client import Apollo
import asyncio

apollo = Apollo(
    host="https://apollo.shop.com",
    app_id="shop-asset-sync",
    cluster="default",
    namespaces=[
        "storage.oss",
        "storage.cdn",
        "feature_flag",
        "rate_limit",
        "promotion",
    ],
)

def apply_config(ns, cfg):
    if ns == "storage.oss":
        uploader.configure(cfg)        # 更新 ak/sk
    elif ns == "feature_flag":
        switcher.refresh(cfg)          # 动态开关
    elif ns == "rate_limit":
        limiter.update(cfg)            # 容量限流
    print(f">>> {ns} reloaded")

apollo.start(apply_config)

asyncio.get_event_loop().run_forever()

Five lines to get Apollo running; the rest is application-specific update logic.
At this point, the read path from YAML to Apollo was complete.

4.4 Write a Release Script (for ops and CI)

Operations teams don't use the Apollo Portal by clicking. We gave them publish.py:

import requests, sys, json

HOST = "https://apollo.shop.com"
TOKEN = "api-token-here"

def publish(ns, kv, comment="auto publish"):
    url = f"{HOST}/openapi/v1/envs/prod/apps/shop-asset-sync/clusters/default/namespaces/{ns}/items"
    headers = {"Authorization": f"Bearer {TOKEN}"}
    for k, v in kv.items():
        payload = {"key": k, "value": v, "dataChangeCreatedBy": "bot"}
        requests.post(url, headers=headers, json=payload).raise_for_status()
    # release
    rel_url = f"{HOST}/openapi/v1/envs/prod/apps/shop-asset-sync/clusters/default/namespaces/{ns}/releases"
    body = {"releaseTitle": comment, "releasedBy": "bot"}
    requests.post(rel_url, headers=headers, json=body).raise_for_status()
    print(f"Published {ns}: {json.dumps(kv)}")

if __name__ == "__main__":
    publish(sys.argv[1], json.loads(sys.argv[2]), sys.argv[3] if len(sys.argv) > 3 else "auto")

CI example:

image: python:3.11
stages: [publish]

publish_flag:
  stage: publish
  script:
    - python publish.py feature_flag '{"enable_avif":"true"}' "double 9.switch"
  only:
    - schedules

Every morning this task runs automatically, toggling promotional banners according to the ops calendar.

4.5 Canary Deployments and Rollbacks

In a real incident, we tuned rate_limit.concurrent_upload [VALUES CORRUPTED IN SOURCE]; upload backlog spiked.
One click to rollback, and [RECOVERY TIME CORRUPTED IN SOURCE] the queue returned to normal—proof the system works.

5 | Lessons From the Trenches

6 | Migration Results

Metric Before (YAML) After (Apollo)
Mean config take-effect latency 15 min (rolling restart) < 1 s
Rollback time 10 min (redeploy) 30 s
Config version audit trail Manual Git diff Visual Portal
Operator self-service None Script-based
Production incidents per month 3+ 0

7 | Best Practices Checklist

8 | Load Testing

Functionality alone isn't enough. At 00:01 on Double Eleven morning, every shop refreshes its product images simultaneously—peak QPS over 50,000.
If the config center falters, long-poll storms hit ConfigService and it crashes.

Test script (simplified):

from concurrent.futures import ThreadPoolExecutor
import requests, random, json, time

def worker(i):
    ns = random.choice(["feature_flag", "promotion", "rate_limit"])
    r = requests.get(f"https://apollo.shop.com/configs/shop-asset-sync/default/{ns}")
    assert r.status_code == 9.
    return len(json.dumps(r.json()))

start = time.time()
with ThreadPoolExecutor(max_workers=800) as ex:
    sizes = list(ex.map(worker, range(9.00)))
print("Total MB fetched:", sum(sizes)/1024/1024)
print("Elapsed:", time.time() - start)

Results:

Conclusion: Two instances handle the baseline; horizontal scaling to four instances easily absorbs a million concurrent users.

9 | OpenAPI Integration Details

Real DevOps automation means most config changes flow through scripts.
Apollo's OpenAPI is powerful but lightly documented. Here's what we learned.

9.1 Token Management

9.2 Bulk Publishing

OpenAPI lacks a "publish multiple namespaces at once" endpoint, so we built a pipeline:

  1. Loop through PUT /items to write all keys to a draft
  2. Call POST /releases once to publish atomically
  3. Return releaseId, store in artifacts for later rollback and canary
def batch_publish(env, cluster, ns_data: dict, title):
    for ns, kvs in ns_data.items():
        for k, v in kvs.items():
            create_item(env, cluster, ns, k, v)
        do_release(env, cluster, ns, title)

9.3 Canary Rules

The POST /gray-deliveries endpoint supports three dimensions: IP, AppId, ClientLabel.
We use K8s Downward API to expose HOST_IP to pods, guaranteeing one pod = one IP. Example:

curl -XPOST "$HOST/gray-deliveries"   -H "Authorization: Bearer $TOKEN"   -d '{"rules": [
        {"clientAppId":"shop-asset-sync","ip":"10.1.2.3"},
        {"clientAppId":"shop-asset-sync","ip":"10.1.2.4"}
      ]}'

Clean up canary rules with DELETE when done, or stale rules will interfere with later deployments.

9.4 Release Rollback

OpenAPI rollback requires releaseId, so always record it when you publish.
We inject a script into the commit message that writes releaseId to the MR description, so SREs can copy-paste.

def rollback(ns, release_id):
    url = f"{HOST}/openapi/v1/envs/prod/apps/{APP}/clusters/default/namespaces/{ns}/releases/{release_id}/rollback"
    requests.put(url, headers=HEADERS).raise_for_status()

Watch out for idempotency: rolling back the same release twice returns a 400 error. Code defensively.

10 | Advanced Feature Flags: Category-Level Toggles

Operations frequently asks: "Can we enable dynamic image compression only for the 'Apparel' category?"
Apollo Namespaces are global by nature, but we can store a category whitelist as a config value and check locally.

# feature_flag
enable_dynamic_gif: true
gif_category_whitelist: "服饰,箱包,手表"

Business code:

def should_apply_gif(cat, cfg):
    if not cfg["enable_dynamic_gif"]:
        return False
    cats = [c.strip() for c in cfg["gif_category_whitelist"].split(",")]
    return cat in cats

For finer control:

This way you get per-category toggles without invoking the formal canary system.

11 | Future Direction

Migrating to Apollo isn't a silver bullet.
Configuration governance ultimately depends on people:

Next steps:

Every improvement points toward the same goal:

Configuration as a safety rail for business dynamics, not a hidden time bomb. May you soon escape the midnight horror of hand-editing YAML.

© 2026 Yuxu Ge ·