Search Architecture Upgrade: From Browser Inference to Edge Computing

I packed a complete hybrid search engine into the browser in previous articles: BM25 keyword retrieval plus multilingual-e5-small ONNX inference for semantic expansion, Int8-quantized keyword vectors, Service Worker caching, zero backend dependencies.

Technically compelling. Then I opened the blog on my phone.

iOS Chrome went blank with a curt error: "Can't open this page."

Time to rethink the architecture. This post documents the migration plan—moving semantic search from the browser to Cloudflare Workers edge nodes.

The Fatal Problem with Browser-Side Model Inference

The root issue: clients must download ~156MB of resources to enable semantic search.

I attempted device detection as a workaround: skip model loading on mobile, use keyword search only.

function isMobileDevice() {
    if (navigator.userAgentData?.mobile) return true;
    if (/Android|iPhone|iPad|iPod|Opera Mini|IEMobile/i.test(navigator.userAgent)) return true;
    return window.innerWidth <= 768;
}

Three hard failures emerge:

1. iPadOS spoofs the macOS User Agent. Since iPadOS 13, Safari reports as macOS by default. The /iPad/ pattern never matches; navigator.userAgentData?.mobile returns false. iPads still attempt to load the model and crash.
2. Service Worker cache corruption. Once registered, a Service Worker persists. If model download completes only halfway, the SW caches incomplete data. On next visit, the SW returns corrupted cache, making the page unusable.
3. Mobile gets degraded search. Mobile users—60%+ of traffic—only ever get keyword search while desktop users have semantic search.

The pure browser approach, however technically interesting, is fragile and bloated. It sacrifices mobile usability for a deployment model.

Target Architecture: Client + Edge

Move the heaviest computation—model inference—from the browser to Cloudflare Workers edge nodes.

构建期 (node)                          运行时
┌─────────────────┐                   ┌──────────────────────┐
│ Markdown/Notebook│                   │ 客户端 (browser)      │
│       ↓          │                   │                      │
│ index-builder    │ → inverted.json → │ BM25 关键词搜索 (即时)  │
│       ↓          │   metadata.json   │       ↓               │
│ embed-builder    │                   │ fetch /api/semantic   │
│ (OpenAI API)     │                   │       ↓               │
│       ↓          │                   │ 融合排序 → 展示结果    │
│ upload → KV      │                   └──────────────────────┘
│                  │                   ┌──────────────────────┐
└─────────────────┘                   │ Cloudflare Worker     │
                                      │ 1. embed query        │
                                      │    (OpenAI, ~150ms)   │
                                      │ 2. KV 读取文档向量     │
                                      │    (edge cache, ~2ms) │
                                      │ 3. 余弦相似度 201×512  │
                                      │    (<1ms)             │
                                      │ 4. 返回 top-K         │
                                      └──────────────────────┘

Key design decisions:

• Model choice: OpenAI text-embedding-3-small (512 dimensions). Natively multilingual; handles Chinese and English without needing a separate multilingual-e5-small running in the browser.
• Vector storage: Cloudflare KV. 201 documents × 512 dimensions ≈ 800KB JSON, well under KV's 25MB per-value limit. No need for specialized vector databases like Vectorize—brute-force searching 201 vectors takes under 1ms.
• Document-level vectors only. No chunking. 201 full articles, 100% recall.
• Keep BM25 unchanged. Client-side keyword search remains untouched: instant response, zero fallback risk.

Build-Time Design: embed-builder.mjs

A new build script will run after index-builder.mjs, generating embeddings for each document and uploading them to KV.

Document Representation

Vectorize each article on title + tags + first 500 characters of content. This gives the model enough context to understand article topics. Titles and tags naturally carry more weight (shorter, higher information density).

Incremental Builds

To avoid re-calling the API on every build, the script maintains a local cache file. Strategy: hash the input text; skip if the hash hasn't changed.

#!/usr/bin/env node
/**
 * Embedding Builder
 * 为所有文档生成 OpenAI embedding，支持增量构建。
 * 输出: .cache/search-embeddings.json
 */
import fs from 'fs';
import path from 'path';
import crypto from 'crypto';

const CONFIG = {
    metadataFile: 'public/search-metadata.json',
    cacheFile: '.cache/openai_embeddings.json',
    outputFile: '.cache/search-embeddings.json',
    model: 'text-embedding-3-small',
    dimensions: 512,
    batchSize: 50,  // OpenAI 支持批量请求
};

function contentHash(text) {
    return crypto.createHash('sha256').update(text).digest('hex').slice(0, 16);
}

async function embedBatch(texts) {
    const response = await fetch('https://api.openai.com/v1/embeddings', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
        },
        body: JSON.stringify({
            input: texts,
            model: CONFIG.model,
            dimensions: CONFIG.dimensions,
        }),
    });
    const data = await response.json();
    return data.data.map(d => d.embedding);
}

async function main() {
    const metadata = JSON.parse(fs.readFileSync(CONFIG.metadataFile, 'utf-8'));

    // 加载缓存
    let cache = {};
    try { cache = JSON.parse(fs.readFileSync(CONFIG.cacheFile, 'utf-8')); } catch {}

    // 找出需要 embed 的文档
    const toEmbed = [];
    for (const [url, doc] of Object.entries(metadata)) {
        const text = [doc.title, doc.text || ''].join('\n').slice(0, 500);
        const hash = contentHash(text);

        if (cache[url]?.hash === hash) continue; // 内容未变，跳过
        toEmbed.push({ url, text, hash });
    }

    console.log(`[Embed] ${toEmbed.length} new/modified, ${Object.keys(metadata).length - toEmbed.length} cached`);

    // 批量调用 OpenAI
    for (let i = 0; i < toEmbed.length; i += CONFIG.batchSize) {
        const batch = toEmbed.slice(i, i + CONFIG.batchSize);
        const embeddings = await embedBatch(batch.map(d => d.text));

        batch.forEach((doc, j) => {
            cache[doc.url] = { hash: doc.hash, embedding: embeddings[j] };
        });
        console.log(`[Embed] Batch ${Math.floor(i / CONFIG.batchSize) + 1} done`);
    }

    // 输出：只包含当前 metadata 中存在的文档
    const output = {
        model: CONFIG.model,
        dimensions: CONFIG.dimensions,
        updated: new Date().toISOString(),
        documents: Object.entries(metadata).map(([url, doc]) => ({
            url,
            title: doc.title,
            date: doc.date,
            text: (doc.text || '').slice(0, 200),
            type: doc.type || 'article',
            cover: doc.cover || '',
            embedding: cache[url]?.embedding || null,
        })).filter(d => d.embedding),
    };

    fs.mkdirSync('.cache', { recursive: true });
    fs.writeFileSync(CONFIG.cacheFile, JSON.stringify(cache));
    fs.writeFileSync(CONFIG.outputFile, JSON.stringify(output));

    console.log(`[Embed] Output: ${output.documents.length} documents, ~${Math.round(JSON.stringify(output).length / 1024)}KB`);
}

main().catch(console.error);

Cost: Initial run on 201 documents, ~40,000 tokens, costs $0.0008. Incremental builds only process new or changed articles.

Uploading to KV

#!/usr/bin/env node
/**
 * 将 embeddings 上传到 Cloudflare KV
 * 使用 Cloudflare API 或 wrangler CLI
 */
import { execSync } from 'child_process';

// 方式一：wrangler CLI
execSync(
    `wrangler kv:key put --namespace-id=${process.env.CF_KV_NAMESPACE_ID} "search_embeddings" --path=.cache/search-embeddings.json`,
    { stdio: 'inherit' }
);

Integration into CI/CD is straightforward: trigger upload automatically after git push kicks off a build.

Worker-Side Design: /api/semantic-search

Add a new route to the existing embed-worker.js. It already has /api/embedding, /api/chat, /api/translate endpoints and a configured KV namespace.

// 新增路由：POST /api/semantic-search
if (path === "/api/semantic-search") {
    const { query, topK = 10 } = await request.json();

    if (!query || typeof query !== 'string') {
        return new Response(JSON.stringify({ error: 'Missing query' }), {
            status: 400,
            headers: { ...corsHeaders, "Content-Type": "application/json" },
        });
    }

    // 1. 查询向量化（OpenAI API，~150ms）
    const embResponse = await fetch("https://api.openai.com/v1/embeddings", {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
            "Authorization": `Bearer ${env.OPENAI_API_KEY}`,
        },
        body: JSON.stringify({
            input: query,
            model: "text-embedding-3-small",
            dimensions: 512,
        }),
    });
    const embData = await embResponse.json();
    const queryVec = embData.data[0].embedding;

    // 2. 从 KV 读取文档向量（edge 缓存，~2ms）
    const embeddings = await env.TRANSLATIONS.get("search_embeddings", { type: "json" });
    if (!embeddings || !embeddings.documents) {
        return new Response(JSON.stringify({ results: [] }), {
            headers: { ...corsHeaders, "Content-Type": "application/json" },
        });
    }

    // 3. 暴力余弦相似度（201×512，<1ms）
    const results = embeddings.documents.map(doc => {
        let dot = 0, normA = 0, normB = 0;
        const docVec = doc.embedding;
        for (let i = 0; i < 512; i++) {
            dot += queryVec[i] * docVec[i];
            normA += queryVec[i] * queryVec[i];
            normB += docVec[i] * docVec[i];
        }
        const score = dot / (Math.sqrt(normA) * Math.sqrt(normB));
        return { url: doc.url, title: doc.title, date: doc.date, text: doc.text,
                 type: doc.type, cover: doc.cover, score };
    });

    // 4. 排序，过滤低分，返回 top-K
    results.sort((a, b) => b.score - a.score);
    const topResults = results.slice(0, topK).filter(r => r.score > 0.3);

    return new Response(JSON.stringify({
        results: topResults.map(({ embedding, ...r }) => ({
            ...r, score: Math.round(r.score * 1000) / 1000,
        })),
    }), {
        headers: { ...corsHeaders, "Content-Type": "application/json" },
    });
}

Expected latency: OpenAI API ~150ms + KV read 2ms + cosine calculation <1ms = **200ms total**. Against the current approach's 5–20 second model download on first load, this is transformative.

Client-Side Simplification

The migration's best part: delete a lot of code.

Deletion Checklist

Expected client resources: 156MB → 2.6MB, a reduction of 98.3%.

Simplified search-client.js

After migration, init() no longer loads vectors or models. semanticSearch() becomes a simple fetch() call:

export class SearchClient {
    constructor(options = {}) {
        this.semanticSearchUrl = options.semanticSearchUrl || '/api/semantic-search';
        this.enableSemantic = options.enableSemantic !== false;
        // 不再需要：keywordVectors, bgeExtractor, modelReady, vectorsReady
    }

    async init() {
        // P0：只加载倒排索引 + 元数据（~2.6MB）
        // 不再有 P1（向量）和 P2（模型）
        const [invertedData, metadataData] = await Promise.all([
            fetch('/public/search-inverted.json').then(r => r.json()),
            fetch('/public/search-metadata.json').then(r => r.json()),
        ]);
        this.invertedIndex = invertedData;
        this.metadata = metadataData;
        this.ready = true;
    }

    // BM25 关键词搜索：完全不变
    keywordSearch(query, limit) { /* ... 原有代码 ... */ }

    // 语义搜索：从 ONNX 推理变成 API 调用
    async semanticSearch(query, limit = 10) {
        if (!this.enableSemantic) return [];
        try {
            const res = await fetch(this.semanticSearchUrl, {
                method: 'POST',
                headers: { 'Content-Type': 'application/json' },
                body: JSON.stringify({ query, topK: limit }),
            });
            if (!res.ok) return [];
            const data = await res.json();
            return (data.results || []).map(r => ({
                ...r, sources: ['ai'],
            }));
        } catch (err) {
            console.warn('[Search] Semantic API failed:', err.message);
            return []; // 优雅降级：语义搜索失败不影响关键词结果
        }
    }

    // 混合搜索：融合逻辑不变（0.6 keyword + 0.4 semantic）
    async search(query, limit = 5) {
        const [kwResults, semResults] = await Promise.all([
            Promise.resolve(this.keywordSearch(query, limit * 3)),
            this.semanticSearch(query, limit * 3),
        ]);
        return this.fuseResults(kwResults, semResults, limit);
    }
}

Cleaned sidebar.js

// 删除：Service Worker 注册
// if ('serviceWorker' in navigator) {
//     navigator.serviceWorker.register('/components/sw.js').catch(() => {});
// }

// 删除：isMobileDevice() 函数
// 删除：模型加载 spinner UI
// 删除：onModelProgress / onModelReady 回调

// 简化：SearchClient 初始化，不再区分移动端
searchClient = new SearchClient({
    enableSemantic: sidebarConfig.semanticSearch !== false,
    semanticSearchUrl: sidebarConfig.semanticSearchApi || '/api/semantic-search',
    // 不再需要 mobile check，服务端搜索对所有设备一视同仁
});

Performance and Cost Projection

Subsequent searches rise from 50ms to 200ms—the only real regression. That 150ms difference is imperceptible to users, and we gain cross-platform stability and near-zero maintenance.

Migration Path

Four backward-compatible steps:

Step 1: Build Pipeline (no impact on live site)

• Add _tools/embed-builder.mjs to generate document vectors
• Add _tools/upload-embeddings.mjs to upload to KV
• Update package.json build script for search

Step 2: Worker Endpoint (no impact on existing functions)

• Add /api/semantic-search route to embed-worker.js
• Deploy and verify with curl

Step 3: Client Switch

• Update search-client.js: semantic search switches from local inference to API call
• Update sidebar.js: remove Service Worker registration and device detection
• Update sidebar-config.json: add semanticSearchApi

Step 4: Cleanup

• Delete lib/transformers/, components/sw.js, public/keywords_vectors.bin
• Delete public/search-vocab.json, _tools/vector-builder.mjs
• Remove @huggingface/transformers from devDependencies

Architecture Review: Expected Trade-Offs

Gains

Costs

Deliberate Departures

From token-level semantic expansion to document-level vector search

The previous version's key optimization was "embed the vocabulary, not documents"—vectorize 8,000 keywords, then find similar keywords at query time and apply BM25. In a browser-constrained environment, this was clever: one model inference pass (for the query), then pure CPU-bound Int8 dot products.

On the server side, this indirection becomes unnecessary. Direct document embedding plus cosine similarity is simpler and should perform just as well (OpenAI's model quality compensates for granularity). The 8,000 keyword vectors, KVEC binary format, and Int8 quantization complete their useful life.

From multilingual-e5-small to text-embedding-3-small

Both support multiple languages, but OpenAI's model performs better on cross-lingual Chinese–English matching and requires no model file management. The cost is one API call per query; for a personal blog's search frequency, this is negligible.

Summary

This migration is a pragmatic trade-off:

• Abandon "zero backend" for cross-platform search
• Abandon offline semantic search for 98% resource reduction and a leap from 20 seconds to 200ms on first search
• Abandon the technical elegance of token-level expansion for a simpler, maintainable architecture

Browser ONNX inference proved semantic search is technically feasible in the browser. Feasible does not mean optimal. For a live blog, moving heavy computation to edge nodes and keeping the client lean is the more practical choice.

The remaining work is implementation. If all goes well, blog search will run stably across any device—keyword results instantly, semantic results in 200ms, no crashes, no 150MB wait. That is the search experience worth building.