Letting an Agent Tune Search: Designing a Boundary You Can Verify

Code: https://github.com/geyuxu/searchops-agent-lab

1. The question

Many agent projects stall at the same place: the agent can propose, but nobody dares let it act.

The reason is usually not that the model is not smart enough. It is that nothing makes letting go safe. "Please be careful" in a prompt is not a guarantee, and a model's self-reported confidence is not an admission criterion. So the agent stays a demo: its suggestions all look right, and nobody hands it the publish button.

Search operations is a good place to test this. It is high-frequency, judgement-heavy, and reversible-but-risky: find poor queries, diagnose causes, adjust synonyms and field weights, roll out, roll back if needed. I ran that loop by hand for years on an e-commerce platform.

So I built a system to answer one question: can an LLM agent safely operate search relevance tuning?

The answer is not "yes" or "no" — it is two substitutions:

• Replace trust in the model with a verifiable evaluation gate
• Replace prompt-level constraints with a structural permission boundary

The data is the public Amazon Shopping Queries dataset (ESCI, Apache-2.0). Product titles, brands, descriptions, search queries and human relevance labels are real public data. Prices, inventory, users and orders are deterministic simulations derived from a hash of the product ID — not Amazon transaction data.

First, what the queries actually look like

"Improve retrieval quality" means nothing in the abstract. These are real queries from the dataset, copied verbatim:

!awnmower tires without rims             leading punctuation, and lawnmower lost its first letter
# 2 pencils not sharpened                numbering convention + negation
1 1/2 leather belts without buckle       fractional size + negation
03 durango front calipers without pads   abbreviated model year + negation
#1 rated resveratrol supplement without tea leaves

Two features run through the whole set: noise (leading punctuation, misspellings, abbreviations) and negation (without / not / no).

Negation is the dangerous one, because it is the easiest thing for an "optimisation" to break — drop the without from fence without holes and you retrieve a page of fences with holes. Any rewriting scheme has to prove it does not make that mistake before anything else.

2. High-level design

Four layers, top to bottom:

Three principles run through all of it, and each replaces a promise with a mechanism:

The agent cannot publish. Not "we ask it not to" — those capabilities are simply absent from its tool registry.

Degradation is typed, not silent. When a model is unavailable, slow, or returns something unusable, search still succeeds on BM25 — but the response carries why.

Reranking cannot lose documents. The merge operates on candidate indices, and the result is provably a permutation of the input.

3. Detailed design

3.1 Permission boundary: make overreach impossible at construction time

Safety classes are not a documentation convention. They are metadata registered alongside each client method:

class SafetyClass(enum.Enum):
    READ = 1               # read-only query, no side effects
    DRY_RUN = 2            # computes and returns a comparison, writes nothing
    GOVERNED_WRITE = 3     # writes state; requires actor / request_id / idempotency key
    PRIVILEGED_WRITE = 4   # grants approval, issues an approval_token
    TOKEN_GATED_WRITE = 5  # changes the live strategy; additionally requires a valid token

#: highest class an automated proposer may hold
MAX_AUTOMATED = SafetyClass.GOVERNED_WRITE

The important part is the default — an unregistered method is treated as maximally dangerous:

def safety_class_of(fn) -> SafetyClass:
    """Return the registered class; unregistered means most dangerous — fail closed, not open."""
    return getattr(fn, "safety_class", SafetyClass.TOKEN_GATED_WRITE)

The registry is built with a double filter — allowlist and safety class — and either failure raises at construction time rather than at runtime:

def build_registry(client) -> dict[str, Tool]:
    registry = {}
    for name in ALLOWED:                       # allowlist: new methods are invisible by default
        if name in FORBIDDEN:
            raise GovernanceViolation(f"{name} appears in both the allowlist and the deny list")
        level = safety_class_of(getattr(type(client), name))
        if level.value > MAX_AUTOMATED.value:  # class: over the ceiling is refused
            raise GovernanceViolation(
                f"{name} has safety class {level.name}, above the automation ceiling {MAX_AUTOMATED.name}")
        registry[name] = Tool(name=name, fn=getattr(client, name), safety_class=level)
    return registry

The allowlist and deny list are redundant on purpose, so that overreach shows up immediately in a test:

def test_registry_rejects_privileged_method_added_to_allowlist(monkeypatch):
    monkeypatch.setattr("searchops_agent.tools.ALLOWED", ALLOWED + ("publish",))
    with pytest.raises(GovernanceViolation):
        build_registry(SearchOpsClient(base_url="http://127.0.0.1:9"))

This test protects a property that decays over time. Adding a method to an allowlist is easy; adding one that happens to be dangerous is just as easy. Making it a red build is more useful than writing it in a document.

3.2 The proposal loop: the agent must be able to substantiate itself

There is an easily missed gap here. In the first implementation the agent could dry-run a single query, but could not run a full offline evaluation — because the evaluation service always ran against the currently published strategy.

Two consequences:

1. Sweeping field weights meant actually publishing and rolling back dozens of times, polluting version history and the audit trail.
2. The agent could not substantiate a proposal. It could only say "I think this is better", with no evidence.

So the evaluation request gained an optional candidate-configuration field. When present, that configuration compiles the query and a full evaluation runs — but it is forced not to persist, and it neither reads nor writes any strategy version:

public record EvaluationRequest(
        List<EvaluationQuery> queries,
        int k,
        boolean persist,
        @JsonProperty("use_ai") Boolean useAi,
        // Candidate strategy configuration (optional). When present it compiles the retrieval
        // query, evaluating a configuration that has not been published. When absent (null) the
        // behaviour is byte-identical to before this field existed — evaluate the live strategy.
        @JsonProperty("strategy_config") @Valid StrategyConfig strategyConfig) {

A trap I walked into. This field was originally a primitive boolean. Jackson 3 flipped the default of FAIL_ON_NULL_FOR_PRIMITIVES from false to true, and Spring Boot 4.1 does not restore Jackson 2 semantics. The existing evaluation script — whose payload does not contain the key — started getting 400s, breaking the entire baseline-reproduction chain. Worse, the running image showed no symptom; it only broke on rebuild. Lesson: new optional fields always take a boxed type.

The response marks a candidate run with strategy_version = -1 and strategy_source = "candidate". The way to verify it actually works is to replicate the published strategy's weights through the candidate path — the metrics must come out bit-identical. If they differ, the two paths compile differently:

published strategy v7        v= 7   NDCG@10 = 0.3524
candidate = replica of v7    v=-1   NDCG@10 = 0.3524   ← identical
candidate = title de-weighted v=-1  NDCG@10 = 0.1544   ← genuinely applied

3.3 A degradation state machine: separating "AI didn't help" from "AI never ran"

In the early implementation, ai_applied was the only observable signal. "AI is disabled", "the request didn't ask for AI", "the call timed out" and "the adapter returned something unusable" all collapsed into the same false.

That produces a class of failure you cannot investigate: after wiring up a real model, the metrics do not move. It looks like "AI has no benefit". In fact it never ran once.

I hit exactly that. The Java client carried a leftover assertion:

// Before: any non-mock provider threw, and the catch below silently degraded to BM25
if (response == null || !"mock".equals(response.path("provider").asText()))
    throw new IllegalStateException("AI adapter returned an invalid provider response");

Meanwhile the handoff document promised, in writing, that swapping in a real AI service required no change to the search backend. That sentence was false because of this one line — and the failure was silent: HTTP 200, normal results, AI simply never in effect.

The fix has two parts. The admission criterion changed from "what is the provider called" to "does the response contain a usable rewrite":

var rewritten = response == null ? "" : response.path("rewritten_query").asText("");
if (rewritten.isBlank()) {
    throw new InvalidAiResponseException("AI adapter response is missing a usable rewritten_query");
}

Then typed status:

public enum AiRewriteStatus {
    APPLIED,           // call succeeded and the rewritten query genuinely differs
    NO_CHANGE,         // call succeeded but the adapter returned the query unchanged
    NOT_REQUESTED,     // the request did not ask for AI. Not a fault
    DISABLED,          // global kill switch is off. Not a fault
    TIMEOUT,           // connect or read timeout
    TRANSPORT_ERROR,   // other network failure, or a 4xx/5xx
    INVALID_RESPONSE;  // missing body, malformed JSON, or no usable result

    public boolean fallback() { return this != APPLIED && this != NO_CHANGE; }
    public boolean failure()  { return this == TIMEOUT || this == TRANSPORT_ERROR
                                    || this == INVALID_RESPONSE; }
}

ai_applied was corrected at the same time: it now means "the query was rewritten", not "the call did not throw". Under the old meaning, a mock provider that matched no rule still returned true, which made the field useless for deciding whether AI affected retrieval at all.

Reranking later got its own state machine rather than reusing this one. A single request can legitimately be "rewrite applied, rerank timed out"; collapsing both into one field loses which stage failed — the same collapse that forced this design in the first place.

All seven states have been observed on the real path: pointing at a closed port yields TRANSPORT_ERROR, an upstream 500 yields TRANSPORT_ERROR, a 200 missing the field yields INVALID_RESPONSE, a slow response yields TIMEOUT.

Amusingly, TRANSPORT_ERROR was initially impossible to trigger: stopping the container meant Docker's network dropped packets, producing a connect timeout classified as TIMEOUT. After moving the applications to host processes, pointing at a closed port returns ECONNREFUSED immediately — and that path became testable for the first time.

3.4 The rerank invariant: never lose a document

Reranking asks the model to reorder the N candidates BM25 retrieved. The model may return too few, too many, IDs that do not exist, or the same ID twice.

So the merge works on indices, not IDs: build a queue of indices, pop one each time the model names it; ignore out-of-set IDs and repeats; in a second pass append every index not yet taken, in original order. Each index can be taken at most once, so a single length check at the end is enough to prove the result is a permutation of the input.

Why this is non-negotiable: if reranking could make a document disappear, one model hallucination becomes a recall regression whose shape is identical to "BM25 didn't retrieve it" — and the investigation goes to the index and the query compiler, never to the model.

A violation throws IllegalStateException and is classified separately as INTERNAL_ERROR rather than folded into INVALID_RESPONSE. Recording your own defect as "the model returned something illegal" sends the investigation in the wrong direction.

The model also returns candidate numbers, not product IDs. Three reasons: an ASIN costs roughly 5–7 tokens against 1–2 for a number; out-of-set detection becomes a range check instead of string comparison; and it is a structural injection defence — product titles are untrusted external data, and under this protocol the only thing the model can emit is a sequence validated as a permutation of 1..N. A successful injection buys nothing but a different ordering.

3.5 The gate: a higher mean is not enough

The statistics are deliberately conservative. Per-query metrics make paired testing possible — the same queries under two strategies are paired samples, not independent ones:

def paired_bootstrap(baseline, candidate, *, iterations=10_000, seed=20260816):
    """Bootstrap the paired differences. Resampling the *query* preserves the pairing."""
    diff = candidate - baseline
    rng = np.random.default_rng(seed)
    idx = rng.integers(0, diff.size, size=(iterations, diff.size))
    means = diff[idx].mean(axis=1)
    return float(diff.mean()), float(np.percentile(means, 2.5)), float(np.percentile(means, 97.5))


def permutation_test(baseline, candidate, *, iterations=10_000, seed=20260816):
    """Paired permutation test: flip the sign of each query's difference to build the null."""
    diff = candidate - baseline
    observed = abs(diff.mean())
    rng = np.random.default_rng(seed)
    signs = rng.choice((-1.0, 1.0), size=(iterations, diff.size))
    null = np.abs((diff * signs).mean(axis=1))
    # +1 smoothing: avoid reporting p=0, an impossible precision at finite iterations
    return float((np.count_nonzero(null >= observed) + 1) / (iterations + 1))

The gate itself is fail-closed. A higher mean is nowhere near sufficient:

if policy.require_significant:
    if primary.delta <= 0:
        promote = False
    elif not survives[primary.metric]:      # still significant after BH correction
        promote = False

Beyond the primary metric it also requires that guard metrics do not degrade significantly, that the zero-result rate does not rise, and that the share of catastrophically degraded queries stays under a ceiling — because a mean can hide a collapsing tail.

The tooling also carries a null-hypothesis self-check: comparing a dataset against itself must return "not significant" on all four metrics. That guards against the hardest error to notice — a test with a systematic bias.

4. Demo: one full governance cycle

Everything above is mechanism. Here is what it looks like running, with a fixed query: laptop stand.

What the operator sees

Live metrics on the left, the currently published strategy on the right (version, status, full config JSON). The three panels below are popular queries, the zero-result queue and the low-NDCG queue — and that last one holds real ESCI queries: !awnmower tires without rims at 0.000, # 2 pencils not sharpened at 0.382. This is where the agent's diagnostic evidence comes from.

Baseline: v7

The top three are a projector stand, a laptop backpack and office stickers. Frankly not good — which is exactly the thing worth improving.

Publishing a harmful change

Drop title from 4.0 to 0.2, raise description to 5.0, and walk it through draft → submit → approve → publish:

Position two is now an action figure doll. That is what a bad strategy change looks like — and it is already live for every user.

Rollback

Roll back to v7 using the approval token issued at publication time:

The results match the baseline screenshot item for item. That is the entire reason rollback exists.

The audit ledger

Every transition records actor, request id, action, idempotency key, before/after version and outcome.

Rows 16 and 17 deserve to be singled out:

ID 17  searchops-agent   STRATEGY_SUBMIT    7 -> 8
ID 16  searchops-agent   STRATEGY_CREATE    7 -> 8

The agent appears exactly twice in the whole ledger, and only as CREATE and SUBMIT. Every APPROVE / PUBLISH / ROLLBACK belongs to a human role. The "structural permission boundary" described earlier ends up as this fact in the audit log — not because a prompt asked the agent to restrain itself, but because those capabilities were never in its tool registry.

One incidental confirmation during the demo: I tried to re-approve an already published strategy in order to obtain a rollback token, and the service returned 409 Conflict. The state machine refused an illegal transition — not an obstacle, precisely its job.

5. Results

Diagnose first, then decide where to invest.

For every query in the baseline with NDCG@10 = 0 and Recall@10 = 0, I traced the true rank of the relevant products:

rank 11–50            30.3%   ← reranking can fix this
rank 51–200           27.3%   ← deeper reranking can fix this
beyond 200 / missed   42.4%   ← only rewriting can fix this

57.6% of total failures are ranking problems, not retrieval problems. Concretely:

query                                         true rank of the relevant product
1 1/2 leather belts without buckle            rank 11
1 ml medical grade syringes without needle    rank 11
08 chevy tailgate without emblem              rank 12
#15 charm                                     rank 14
03 durango front calipers without pads        rank 16

The product is right there, a few positions short of the top ten. Rewriting cannot fix this class — changing the query only changes the candidate set, and the correct answer had already been retrieved.

That redirected AI investment from rewriting to reranking.

In parallel, a 218-configuration sweep of field weights confirmed that conventional tuning had exhausted its headroom (+0.0035 on the holdout, p=0.1202, not significant) — which removes the "you could have just tuned it" explanation.

The sweep also turned up two things no amount of tuning would have solved:

The category field is informationally dead. Eighteen weight settings — including deleting the field entirely — produced bit-identical metrics. Querying the index explained it: a terms aggregation on category.keyword returns exactly one bucket, Other, with doc_count=20000. Every document, 100% document frequency, BM25 IDF ≈ 0. The field had been carrying a weight of 1.2 into scoring for exactly zero benefit.

Scaling weights uniformly is a no-op. Raising title from 4.0 to 8.0 left the metrics identical to ten decimal places; dropping it to 0.1 moved them to 0.2123. With multi_match/best_fields and no tie_breaker, score = max_f(w_f · bm25_f), so multiplying the whole weight vector by any positive constant rescales scores without reordering anything. Before noticing this, a large slice of the search space was duplicates of itself.

The holdout is a hash-deterministic split with a fixed seed, train 1400 / holdout 600, sized from a measured power analysis. Final results:

NDCG@10 improved 25.6% relative, with all four metrics significant at all three depths and surviving multiple-comparison correction.

The control is query rewriting under the same model, same evaluation, same queries: Δ+0.0025, p=0.5942, not significant, gate BLOCK.

That contrast matters more than the number. It does not say "things improved because we used an LLM". It says things improved because we used it in the right place — and where that was came from the diagnosis, not from guessing.

What actually happened

Reranking pulls judged-relevant products up into the top ten (spot check, depth 50; brackets are the ranks of judged-relevant products):

#1 rated resveratrol supplement without tea leaves    [2,4,9]  →  [1,2,3]
$13 bb guns without a yellow tube                     [50]     →  [5]
+dark chocolate peanuts covered not milk              [1,8,9,10,28] → [4,5,12,13,14]   ← worse

The third is a counter-example, and exactly the predicted kind: the model read not milk too aggressively and pushed two judged-relevant products out of the top ten. The gain is not monotonic.

For rewriting, the wins and losses are two sides of the same behaviour — the model wins when it corrects a brand spelling and loses when it guesses an unfamiliar token into a real brand:

win    rockshocks front fork  →  rockshox front fork      NDCG@10 +0.826
       troybuilt              →  troy bilt                          +0.569
       zenphone 6             →  asus zenphone 6                    +0.485

loss   k cliffs ... backpack  →  guessed into a real but unrelated brand    large drop

Coverage also explains why rewriting measured as nothing overall: only 152 of 600 queries were actually rewritten (25.3%), and only 64 changed their NDCG@10 at all (37 up, 27 down). The remaining 536 contribute an identically zero difference, diluting whatever those 64 did.

One more detail in the depth effect: 20→50 buys +0.0239, while 50→100 buys only +0.0142. The diagnosis showed similar proportions of relevant products at ranks 11–50 and 51–200, so if the gain depended only on coverage the two steps should be comparable. They are not — deeper candidates are harder for the model to recognise, having lower literal overlap with the query.

6. Limits

These must travel with the result, or the number misleads.

The evaluation set is sparsely judged. Only about 12% of top-10 positions carry a human label, and each query averages 1.68 relevant judgements. Unjudged documents score zero — so a genuine improvement that surfaces a good but unlabelled product is scored as a regression.

That also explains why rewriting looked bad: its Recall@10 rose slightly while NDCG@10 fell slightly, the classic fingerprint of "the newly promoted documents are unjudged". So −0.0061 cannot be read as "rewriting showed users worse results".

Statistical power is limited. On the 600-query holdout the minimum detectable NDCG@10 difference is about 0.016. True effects smaller than that cannot be measured here.

Degradation biases the effect downward. Ten of 600 rerank queries degraded (timeout or transport error) and fell back to baseline values, so +0.1207 errs on the low side if at all.

Hosted models cannot be reproduced exactly. The model is pinned to a dated snapshot (qwen3.7-flash-2026-07-15) and recorded with each run, but vendor weight updates do not guarantee historical reproducibility.

Non-determinism survives temperature=0. Across two full runs, 6 of 200 queries differed, with a rerun noise band of roughly ±0.005. Any small-magnitude conclusion therefore needs repeated runs.

7. Code

• GitHub: https://github.com/geyuxu/searchops-agent-lab
• Dataset: amazon-science/esci-data (Apache-2.0)

experiments/ holds the split manifest, baselines, sweep logs and every measured artifact; agent/searchops_agent/eval/ holds the statistics and the gate; the permission boundary lives in agent/searchops_agent/tools.py and safety.py.

One command brings up the whole environment:

cd platform && make doctor && make bootstrap && make data && make up && make seed && make evaluate

For development, stateful infrastructure runs in containers and applications run as host processes:

make infra-up   # postgres / redis / elasticsearch only
make dev-info   # prints how to start each application locally

8. Back to the question

Can an agent safely operate search relevance tuning?

This system's answer: yes — provided its capability boundary is guaranteed by code structure and its output is adjudicated by statistical testing. Neither depends on trusting the model, and neither depends on the prompt.

Inside that boundary the agent is free to fail. Evaluating candidate strategies over and over pollutes nothing, and being blocked by the gate costs nothing. The only thing it cannot reach is the last step: pressing publish.