Self-Training Small RAG Models via Retrieval Outcome Signals

This project investigates a way to reduce manual preference labelling: the same model answers each question twice, once with retrieved documents and once without them, and the stronger answer becomes a potential training signal.

This is my MSc dissertation at the University of York. The research is still in progress, so this page explains the question and experimental design rather than claiming a result in advance.

Research background

Preference training normally needs people to compare two answers and choose the better one. That produces useful data, but it is slow and expensive.

RAG systems already run a small experiment every time they answer a question: they retrieve documents and then generate an answer from them. If those documents clearly improve the answer, perhaps that improvement can supply part of the preference data automatically.

The central research challenge is to establish when retrieval genuinely improved the answer, rather than merely changing its length, style, or score by chance.

Research role and scope

I designed and am conducting this MSc research independently. The work covers the research hypothesis, experimental design, data construction, and evaluation:

I have:

• proposed using the difference between answers with and without retrieved evidence to construct preference data;
• designed a controlled comparison in which retrieval context is the only intended change;
• defined filtering rules that retain only clear answer differences and reduce misleading training signals;
• built baselines, cross-domain tests, and experiment records to separate a training effect from retrieval quality or random variation;
• identified risks such as answer-length bias, scorer bias, and data leakage and included them in the evaluation plan.

The focus is to translate the research hypothesis into a reproducible experimental process and an evidence base strong enough to support or reject it.

Experimental design

For every question, I keep the model and generation settings unchanged and create two answers:

1. Answer A: the model answers without retrieved documents.
2. Answer B: the model sees retrieved documents before answering.

I then compare the answers. Depending on the dataset, the comparison can ask:

• Is the final answer correct?
• Is it supported by the retrieved text?
• Did the model add claims that the evidence does not support?
• Is the difference large enough to trust, or are the answers effectively tied?

Only clear wins become preference pairs. Ambiguous examples are discarded. The retained pairs can be used for DPO, where the model learns to prefer one kind of answer over another.

Controlling the retriever

If I change the retriever and the generator at the same time, I cannot tell why the result changed. Better documents might make the model look better even when preference training did nothing.

Keeping the retriever fixed gives the experiment a clean boundary: the retrieved evidence stays comparable, while I measure what changed in the answer model.

Main risks

Retrieval does not always help. It may return an irrelevant passage or a passage that conflicts with another source. A RAG answer can also look more convincing simply because it is longer.

The main risks are therefore:

• training on noisy or incorrect preferences;
• rewarding long answers instead of good answers;
• letting test data leak into the training process;
• teaching the model to repeat mistakes made by the scorer;
• seeing an improvement on one dataset that disappears elsewhere.

These risks are why the project includes filtering, baseline comparisons, fixed random seeds, and a separate cross-domain evaluation.

Reproducibility and traceability

For each result, I record the corpus version, retrieved passages, prompt, generation settings, scoring method, and model checkpoint. If a score changes, I should be able to trace the exact path that produced it.

This is less glamorous than training the model, but it matters. Without a traceable experiment pipeline, a small configuration change can be mistaken for a research finding.

Research objective and expected value

I am not trying to prove that automatic signals can replace people completely. The more useful question is narrower: under what conditions are retrieval outcomes reliable enough to reduce the amount of human preference labelling?

Even a partial answer would be valuable. It could show where automatic preference construction is safe, where human review is still necessary, and how small RAG systems can improve with a limited training budget.

If the method proves useful, it would offer resource-constrained teams a more controlled improvement loop: the system could identify clear cases first, while human review focuses on difficult or disputed examples.