Yuxu Ge
Article · 2025-04-29

Building RAG Search Service Design with Spring AI

The system architecture uses a layered, modular design with these core components:

These components integrate through Spring Boot configuration and organization, ultimately providing a RESTful API service that includes document ingestion and question-answering query endpoints. The diagram below illustrates how system components interact:

(Architecture overview: Users submit requests through the Controller. RAG Service calls Hybrid Retriever, which queries both Elasticsearch and Qdrant separately to fetch documents. The embedding model generates vectors for Qdrant storage and retrieval. ChatClient wraps LLM model invocation (local or cloud). Retrieved relevant documents serve as prompt context, which the LLM uses to generate an answer returned to the user.)

Core Module Design

Vector Storage and Hybrid Retrieval Module

Qdrant Vector Store Configuration: The system uses Qdrant to persist document vectors and metadata. By adding the spring-ai-qdrant-store dependency, Spring AI automatically configures Qdrant-related beans. We need to provide the Qdrant client and VectorStore bean, for example:

@Bean
public QdrantClient qdrantClient() {
    // 构建连接到 Qdrant 的 gRPC 客户端
    QdrantGrpcClient.Builder grpcClientBuilder = 
        QdrantGrpcClient.newBuilder("qdrant-host", 6334, false);
    grpcClientBuilder.withApiKey("<QDRANT_API_KEY>");
    return new QdrantClient(grpcClientBuilder.build());
}

@Bean
public VectorStore vectorStore(QdrantClient client, EmbeddingModel embeddingModel) {
    return QdrantVectorStore.builder(client, embeddingModel)
            .collectionName("documents")       // 指定集合名称
            .initializeSchema(true)            // 若未预建集合则自动创建
            .build();
}

This configuration creates a Qdrant implementation of the VectorStore interface for storing vectors and performing similarity searches. The injected EmbeddingModel is used to generate vector representations when storing documents. Note: you can create collections in Qdrant beforehand, specifying vector dimensions and similarity metrics, or let the program auto-create them via initializeSchema(true) (by default using Cosine distance, with dimensions depending on the embedding model). Ensure the Qdrant instance is running and accessible, for example by launching a Qdrant container via Docker.

Elasticsearch Full-text Search Configuration: Elasticsearch stores text indexes for documents, enabling keyword-based retrieval. Integration can use Spring Data Elasticsearch or the REST API. Configuration requires the Elasticsearch address (e.g., localhost:9200), index name (e.g., documents), and index mapping (containing fields like document ID and content). For simplicity, leverage Spring Data Elasticsearch by defining a document entity and corresponding Repository. For example, define DocumentEntity(id, content, metadata) and establish full-text indexes.

Document Index Structure: To facilitate merging hybrid retrieval results, we assign each document segment a unique ID and share that ID across Qdrant and Elasticsearch as the primary key. When documents enter the system, they typically split into multiple chunks—each chunk is a text segment suited to retrieval and context length. We index each segment as an independent Document. During retrieval, whether we find a segment via vector or keyword search, we can identify its content by ID. Store document content in the Elasticsearch index, while Qdrant primarily stores vectors and essential metadata (such as document ID and source) as payload.

Hybrid Retriever Implementation: Create a hybrid retriever, for example HybridRetriever implements DocumentRetriever. Its retrieve(Query query) method performs these steps:

 List<Document> vectorDocs = vectorStore.similaritySearch(
       SearchRequest.builder()
           .query(userQuery)
           .topK(5)
           .build()
 );

This returns the 5 document segments most similar to the query.

Through HybridRetriever, we combine semantic and exact matching to maximize finding relevant content. For instance, when user queries employ different phrasing than document terminology, vector retrieval finds semantically similar content while keyword retrieval ensures exact matching terms are not missed.

LLM Inference and Switching Module

ChatClient and ChatModel Abstraction: Spring AI provides the ChatModel interface representing concrete large language models (such as OpenAI ChatGPT, Anthropic Claude, or local Llama), and ChatClient encapsulates a unified client for interacting with these models. This service injects a ChatClient instance to invoke the LLM for answer generation. Thanks to ChatClient's abstract design, we can flexibly swap the underlying ChatModel without changing business code. For example, we could switch from OpenAI's GPT-4 to a local Llama2 model through configuration changes.

1. OpenAI Model Integration: When using OpenAI's cloud service, add the spring-ai-openai dependency and provide OpenAI's API key and chosen model name in application configuration. For example, in application.yml:

spring:
  ai:
    openai:
      api-key: YOUR_OPENAI_API_KEY
      chat:
        model: gpt-4

This enables Spring AI's automatic configuration for OpenAI Chat API. The injected ChatClient will default to the configured OpenAI model. Through ChatClient's fluent API, we can conveniently send conversation messages and receive replies. For example:

ChatResponse response = chatClient.prompt()
    .user("请解释RAG的作用")
    .call()
    .chatResponse();
String answer = response.content();

This call sends the user message to the configured OpenAI model and retrieves an answer.

2. Local Ollama Model Integration: When adopting local LLM models, install and run the Ollama service. Ollama manages local model inference and interacts with the application via REST endpoints. After adding the spring-ai-ollama dependency, Spring AI provides OllamaChatModel and OllamaApi to connect to a local Ollama instance. Configuration-wise, specify the Ollama service address and model in application.yml, for example:

spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      default-model: llama2

Assuming you've downloaded a model named "llama2" via "ollama pull", Ollama will provide inference using that model. When the application starts, it automatically creates an API client connecting to Ollama (OllamaApi) and the corresponding ChatModel implementation. ChatClient can then use the local model for conversation.

When using Ollama, the call pattern resembles OpenAI, except the local service produces the result. For example:

ChatResponse response = chatClient.prompt()
    .system("你是资深Java助理")   // 可以设定初始系统提示
    .user("给出RAG搜索服务的优势")
    .call()
    .chatResponse();

ChatClient sends the request to the local Ollama and retrieves the model's reply.

3. Inference Mode Switching: To toggle between local and cloud models, use Spring Profiles or configuration switches. For instance, define an llm.mode configuration property (values: local or openai). Through conditional configuration, inject the Ollama ChatClient bean in local mode and the OpenAI ChatClient bean in openai mode. Spring AI's support for different ChatModels is out-of-the-box; developers need only change configuration to seamlessly switch LLM providers, greatly reducing the code changes required for switching.

Text Embedding Generation Module

1. Local HuggingFace Embeddings: To ensure data security and reduce external dependencies, our approach supports local embedding models. Using Spring AI's Transformer ONNX support, we can directly load HuggingFace embedding models (converted to ONNX format) in Java and generate vectors. For example, choose the sentence-transformers/all-MiniLM-L6-v2 model, which outputs 768-dimensional sentence vectors. First, prepare the ONNX model file and tokenizer file (Spring AI can automatically download and cache these). Then configure the EmbeddingModel bean:

@Bean
public EmbeddingModel embeddingModel() {
    TransformersEmbeddingModel model = new TransformersEmbeddingModel();
    // 可选：指定模型和分词器资源路径或URL（否则使用默认的all-MiniLM-L6-v2）
    model.setModelResource("classpath:/onnx/all-MiniLM-L6-v2/model.onnx");
    model.setTokenizerResource("classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json");
    return model;
}

This TransformersEmbeddingModel loads the local ONNX model. When embed() is called, it outputs corresponding vector representations for input text lists. As a Spring bean, Spring AI automatically calls its afterPropertiesSet() for initialization. On first use, model files may be loaded or downloaded and cached from the specified location. Subsequently, call embeddingModel.embed(List texts) to obtain embedding vectors.

If using Ollama as the local engine, an alternative is leveraging Ollama's Embedding API. Spring AI provides OllamaEmbeddingModel wrapping Ollama's vector generation interface. Pull the corresponding embedding model (Ollama supports directly pulling HuggingFace embedding models, e.g., ollama pull hf.co/intfloat/e5-small-v2), then:

@Bean
public EmbeddingModel embeddingModel(OllamaApi ollamaApi) {
    // 假设使用一个名为'e5-small-v2'的embedding模型
    var options = OllamaOptions.builder().model("e5-small-v2").build();
    return new OllamaEmbeddingModel(ollamaApi, options);
}

With this approach, all text vectors are generated by the local Ollama service, equivalent to directly using the HuggingFace model.

2. OpenAI Embedding API: To quickly obtain embeddings using OpenAI's pre-trained models, use OpenAI's embedding endpoint (e.g., the text-embedding-ada-002 model). Spring AI's OpenAiEmbeddingModel wraps the invocation logic; provide the API key and optionally the model name:

@Bean
public EmbeddingModel embeddingModel() {
    return new OpenAiEmbeddingModel(new OpenAiApi(System.getenv("OPENAI_API_KEY")));
}

By default, this uses the Ada model to generate 1536-dimensional embedding vectors. This approach suits leveraging OpenAI's powerful vector quality but requires considering network latency and API costs.

3. Embedding Mode Switching: Similar to LLM selection, switch embedding model sources via configuration. For instance, set embedding.mode to local or openai. When set to local, enable TransformersEmbeddingModel or OllamaEmbeddingModel; when openai, use OpenAiEmbeddingModel. Provide a default implementation during development and allow modification via configuration files. For example, default to local models, but if the user provides an OpenAI API key, automatically switch to OpenAI embedding service.

Regardless of implementation, our VectorStore binds an EmbeddingModel during initialization, transforming text during vector database storage and retrieval. For instance, the configured embeddingModel injects into QdrantVectorStore; when vectorStore.add(documents) is called, each Document's text content is converted to vectors by EmbeddingModel and stored in Qdrant.

Data Flow Explanation

Document Indexing Process (ETL Offline Ingestion)

  1. Document Acquisition and Preprocessing: Through an administration interface or batch processing task, obtain raw documents for inclusion in the knowledge base. Documents can be plain text, PDF, Markdown, etc. Use Spring AI's document reader to convert documents into text content, then chunk the text by paragraphs or fixed length to generate a segment list. Encapsulate each segment as a Spring AI Document object containing content text and metadata (such as document ID, title, source).

  2. Embedding Vector Generation: For each document segment, call EmbeddingModel to generate its semantic vector representation. For example, use a local model to convert each segment into a 768-dimensional vector. Vectors typically reside in the Document object's embedding field or are computed during vector database insertion. This illustrates the conversion from document to vector: first partition documents into small chunks, then map each chunk to a high-dimensional vector representation using the embedding model, capturing semantic information.

  3. Vector Storage Ingestion: Call vectorStore.add(documents) to batch-insert document segments into the Qdrant vector database. Spring AI's VectorStore interface performs insertion against the underlying database and automatically handles vector data and metadata storage. For Qdrant, each Document's embedding vector and supplementary payload are stored in the designated collection. If the collection doesn't exist and initializeSchema=true, the VectorStore implementation automatically creates the collection (with appropriate dimensions and index parameters). After successful insertion, the document's semantic representation is persisted in Qdrant for similarity retrieval.

  4. Full-text Index Ingestion: Index document segments into Elasticsearch. For each Document, extract necessary fields (id, content, metadata) and invoke Elasticsearch's indexing API. If using Spring Data Elasticsearch, call Repository's save() method to batch-save segment entities. If using Elasticsearch's REST API directly, construct bulk indexing requests. Each segment becomes a document record in Elasticsearch, with the content field processed for full-text indexing. Consider applying Elasticsearch's built-in Chinese tokenization (if content is in Chinese) to improve retrieval effectiveness.

  5. Indexing Complete: After the above process, knowledge base documents are effectively stored in a hybrid form: Qdrant holds each segment's vector plus metadata, Elasticsearch holds the segment's searchable text. Subsequent queries can leverage both storage capabilities.

Note: The document ingestion process can run as offline batch processing or via an API for dynamic document uploads. This design can implement a /documents/load endpoint (or use the /load concept from Stackademic articles) accepting file upload requests. The service receives files and executes the above steps to complete indexing, returning status or results to the client. For brevity, endpoint details are not elaborated here.

Online Retrieval and QA Flow

  1. User Query Input: Users submit a query request to the RAG service via frontend or API (e.g., HTTP GET/POST /ask?question=...). The query is a natural language question, such as "How does this system support local models?". The Controller layer receives the request, wraps the query string as a Query object, and passes it to the service layer.

  2. Hybrid Retrieval: The RAG service calls HybridRetriever to retrieve relevant information from the knowledge base:

    • Elasticsearch keyword search: Query Elasticsearch using match or multi_match against the content field to search for keywords in the user's question. If matches exist, return the top N highest-relevance segments, e.g., N=5.
    • Qdrant vector search: Encode the user question string to a vector via EmbeddingModel, call vectorStore.similaritySearch() to obtain the top M most similar segments, e.g., M=5. Optionally set a similarity threshold to ignore results below it.
    • Result fusion: Merge the segment sets from both steps. Suppose we obtain up to 10 relevant segments. We can simply sort these segments by importance (e.g., placing Elasticsearch results first if they contain keywords) or pass them unsorted to the LLM. In this design, we pass segment text directly as context, so order has minimal impact, but placing more relevant segments first can improve answer quality.

  3. Prompt Construction: Assemble the user question and retrieved document segments into prompt content for the LLM. Typically use a predefined prompt template, for example:

请根据以下文档内容回答问题。如果无法从中找到答案，请回复“未找到相关信息”。

文档内容:
{{{context}}}

问题: {{{question}}}

答案:

The {{{context}}} placeholder is filled by concatenating retrieved document segment texts; {{{question}}} is the original user question. The resulting complete prompt provides reference information for the question. Spring AI's PromptTemplate can assist this step, or manually concatenate strings. This shows using PromptTemplate to insert retrieved content during query and render the final prompt before sending to the model.

Spring AI also supports using QuestionAnswerAdvisor directly to simplify the process. If using ChatClient.prompt().advisors(new QuestionAnswerAdvisor(vectorStore)), it automatically queries the vector store before calling the model and attaches results. Since we've implemented a custom HybridRetriever (including Elasticsearch), we construct the prompt ourselves to incorporate both retrieval results.

  1. LLM Answer Generation: Call the LLM model to generate an answer. Execute via the previously configured ChatClient:
String promptText = promptTemplate.render(Map.of("context", combinedDocsText, "question", userQuestion));
ChatResponse response = chatClient.prompt(promptText).call().chatResponse();
String answer = response.content();

ChatClient sends our complete prompt to the underlying ChatModel (OpenAI or Ollama). The model, receiving the query with reference content, formulates an answer based on the provided context, reducing hallucination and error probability. The generated answer returns via ChatResponse; extract its content string as the final answer.

  1. Result Return: Wrap the LLM's answer in a response and return it to the caller via Controller. You can return only the answer text or include metadata such as a list of source documents cited. This design focuses on core functionality and assumes only answer text is returned. On the frontend interface, users see the answer generated based on their question.

  2. Conversation Memory (Optional): For multi-turn conversation support, combine Spring AI's ChatMemory functionality, providing previous questions and answers as conversation history to ChatClient, enabling context recall. However, basic RAG QA scenarios can omit conversation history, treating each query independently.

The entire query flow, from the user's perspective, is simply providing a question and receiving an answer. Behind the scenes, dual-channel retrieval (keyword and semantic) occurs, then the large model synthesizes the documents to formulate a result, realizing "knowledge-base-grounded QA". This RAG technique effectively mitigates LLM context length limits, knowledge cutoff, and hallucination issues.

Key Components and Interface Definitions

Configuration and Component Summary

Main Dependencies: The project includes these critical dependencies:

Application Configuration: Manage the above mode-switching parameters via application.yml, for example:

app:
  llm: mode: local        # 本地(local)或openai
  embedding: mode: local  # 本地(local)或openai
spring:
  ai:
    # OpenAI 配置（仅当使用openai模式）
    openai:
      api-key: xxxx
      chat.model: gpt-4
    # Ollama 配置（仅当使用local模式）
    ollama:
      base-url: http://ollama:11434   # 假设Docker容器内服务名
      default-model: llama2-7b
    # Qdrant 向量库配置
    vectorstore:
      qdrant:
        collection-name: documents
        initialize-schema: true

This example shows custom app configuration segments controlling modes and Spring AI's built-in configuration properties. Through profiles or conditional injection, select different ChatModel configurations based on app.llm.mode.

Key Beans: A summary of important Spring beans involved in this design and their configuration methods:

@Bean
@ConditionalOnProperty(name="app.llm.mode", havingValue="openai")
ChatClient chatClientOpenAI(OpenAiChatModel model) {
    return ChatClient.builder(model).build();
}

@Bean
@ConditionalOnProperty(name="app.llm.mode", havingValue="local")
ChatClient chatClientOllama(OllamaChatModel model) {
    return ChatClient.builder(model).build();
}

The pseudo-code above illustrates selecting different ChatModels based on configuration to create ChatClient. In practice, using Spring Boot Starter and configuration files, most cases simply @Autowired ChatClient (the underlying configuration has already selected the model).

RagService Interface Definition

RagService can define the following interface for Controller invocation:

public interface RagService {
    /** 根据用户问题返回答案 */
    String getAnswer(String question);
}

The implementation class RagServiceImpl completes the work following the online retrieval QA flow steps:

@Service
public class RagServiceImpl implements RagService {
    @Autowired private HybridRetriever retriever;
    @Autowired private ChatClient chatClient;
    @Value("${app.maxDocs:6}") private int maxDocs;

    @Override
    public String getAnswer(String question) {
        // 1. 调用混合检索获取相关文档片段列表
        List<Document> docs = retriever.retrieve(new Query(question));
        if (docs.isEmpty()) {
            return "抱歉，未能找到相关信息。";
        }
        // 截取最多maxDocs篇，以防过长
        List<Document> topDocs = docs.size() > maxDocs ? docs.subList(0, maxDocs) : docs;
        // 2. 构造提示上下文文本
        StringBuilder context = new StringBuilder();
        for (Document doc : topDocs) {
            context.append(doc.getContent()).append("\n");
        }
        String prompt = String.format("基于以下内容回答问题：\n%s\n问题：%s\n回答：", context, question);
        // 3. 调用LLM生成回答
        ChatResponse response = chatClient.prompt(prompt).call().chatResponse();
        return response.content();
    }
}

In the above code:

Through this implementation, RagService provides a high-level interface externally, concealing internal complexity, and Controller need only pass the question string to obtain an answer.

Example Controller

Finally, provide an example controller demonstrating interface definition and invocation flow:

@RestController
@RequestMapping("/api")
public class QaController {
    @Autowired
    private RagService ragService;

    // 提问接口
    @GetMapping("/ask")
    public ResponseEntity<String> askQuestion(@RequestParam("q") String question) {
        String answer = ragService.getAnswer(question);
        return ResponseEntity.ok(answer);
    }

    // 文档上传接口（可选，实现文档入库）
    @PostMapping(value="/documents", consumes=MediaType.MULTIPART_FORM_DATA_VALUE)
    public ResponseEntity<String> uploadDocument(@RequestPart("file") MultipartFile file) {
        // 调用文档处理服务将文件内容存入ES和Qdrant
        // DocumentLoader.load(file);
        return ResponseEntity.ok("文档已上传并索引完成");
    }
}

Through the above Controller, frontend or users interact with the RAG service via HTTP requests, enabling dynamic QA.

Docker Compose Environment Deployment

For convenience in development and deployment, a Docker Compose configuration simultaneously launches required external services (Qdrant, Elasticsearch, Ollama) and the Spring Boot application. The project's docker-compose.yml example is as follows:

version: '3.9'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.9.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false       # 禁用安全认证
      - ES_JAVA_OPTS=-Xms1g -Xmx1g         # 内存配置，可根据需要调整
    ports:
      - "9200:9200"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9200"] 
      interval: 30s
      retries: 3

  qdrant:
    image: qdrant/qdrant:v1.3.5
    container_name: qdrant
    ports:
      - "6333:6333"   # HTTP API
      - "6334:6334"   # gRPC API
    volumes:
      - qdrant_data:/qdrant/storage

  ollama:
    image: ollama/ollama:0.1.9
    container_name: ollama
    ports:
      - "11434:11434"  # Ollama REST API 默认端口
    volumes:
      - ollama_data:/root/.ollama

  rag-service:
    build: .
    container_name: rag-service
    environment:
      SPRING_PROFILES_ACTIVE: "local"          # 激活本地LLM配置。如切换OpenAI则设为cloud等。
      OPENAI_API_KEY: "${OPENAI_API_KEY:-}"    # 可选，提供OpenAI密钥
      QDRANT_GRPC_HOST: "qdrant"               # 若应用读取环境配置连接Qdrant
      QDRANT_GRPC_PORT: 6334
    depends_on:
      - elasticsearch
      - qdrant
      - ollama
    ports:
      - "8080:8080"

Explanation:

Network Configuration: Compose places these services in the same network by default; container names serve as hostnames. Therefore, when the application connects to Qdrant and Elasticsearch, it can use qdrant:6334 and elasticsearch:9200 as addresses. Spring AI's Docker Compose integration module can even automatically discover services and configure connections based on container names—for instance, containers with names containing "ollama/ollama" are recognized as Ollama services.

Startup: Run docker-compose up -d to start all containers in the background. After Elasticsearch and Qdrant complete startup (verify via health check logs or API), the Spring Boot application automatically connects to them. At this point:

Through Docker Compose, deploy the entire RAG system's dependencies with one command, convenient for testing locally or on servers.

Summary

This design document details a comprehensive RAG search service solution built on the Java Spring ecosystem. Through Spring AI's abstractions and integration capabilities, we've realized a modular Retriever-VectorStore-RAG Service architecture supporting hybrid retrieval, flexible LLM and embedding switching, and containerized deployment. Core technology selections—Spring Boot, Spring AI, Elasticsearch, Qdrant, and Ollama—are currently popular and well-supported components. The document provides comprehensive architecture and data flow analysis, delivers clear configuration and code examples, and ensures implementation details are reproducible. Developers can construct a verifiable RAG QA system following this guide, balancing answer accuracy, deployment flexibility, and data privacy, providing robust support for enterprise knowledge base QA, intelligent search, and similar application scenarios.

© 2026 Yuxu Ge ·