Yuxu Ge
Article · 2022-02-14

Building a Search System with Cassandra, SolrCloud, Zookeeper, Kafka, and Python Microservices

This system design assumes a Linux infrastructure (CentOS or Ubuntu) with JDK installed, since Cassandra, Solr, and Kafka all require Java. Deployment uses multiple servers organized as follows:

All components communicate over internal networks with firewall rules allowing necessary ports (Cassandra 9042, Solr 8983, Zookeeper 2181, Kafka 9092, and microservice ports as configured). The examples below use placeholder IPs like 10.201.X.X; substitute your actual addresses in production.

Installing and Configuring Cassandra

Install Cassandra

Download Cassandra from the Apache site (this example uses 3.11.12), upload to the target server 10.201.X.X, and extract:

$ wget https://downloads.apache.org/cassandra/3.11.12/apache-cassandra-3.11.12-bin.tar.gz
$ tar -zxvf apache-cassandra-3.11.12-bin.tar.gz
$ cd apache-cassandra-3.11.12

Cassandra requires no compilation. To start the node, enter the Cassandra directory and run bin/cassandra. Before the first startup, modify the configuration to ensure node communication works correctly.

Configure Cassandra nodes

Edit conf/cassandra.yaml to match your network environment:

seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "10.201.X.X,10.201.X.X"

At least one node IP in the cluster should be listed as a seed.

Start Cassandra:

$ bin/cassandra -R   # 后台启动 Cassandra (-R 可去除超级用户模式限制)

Initial startup may take a moment for initialization. Verify connectivity with the CQL shell:

$ bin/cqlsh 10.201.X.X 9042   # 连接到本机的 Cassandra CQL 服务
Connected to SearchCluster at 10.201.X.X:9042.
cqlsh>

Once connected, create a keyspace and tables. For example, create the search keyspace:

CREATE KEYSPACE search 
WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} 
AND durable_writes = true;

Note: The above configuration uses simple replication strategy and replication factor 1, suitable only for testing. Production clusters should use NetworkTopologyStrategy and appropriate replication factors based on datacenter topology.

Design Cassandra tables for your search use case by organizing data around query patterns. Cassandra's strength lies in storing wide-row data with query-driven schema design. Create multiple tables in the search keyspace:

For search workloads, Cassandra acts as the persistent store and data source. Solr handles the actual query execution via its indices, so Cassandra is typically not queried directly during search; instead, product data is pre-loaded into Solr by the indexing service.

Installing and Configuring Zookeeper

Zookeeper serves two roles in this architecture: coordination for Kafka and configuration management for SolrCloud. This example uses a 3-node cluster to ensure high availability.

Install Zookeeper

Download Zookeeper (e.g., version 3.7.0) and extract on each server:

$ wget https://downloads.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz
$ tar -zxvf apache-zookeeper-3.7.0-bin.tar.gz
$ mv apache-zookeeper-3.7.0-bin /opt/app/zookeeper

Assume three Zookeeper servers with IPs 10.201.X.X, 10.201.X.X, and 10.201.X.X respectively.

Configure Zookeeper

Copy the example configuration on each node:

$ cp conf/zoo_sample.cfg conf/zoo.cfg

Edit zoo.cfg and configure the following:

server.1=10.201.X.X:2888:3888
server.2=10.201.X.X:2888:3888
server.3=10.201.X.X:2888:3888

The N in server.N corresponds to each node's unique ID. Port 2888 is for cluster communication, and 3888 is for leader election.

Start the Zookeeper cluster

On each server, run:

$ ./bin/zkServer.sh start

After startup, check the status with zkServer.sh status. One node should be the leader, and others followers. To verify Zookeeper availability, connect with zkCli:

$ ./bin/zkCli.sh -server 10.201.X.X:2181

Test by creating a node:

[zk: 10.201.X.X:2181(CONNECTED) 0] create /solr_demo "test"

Successful creation confirms Zookeeper is operational. We will later store SolrCloud configuration data at /solr_demo in Zookeeper (a chroot path).

Installing and Configuring Kafka

Install Kafka

Download Kafka from the Kafka website or Apache archives (this example uses Kafka 2.x):

$ wget https://downloads.apache.org/kafka/2.7.0/kafka_2.12-2.7.0.tgz
$ tar -zxvf kafka_2.12-2.7.0.tgz
$ mv kafka_2.12-2.7.0 /opt/app/kafka

Extract the archive on each Kafka server. Kafka requires JDK 1.8+.

Configure Kafka brokers

Edit config/server.properties on each Kafka node:

zookeeper.connect=10.201.X.X:2181,10.201.X.X:2181,10.201.X.X:2181

If SolrCloud uses a /solr_demo subpath in Zookeeper, Kafka can connect to the root directory directly; Kafka will create /brokers and other paths in Zookeeper as needed.

Adjust Kafka memory and other parameters as needed, but defaults are adequate for initial testing.

Start Kafka

Start each Kafka broker in sequence:

$ ./bin/kafka-server-start.sh -daemon config/server.properties

The -daemon flag runs the process in the background. After startup, Kafka registers itself with Zookeeper. You can verify this by checking the /brokers/ids node in Zookeeper with a Zookeeper client.

Create topics

Create Kafka topics to carry product data updates. For example, create a topic named product_update with replication factor 2 and 3 partitions:

$ ./bin/kafka-topics.sh --create --topic product_update --partitions 3 --replication-factor 2 --bootstrap-server 10.201.X.X:9092

(--bootstrap-server specifies any Kafka node address). Verify creation succeeded by listing current topics:

$ ./bin/kafka-topics.sh --list --bootstrap-server 10.201.X.X:9092

The Kafka cluster is now ready.

Installing and Configuring SolrCloud

Solr is the core search indexing component. We use SolrCloud mode to support sharding and high availability (this example uses Solr 7.7.3).

Install Solr

Download Solr (zip or tgz format) and extract on each node:

$ wget https://archive.apache.org/dist/lucene/solr/7.7.3/solr-7.7.3.tgz
$ tar -zxvf solr-7.7.3.tgz
$ mv solr-7.7.3 /opt/app/solr

Prepare SolrCloud configuration

Before starting SolrCloud, upload the Solr core configuration to Zookeeper or let Solr auto-upload on startup. This example uses a collection named "product" with schema and Data Import Handler (DIH) configurations prepared in advance. Follow these steps:

ZK_HOST="10.201.X.X:2181,10.201.X.X:2181,10.201.X.X:2181/solr_demo"

The /solr_demo path means SolrCloud will store configuration data under that path in Zookeeper. Ensure this path exists (create it earlier with zkCli).

Complete this preparation on one machine, then package the solr directory as solr.zip and distribute to all Solr servers to ensure consistency.

Upload Solr configuration and start SolrCloud

Assume the development team provided Solr core configurations (schema.xml, solrconfig.xml, data-config.xml, etc.) in /home/netty/solr_core_config/product. Upload this to Zookeeper:

$ cd /opt/app/solr
$ ./bin/solr zk upconfig -z 10.201.X.X:2181,10.201.X.X:2181,10.201.X.X:2181/solr_demo \
    -n product -d /home/netty/solr_core_config/product

This command uploads the local product configuration directory to Zookeeper, registering it as the "product" config set. (To view existing configs, use downconfig, e.g., ./bin/solr zk downconfig -n product -d ./downloaded_conf -z ...).

Start Solr nodes

On each Solr server, start Solr in cloud mode to join the cluster:

$ ./bin/solr start -c -m 1g -z 10.201.X.X:2181,10.201.X.X:2181,10.201.X.X:2181/solr_demo -p 8983 -d /opt/app/solr/solr_data

Parameter meanings: -c enables cloud mode; -m 1g sets max heap to 1GB; -z specifies the Zookeeper address list and path; -p 8983 sets the listening port (default); -d /opt/app/solr/solr_data specifies the Solr instance data directory (we copy the example/server contents here for independent configuration).

After starting all Solr nodes, they automatically register with Zookeeper and form a cluster. Next, create a collection via the Solr API using the bin/solr script or Solr Admin UI:

$ ./bin/solr create -c product -n product -shards 2 -replicationFactor 2 -p 8983

This creates a collection named "product" using the "product" config set uploaded earlier, with 2 shards and 2 replicas per shard. The script distributes cores across nodes, and SolrCloud begins serving search requests. Verify by opening a browser to any Solr node's admin interface, e.g., http://10.201.X.X:8983/solr, to view cluster status and collection lists.

Note: SolrCloud relies on Zookeeper for configuration and state. Ensure your Zookeeper cluster is stable. Incorrect ZK settings (wrong address or path) will prevent Solr from starting.

Python Microservices Development and Deployment

With data storage and indexing in place, Python microservices connect these components. The original Java/Spring Boot modules handled data synchronization and search; we reimplement these with Python examples.

Microservice responsibilities

  1. Data sync service (Kafka consumer): continuously consumes the Kafka product_update topic. When product data updates arrive, the service writes new data to Cassandra using the DataStax cassandra-driver and then triggers Solr index updates. Index updates may be incremental (via Solr HTTP API) or batch imports (via DIH), depending on strategy. Python can use kafka-python or confluent-kafka to consume messages, cassandra-driver to write to Cassandra, and requests or pysolr to call Solr APIs.

  2. Search API service: exposes a search REST interface. Client requests are received and forwarded to SolrCloud, results are formatted and returned. Implement using Flask or FastAPI. Internally, use pysolr or direct HTTP requests to query Solr, then convert results to JSON. This service corresponds to the original Java search query module and can also supplement results from Cassandra—for example, retrieve product IDs from the search index and then query Cassandra for the latest stock levels if not indexed.

  3. Supporting services: bulk import (like product-upload) or job triggering (like job-trigger) can also be implemented in Python. If these services need to integrate with existing scheduling frameworks (e.g., xxl-job), call their HTTP endpoints or implement the protocol.

Microservice code and startup scripts

Build a search API service with FastAPI and a Python startup script. Assume the search service code is in app.py (simplified example):

# app.py (FastAPI 简易示例)
from fastapi import FastAPI
import pysolr

solr = pysolr.Solr('http://10.201.X.X:8983/solr/product', timeout=10)  # Solr 地址

app = FastAPI()

@app.get("/search")
def search(q: str):
    # 在 Solr 中查询
    results = solr.search(q)
    # 提取需要的字段返回
    docs = [doc for doc in results]
    return {"query": q, "results": docs}

Production may require more complex query construction and result processing. Next, write a management script service_control.py to start or stop the FastAPI service. Use subprocess to invoke Uvicorn, mimicking traditional Shell scripts with java -jar ... &:

#!/usr/bin/env python3
# service_control.py
import subprocess, sys, os, signal

APP_COMMAND = ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7220"]

def start():
    """启动服务"""
    # 将输出重定向到日志文件
    logfile = open("service.log", "a")
    # 使用 nohup & 类似效果启动子进程
    process = subprocess.Popen(APP_COMMAND, stdout=logfile, stderr=logfile, preexec_fn=os.setpgrp)
    print(f"Service started with PID {process.pid}")

def stop():
    """停止服务"""
    # 查找运行中的进程（通过端口或命令名）
    try:
        # 利用 pgrep 查找 uvicorn 进程
        result = subprocess.run(["pgrep", "-f", "uvicorn.*7220"], capture_output=True, text=True)
        pids = result.stdout.strip().split()
        if not pids:
            print("Service is not running.")
            return
        for pid in pids:
            os.kill(int(pid), signal.SIGTERM)
        print("Service stopped.")
    except Exception as e:
        print(f"Error stopping service: {e}")

def restart():
    """重启服务"""
    stop()
    start()

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: service_control.py [start|stop|restart]")
        sys.exit(1)
    cmd = sys.argv[1]
    if cmd == "start":
        start()
    elif cmd == "stop":
        stop()
    elif cmd == "restart":
        restart()
    else:
        print("Unknown command:", cmd)

This script mirrors Java service startup logic:

Place service_control.py on the server with execute permissions, then use ./service_control.py start to launch the Python microservice or ./service_control.py stop to stop it. For the data sync service (Kafka consumer), create a similar startup approach—write a consumer.py script and manage it with a control script.

Note: For production, use more robust hosting methods (Supervisor, systemd, or container orchestration). The scripts above are examples only.

Cluster Deployment Considerations

With all components deployed, pay attention to these cluster setup details and best practices:

Summary and Lessons Learned

This implementation built a distributed search system using Cassandra, SolrCloud, Zookeeper, Kafka, and Python microservices. Compared to monolithic search applications, this architecture offers scalability and modularity: Cassandra provides high write throughput and horizontal scale, SolrCloud offers powerful indexing, Kafka decouples data flow asynchronously, and Python microservices enable flexible business logic.

Key takeaways from this work:

This deployment demonstrates a complete search solution. Building such a system from scratch requires knowledge across multiple technologies. Through step-by-step installation, configuration, and tuning, we mastered how each component integrates. Future enhancements might containerize this architecture on Kubernetes for automated scaling and operations. We hope this experience proves useful for engineers building similar search platforms.

© 2026 Yuxu Ge ·