From Offline Deployment to Kubernetes Pipeline: A Field Engineer's Practical Summary

Deploying applications in air-gapped environments required a methodical offline deployment approach. Our process centered on these steps:

• Prepare an executable JAR: Build an uber JAR (with dependencies bundled) in an isolated network environment through CI, or download all dependencies offline in advance. Ensure versions are correct and dependencies complete; failure to do so leads to broken deployments when the system cannot download packages at runtime.
• Transfer the package securely: Move the JAR to the target server. Verify file integrity before and after transfer using MD5 checksums. For example, copy via SCP:

scp app-1.0.jar [email protected]:/opt/deploy/app-1.0.jar

• Configure the runtime environment: Install the required JDK offline on the target server. Pre-arrange any specific configuration needed—database connection strings, cache addresses, and similar—through configuration files or environment variables. We placed an application.properties file in the same directory as the JAR or passed configuration paths as startup arguments.
• Start the application process: Use nohup or systemd to launch the JAR and keep it running in the background:

nohup java -jar /opt/deploy/app-1.0.jar --spring.config.location=/opt/deploy/config/ &

Even if the terminal closes, the application continues running. Log output is redirected to nohup.out or a designated log file.

• Verify the deployment: Check application logs and listening ports to confirm a successful start. For example, monitor logs in real time with tail -f nohup.out to catch errors, and verify port binding with netstat -tunlp | grep 8080.

This process ensured smooth deployment in disconnected networks. However, the manual approach had clear drawbacks: each update required human intervention, multi-server deployments were prone to version inconsistencies and missed steps, and as release frequency increased, the need for a more automated and standardized approach became obvious.

Cassandra Large-Table Export

During operations, we needed to export a high-volume Cassandra table containing hundreds of millions of records as a backup. Without the right tools, this task was genuinely difficult:

• Initial attempts and problems: We first tried querying all data directly via CQL and writing it to a file, but the sheer volume caused client-side memory exhaustion or timeouts. We then tried the COPY command built into cqlsh:

COPY keyspace_name.table_name TO 'export.csv';

This command exports query results directly to CSV. However, with hundreds of millions of rows, COPY TO was prohibitively slow and frequently failed midway due to network latency or timeouts, making recovery cumbersome.

• Optimization: partitioned batch export: We adopted a partitioned export strategy, dividing the large table into manageable chunks and exporting each sequentially. The approach used primary keys or time ranges for partitioning: scripts queried data by range, exporting tens of thousands of rows at a time and appending to the output file. This segmented approach prevented single transfers from overwhelming the system. We monitored Cassandra node health and scheduled exports outside business peak hours to minimize impact on live reads and writes.

• Adopting professional tooling: We later introduced DataStax's Bulk Loader (DSBulk), a tool purpose-built for Cassandra bulk import and export. With DSBulk, a single command exports an entire table:

dsbulk unload -k keyspace_name -t table_name -url export_data/ -maxRetries 5

DSBulk optimizes reading internally with parallel processing and provides features like resumable exports. In a test, exporting a table of approximately 50 million rows to CSV was reduced from several hours to under one hour—a several-fold improvement in efficiency.

• Results and validation: After export, validate data completeness. We compared exported row counts against Cassandra records to catch any omissions and randomly sampled content for spot checks. The resulting CSV files were compressed and archived for potential recovery or analysis later.

This approach resolved the Cassandra export challenge. Where specialized tools were unavailable, partitioned export was a workable compromise; once professional tools became available, large-scale data migration became both more reliable and faster.

Common Startup Failure Cases

During application deployment and operation, we encountered Java startup failures traceable to third-party components: Atomikos for distributed transaction management and Curator for ZooKeeper client operations. Both cases provide instructive examples of diagnosis and resolution.

Atomikos-Induced Startup Exception

One service used Atomikos as its distributed transaction manager (for multi-datasource transactions). When starting two service instances on the same server, Atomikos initialization threw an exception and blocked startup. The log excerpt:

com.atomikos.icatch.SysException: Error in init: Log already in use? tmlog in ./
    at com.atomikos.icatch.impux.TransactionServiceImp <...> 
Caused by: com.atomikos.recovery.LogException: Log already in use by another process.

The error showed Atomikos unable to create its transaction log file due to a lock conflict (Log already in use). The root cause: multiple applications using Atomikos on the same machine were all trying to use the same default transaction log path, preventing the later-launched process from acquiring the file lock.

Our resolution: We confirmed that the first service was using Atomikos' default transaction log (typically in a transaction-logs folder under the application's working directory). To resolve the conflict, we applied one of two approaches:

• Option 1: Isolate transaction log paths – Modify each application's Atomikos configuration to use distinct log directories or filenames. For example, in Spring Boot configuration:

spring.jta.atomikos.log-dir=./transaction-logs-app2

The second application's transaction log now writes to a separate directory, eliminating contention with the first.

• Option 2: Stagger startup or consolidate applications – If business requirements allow, merge related modules into a single JVM deployment, eliminating resource contention. Alternatively, ensure only one Atomikos instance runs at a time. For separate deployments, containerization or similar isolation strategies can help.

After applying the log path modification, we restarted the application. Atomikos initialized successfully without further conflicts. This case reminded us that middleware defaults often assume single-instance deployments. When hosting multiple instances on one host, check for shared resource conflicts—files, ports, and similar—and use configuration to isolate them.

Curator-Induced Startup Hang

Another issue involved Apache Curator, a popular ZooKeeper client framework. A microservice used Curator during startup to register with ZooKeeper, but we found the process hung for extended periods in one environment, with logs repeatedly showing:

org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
	at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:123)
	... 

The log indicated Curator's connection was lost and continuously retrying (ConnectionLoss means the ZooKeeper cluster was unreachable). This caused the application to block at startup. Investigation revealed several possible causes:

• ZooKeeper service not running: We first suspected ZooKeeper itself was unavailable. Logging into the ZooKeeper host and running zkServer.sh status confirmed the service had not started—it had been accidentally shut down in this test environment, and we hadn't noticed.

• Firewall or network problems: After starting the ZooKeeper service, connections still failed. This prompted us to check server firewall settings, and we found ZooKeeper's default port was blocked. After opening the firewall for the relevant port, the Curator client connected successfully.

• Incorrect connection configuration: A frequent cause is misconfigured connection strings—wrong ZooKeeper cluster IP addresses or ports, or DNS resolution failures. While this didn't occur in this incident, we verified configuration during our investigation to rule it out.

Our solution: We addressed each cause: started the ZooKeeper service and adjusted firewall rules to allow access to ZooKeeper's port (2181 by default). After restarting the microservice, Curator connected and the application started cleanly. To prevent recurrence, we improved startup scripts by adding health checks for dependent services before application deployment—for example, automatically detecting ZooKeeper status and prompting or delaying startup if it's not ready. We also tuned Curator's timeout and retry parameters to fail fast during connection issues rather than hanging indefinitely.

These two cases taught us that startup troubleshooting requires rapid log-based diagnosis combined with awareness of component configuration and runtime dependencies. Whether managing transaction managers or service registry clients, understanding how they work and what parameters they expose makes root-cause discovery and resolution much faster.

Kubernetes Standard Release Pipeline

Having addressed initial deployment and runtime issues, we introduced Kubernetes to restructure the release pipeline. Our goal was end-to-end automation from build through deployment to release, replacing ad-hoc manual steps with standardized workflows. Here is how we implemented K8s-based releases in practice:

• Containerize the application: First, we created a Docker image for our application. We wrote a minimal Dockerfile that embeds the executable JAR:

FROM openjdk:8-jre-slim
COPY app-1.0.jar /app/app.jar
CMD ["java", "-jar", "/app/app.jar", "--spring.config.location=/app/config/"]

We also packaged configuration files needed at runtime into the image's /app/config/ directory (or mounted them via ConfigMap; see below). This ensured the container found the correct configuration on startup. After writing the Dockerfile, we built the image using an internal CI tool and pushed it to a private image registry (e.g., registry.example.com/myteam/app:1.0).

• Write Kubernetes deployment manifests: Next, we wrote Kubernetes manifest files defining Deployments, Services, and other resources. A sample Deployment manifest excerpt:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp-container
        image: registry.example.com/myteam/app:1.0
        ports:
        - containerPort: 8080
        env:
        - name: JAVA_OPTS
          value: "-Xms512m -Xmx512m"
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

This manifest declares a two-replica deployment and configures containers to use the image we built. It also sets readiness and liveness probes that periodically hit the application's /health endpoint. These probes ensure the application only receives traffic when healthy and automatically restart containers on failure, improving release reliability.

• Configuration management: Rather than hardcoding application configuration into images, we use ConfigMap and Secret for configuration and sensitive data. In the Deployment manifest, we reference these via volume mounts or environment variables—for example, database connection strings via Secret, injected as environment variables at startup. This allows different configuration per environment (test, production) without rebuilding the image.

• CI/CD pipeline: We integrated build and deployment into CI/CD tooling (Jenkins, GitLab CI, etc.). When code merges to the main branch, the pipeline automatically: compiles and tests → builds Docker image → pushes image → deploys to the K8s cluster. The deployment stage applies pre-written K8s manifests to the cluster using kubectl apply -f or similar tools. Combined with the Deployment controller's rolling update strategy, new versions roll out by progressively replacing old containers with new ones, achieving zero-downtime or minimal-interruption releases.

• Release standards and gates: To maintain release quality, we established a pre-release checklist:

  • Confirm the new version passes full regression testing in the staging environment.
  • Image scans are free of high-severity vulnerabilities.
  • YAML manifests comply with internal standards (labels, resource requests and limits fully specified, etc.).
  • For major releases, use canary or phased rollout: deploy to a small subset of instances first, monitor behavior, then gradually expand to full capacity.

Kubernetes standardized our release process into a pipeline workflow, enabling one-click deployments and rollbacks while dramatically reducing human error. Every deployment is logged and monitored, making issue tracking and rapid recovery far more straightforward.

Logging and Monitoring Infrastructure

As the system evolved toward containers and distributed architecture, we built a comprehensive logging and monitoring stack for operational troubleshooting and performance tuning.

• Centralized logging system: Previously, logs were scattered across servers, requiring manual login and search to troubleshoot. We deployed an ELK/EFK stack to aggregate container logs. Specifically, we ran Filebeat or Fluentd log collectors on the Kubernetes cluster, capturing logs from container stdout and stderr and shipping them to central storage (Elasticsearch). Within our applications, we standardized on JSON-formatted logs containing timestamps, severity, thread ID, request ID, and other fields—making filtering and searching in Kibana straightforward. Now when a service has an error, we query Kibana once to see logs from all instances, tracing the timeline to identify the problem far more quickly.

• Performance metrics monitoring: We built a monitoring system using Prometheus and Grafana. Prometheus periodically scrapes metric data from each service (including system metrics like CPU and memory, plus application-specific metrics like request counts and error rates), while Grafana visualizes the data. We integrated the Micrometer library into applications to expose metrics to Prometheus. Via custom dashboards, we monitor QPS, response time distributions, database connection counts, and other key metrics in real time. Combined with Alertmanager alert rules, whenever a metric exceeds a threshold (e.g., sustained high CPU, sudden error spike), the system notifies the team via SMS or Slack, enabling rapid response.

• Distributed tracing: Beyond logs and metrics, we evaluated distributed tracing tools (SkyWalking, Jaeger) for tracking requests across services. In complex microservice environments, these tools help us trace a user request's path through multiple services and identify bottlenecks or failures. However, given deployment and operational overhead, we chose to pilot tracing selectively on critical paths rather than comprehensive instrumentation, with logs and metrics remaining our primary operational tools.

Through this logging and monitoring infrastructure, we dramatically improved system observability. We moved from guesswork during outages to data-driven troubleshooting. Not only did mean time to recovery (MTTR) drop, but routine performance optimization became evidence-based. Operations became far less reactive.

Impact Assessment: Before and After

Comparing results across the improvements:

The improvements are significant. Release efficiency improved dramatically—from half-hour manual deployments requiring multiple people to pipeline-driven releases typically completed in minutes with minimal intervention. Incident diagnosis improved even more: what once took 1–2 hours of concentrated log analysis now surfaces in minutes via centralized logging and monitoring alerts. Overall, these practices transformed our team's DevOps model and provided the stability needed for rapid business iteration.

Lessons and Future Directions

Through this journey from offline deployment to Kubernetes pipelines, our team gained valuable experience and confirmed the value of modern infrastructure practices. Here are the key takeaways and future directions:

• Key lessons:

  • Infrastructure as Code is critical: deployment scripts, K8s manifests, monitoring configurations—all should be versioned, code-reviewed, and executed through pipelines for consistency.
  • Tool selection must match real constraints: Specialized tools like DSBulk dramatically improve efficiency; avoid chasing novelty for its own sake. Choose tools aligned with team capabilities and introduce them gradually.
  • Failure rehearsal matters: After implementing automation and monitoring, regularly practice failure scenarios (single points of failure, rollback on deployment errors, etc.) so the team is ready to respond effectively when they occur.

• Future work: We plan to advance continuous delivery further—enabling one-click multi-environment deployments and implementing advanced strategies like blue-green and canary releases. On the observability front, we'll introduce distributed tracing for end-to-end request visibility and evaluate service mesh technologies for traffic control and security policies. These are our next priorities.

Finally, we hope this practical summary proves useful to readers. Infrastructure modernization is gradual. From early experimentation with offline deployment to cloud-native practices today, each step brought both challenges and learning. As hands-on engineers, we should embrace the changes modern tools bring while staying alert to subtle issues, building experience, and continuously improving stability and delivery efficiency. We look forward to sharing more practical insights with the community as we continue.