Python Application Logs: Troubleshooting ELK Integration

• Logs present but not in ES = Filebeat/Logstash pipeline issue
• First: does Filebeat actually see the log files?
• Next: does Logstash actually receive the events?
• Finally: does Elasticsearch actually index them?

Troubleshooting Process

1. Check Filebeat inputs

filebeat.inputs:
  - type: log
    paths:
      - /var/log/my_service/*.log
    fields:
      project: my_service

• Log directory on the actual server: /var/log/my_service-neo/
• Filebeat configuration: hardcoded to my_service → path mismatch

2. Create a temporary symlink for load testing

ln -s /var/log/my_service-neo /var/log/my_service

• After creating the symlink, logs appeared immediately in Kibana
• This proved that Filebeat, Logstash, and Elasticsearch themselves were working fine—the path was the culprit

3. Trace the source of the "-neo" suffix

• The Python application's project name was changed manually in settings.py:

  SERVICE_NAME = "my_service-neo"
  

• The deployment pipeline and Filebeat rules still referenced the old name my_service

Any change to the project name that isn't synchronized across all configuration files becomes a hidden failure point.

Root Cause Analysis

• Local logging works → Python logging configuration is correct
• Filebeat sees nothing → collection rules are stale
• Upstream automation scripts and Kubernetes ConfigMaps also hardcoded the old path

Solution

Two options presented:

• Option A: Revert the project name
  • Simple, but requires rolling back Tags and image repository names—not recommended
• Option B: Update all collection rules ✅
  • Update Filebeat, Logstash, and custom monitoring to use the new project name
  • Maintain a single source of truth across the entire pipeline

We implemented Option B.

Structured Logging on the Python Side

import logging
import logging.config
import json_log_formatter

formatter = json_log_formatter.JSONFormatter()

LOG_CONFIG = {
    "version": 1,
    "formatters": {
        "json": {
            "class": "json_log_formatter.JSONFormatter",
            "format": "%(asctime)s %(levelname)s %(name)s %(message)s"
        }
    },
    "handlers": {
        "file": {
            "class": "logging.handlers.TimedRotatingFileHandler",
            "filename": "/var/log/my_service-neo/app.log",
            "when": "midnight",
            "backupCount": 7,
            "formatter": "json"
        }
    },
    "root": {
        "handlers": ["file"],
        "level": "INFO"
    }
}

logging.config.dictConfig(LOG_CONFIG)
logger = logging.getLogger(__name__)

logger.info({"event": "service_start", "version": "1.2.3"})

• Use json_log_formatter to output logs as pure JSON for frictionless Filebeat parsing
• The filename field includes the new project name my_service-neo

Filebeat Dynamic Discovery

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      templates:
        - condition:
            equals:
              kubernetes.labels.app: my_service-neo
          config:
            - type: log
              paths:
                - /var/log/my_service-neo/*.log
              json.keys_under_root: true
              json.add_error_key: true

• Filebeat automatically collects logs from any pod labeled app=my_service-neo
• Eliminates hardcoded paths entirely

Logs Persisting to Elasticsearch

• Logstash uses the json codec and applies Index Lifecycle Management based on date
• Elasticsearch index naming: my_service-neo-%{+yyyy.MM.dd}
• Update the index pattern in Kibana's Saved Search feature to complete the setup

Lessons Learned

• Synchronize project name changes across the entire pipeline
• Use JSON for logs—fewer fields but more structured and robust
• Use Filebeat Autodiscover—eliminates manual YAML maintenance
• Monitor collection volume—Filebeat's event.published_time helps detect data loss
• Plan before rolling out: canary testing and manual pipeline load testing are the fastest safeguard