Security Logging

Real-time monitoring, logging, and alerting are critical components of securing containerized applications. These practices enable teams to detect anomalies, track security incidents, and respond to threats before they escalate. By combining metrics, logs, and event-based detection, you can create a robust observability layer that complements container security controls like runtime protection and network policies.

---

Real-Time Monitoring Tools

Prometheus + Grafana

Prometheus is a powerful time-series database that collects metrics from containers and hosts. When paired with Grafana, it provides visual dashboards for monitoring resource usage, container health, and security-related metrics.

Example: Monitor Docker container metrics

# prometheus.yml
scrape_configs:
  - job_name: 'docker'
    static_configs:
      - targets: ['localhost:9323']  # Docker stats endpoint

Commands to enable Docker metrics:

# Ensure Docker's metrics endpoint is exposed
echo "DOCKER_OPTS=\"--iptables --ip-forward --enable-cgroup --storage-driver=overlay2\"" | sudo tee -a /etc/default/docker
sudo systemctl restart docker

Falco

Falco is a lightweight, open-source tool that monitors system calls and container events in real time. It excels at detecting suspicious behavior such as unauthorized process executions or filesystem changes.

Example: Falco rule to detect unexpected process execution

# falco.yaml
- rule: Unexpected process execution
  desc: Detect processes that are not whitelisted
  condition: (evt.type = process) and (evt.type = exec) and (container.image != "base_image") and (not (proc.name = "sh" and proc.args = "-c"))
  output: Unexpected process execution: %proc.name
  priority: medium
  tags: container, security

Command to start Falco:

falco --config falco.yaml

---

Logging and Centralized Analysis

ELK Stack (Elasticsearch, Logstash, Kibana)

The ELK stack aggregates logs from containers, hosts, and applications into a centralized repository. It allows for real-time analysis, filtering, and visualization of security-relevant events.

Example: Logstash configuration to parse Docker logs

# logstash.conf
input {
  file {
    path => "/var/lib/docker/containers/*/*.log"
    start_position => "beginning"
  }
}
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "ISO8601" ]
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

Command to tail Docker logs:

docker logs --tail 100 --since 5m <container_id>

Alternative Logging Solutions

• Loki: A lightweight, log aggregation system designed for containerized environments.
• Graylog: A centralized logging platform with advanced search and alerting capabilities.

---

Alerting and Incident Response

Integrating with Alerting Systems

Tools like Prometheus Alertmanager or third-party services (e.g., PagerDuty, Opsgenie) can trigger alerts based on predefined thresholds or anomalies. For example, a spike in CPU usage or a sudden increase in failed login attempts can trigger an alert.

Example: Prometheus alert rule for high CPU usage

# alert.rules.yml
- alert: HighCPUUsage
  expr: (container_cpu_usage_seconds_total{container_label_app!~"base_image"} / container_limits_cpu_cores{container_label_app!~"base_image"}) > 0.8
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High CPU usage in {{ $labels.container }} ({{ $labels.instance }})"
    description: "CPU usage exceeds 80% for 5 minutes."

Correlating Logs and Metrics

Combine logs with metrics to identify patterns. For example, a sudden increase in network traffic (detected via Prometheus) paired with a suspicious process execution (detected via Falco) could indicate a container breakout attack.

---

Key takeaways

• Use Prometheus + Grafana for real-time metrics and visualization of container health and resource usage.
• Deploy Falco for event-based detection of suspicious container behavior.
• Centralize logs with the ELK stack or Loki to enable efficient analysis and correlation.
• Integrate with alerting systems to automate incident response and reduce dwell time for threats.
• Always correlate logs, metrics, and events to gain a holistic view of container security posture.