Container Secrets

Secure Secret Management in Containerized Applications¶

Secrets such as API keys, passwords, and certificates must be protected throughout their lifecycle in containerized systems. While Kubernetes Secrets and environment variables provide basic mechanisms, they require additional safeguards to prevent exposure. HashiCorp Vault offers a more robust solution by centralizing secret storage and enabling dynamic access. Below are best practices for secure secret management.

1. Kubernetes Secrets: Base64 Encoding with Additional Safeguards¶

Kubernetes Secrets are stored as base64-encoded strings, which are not inherently secure. Use them in combination with other strategies:

Best Practices¶

Avoid plaintext exposure: Never embed secrets directly in Dockerfiles or Kubernetes manifests. Instead, use kubectl create secret to store them securely.
Use secret volumes: Mount secrets as files in a Pod using a secretVolume, limiting access to specific processes.
Limit scope: Assign secrets to minimal pods or services via Role-Based Access Control (RBAC).
Rotate secrets: Automate rotation using tools like kubeseal or external secret managers.

Example: Mounting a Secret as a File¶

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: my-image
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
  volumes:
  - name: secret-volume
    secret:
      secretName: my-secret

2. HashiCorp Vault: Centralized, Encrypted Secret Storage¶

Vault provides dynamic secrets, encryption, and fine-grained access control. It is ideal for production environments where secrets must be ephemeral and protected from long-term storage.

Best Practices¶

Use Vault Agent Injector: Automate secret injection into pods via Kubernetes sidecars. This avoids hardcoding credentials in manifests.
Leverage token-based access: Store secrets in Vault as encrypted data, accessible via temporary tokens with lease durations.
Integrate with Kubernetes: Use Vault's Kubernetes auth method to authenticate workloads and retrieve secrets dynamically.
Enable audit logging: Track access to secrets for compliance and monitoring.

Example: Vault Secret Retrieval via Agent Injector¶

apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: vault-secret
spec:
  prefix: /secrets/
  service: vault-secrets:8200

# Retrieve a secret from Vault
curl http://vault-secrets:8200/v1/secret/data/my-key

3. Encrypted Environment Variables¶

Environment variables are convenient but risky if exposed. Encrypt them using tools like AWS KMS, GCP KMS, or open-source solutions like vault-encrypt.

Best Practices¶

Encrypt at rest: Store encrypted variables in environment files or CI/CD pipelines. Decrypt them at runtime using a key management service (KMS).
Avoid plaintext logs: Configure applications to filter or mask sensitive variables in logs.
Use secret management tools: Replace plaintext variables with references to encrypted secrets managed by a central system.

Example: Encrypted Environment Variables with AWS KMS¶

# Encrypt a secret using AWS KMS
aws kms encrypt --key-id alias/my-key --plaintext "my-secret-value" > encrypted.bin

# Decrypt in the container
aws kms decrypt --key-id alias/my-key --ciphertext-blob fileb://encrypted.bin

4. Avoid Hardcoding Secrets¶

Never embed secrets in Dockerfiles, Helm templates, or configuration files. Use external tools to inject secrets during deployment:

CI/CD pipelines: Use secure variables in Jenkins, GitHub Actions, or GitLab CI.
Secrets engines: Deploy HashiCorp Vault or Azure Key Vault as secrets engines in Kubernetes.

Key takeaways¶

Kubernetes Secrets should be used with additional safeguards like encryption and access controls.
HashiCorp Vault provides dynamic, encrypted secret management and is ideal for production workloads.
Encrypted environment variables reduce exposure but require integration with KMS for decryption.
Always avoid hardcoding secrets in code or manifests; use external secret management tools.