Secure Kubernetes Secrets Management Guide

Kubernetes Secret objects are base64 encoded, not encrypted. Anyone with read access to the cluster can decode them. Storing them in Git is even worse — they are permanently visible in history. This guide covers the full stack of Kubernetes secrets management from etcd encryption to GitOps-safe patterns.

The Problem with Default Kubernetes Secrets

# Default secret storage — base64 is NOT encryption
kubectl create secret generic my-secret --from-literal=api-key=sk_live_abc123

# Anyone with kubectl access can trivially decode it
kubectl get secret my-secret -o jsonpath='{.data.api-key}' | base64 -d
# sk_live_abc123

Secrets are stored in etcd as base64. If you have etcd access (or a backup), you have the secrets.

Step 1: Encrypt etcd at Rest

Kubernetes can encrypt etcd data using a key you provide. This protects secrets if etcd backups or snapshots are accessed.

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - aescbc:
          keys:
            - name: key1
              # Generate: head -c 32 /dev/urandom | base64
              secret: "$(head -c 32 /dev/urandom | base64)"
      - identity: {}   # Fallback for unencrypted reads during migration

# Add to kube-apiserver flags (kubeadm clusters: /etc/kubernetes/manifests/kube-apiserver.yaml)
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# Restart kube-apiserver (kubeadm restarts it automatically when the manifest changes)

# Verify encryption is working — existing secrets should be migrated
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
# This re-writes all existing secrets through the encryption provider

# Confirm: read directly from etcd (should see encrypted data)
ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key /etc/kubernetes/pki/etcd/healthcheck-client.key \
  | hexdump -C | head -4
# Should show: /registry/secrets/default/my-secret followed by encrypted binary

Step 2: Sealed Secrets for GitOps

Sealed Secrets allows you to commit encrypted secrets to Git. The SealedSecret is decrypted only inside the cluster by the controller.

# Install the controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# Install kubeseal CLI
curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.26.0/kubeseal-0.26.0-linux-amd64.tar.gz | tar xz
sudo mv kubeseal /usr/local/bin/

# Create a regular secret (do not apply this to the cluster)
kubectl create secret generic my-secret \
  --from-literal=api-key=sk_live_abc123 \
  --dry-run=client -o yaml > my-secret.yaml

# Seal it for the cluster
kubeseal --format yaml < my-secret.yaml > my-sealed-secret.yaml

# Now commit my-sealed-secret.yaml to Git — it is safe
# The content is encrypted with the cluster's public key

cat my-sealed-secret.yaml
# apiVersion: bitnami.com/v1alpha1
# kind: SealedSecret
# spec:
#   encryptedData:
#     api-key: AgBy8ELfsd98s...  (RSA-encrypted ciphertext)

# Apply the sealed secret — controller decrypts it automatically
kubectl apply -f my-sealed-secret.yaml

# Verify it created a regular Secret
kubectl get secret my-secret -o jsonpath='{.data.api-key}' | base64 -d

Rotating the Sealing Key

# Back up the sealing key before anything else
kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-key-backup.yaml
gpg -c sealed-secrets-key-backup.yaml  # encrypt the backup

# Rotate the sealing key (creates a new key, keeps old one for decryption)
kubectl label secret -n kube-system sealed-secrets-key sealedsecrets.bitnami.com/sealed-secrets-key=active

# Re-seal all secrets with the new key
for secret in $(find . -name "*.sealed.yaml"); do
  kubeseal --re-encrypt --format yaml < "$secret" > "${secret}.new"
  mv "${secret}.new" "$secret"
done

Step 3: External Secrets Operator

External Secrets Operator (ESO) pulls secrets from AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, etc., and creates Kubernetes Secrets automatically. Secrets are never stored in Git.

# Install ESO
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

# SecretStore — connects to AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

# ExternalSecret — defines what to pull and where to put it
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secrets
  namespace: production
spec:
  refreshInterval: 1h   # pull updates every hour
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: my-app-secret   # creates a Kubernetes Secret with this name
    creationPolicy: Owner
  data:
    - secretKey: api-key         # key in the Kubernetes Secret
      remoteRef:
        key: prod/myapp/stripe   # AWS Secrets Manager secret name
        property: secret_key     # JSON field in the secret
    - secretKey: db-password
      remoteRef:
        key: prod/myapp/database
        property: password

kubectl apply -f secretstore.yaml externalSecret.yaml

# Check sync status
kubectl get externalsecret my-app-secrets -n production
# NAME               STORE                  REFRESH INTERVAL   STATUS
# my-app-secrets     aws-secretsmanager     1h                 SecretSynced

Step 4: Vault Agent Sidecar

For the most granular control, Vault Agent runs as a sidecar and writes secrets to a shared memory volume.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      serviceAccountName: myapp
      # Vault Agent init container — fetches secrets before app starts
      initContainers:
        - name: vault-agent-init
          image: hashicorp/vault:latest
          command: ["vault", "agent", "-config=/vault/config/agent.hcl", "-exit-after-auth"]
          volumeMounts:
            - name: vault-config
              mountPath: /vault/config
            - name: secrets
              mountPath: /vault/secrets
      containers:
        - name: app
          image: myapp:latest
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: myapp-db   # created by Vault Agent template
                  key: password
          volumeMounts:
            - name: secrets
              mountPath: /var/secrets
              readOnly: true
      volumes:
        - name: vault-config
          configMap:
            name: vault-agent-config
        - name: secrets
          emptyDir:
            medium: Memory   # never written to disk

Audit: What Can Access Your Secrets

# List all subjects that can read secrets in production namespace
kubectl auth can-i get secrets --namespace production --list | grep "yes"

# Or with rakkess (more detailed)
kubectl-access_matrix --namespace production

# Who can list ALL secrets cluster-wide (dangerous)
kubectl get clusterrolebindings -o json | jq -r '
  .items[] |
  select(.roleRef.name == "cluster-admin" or
         (.roleRef.name | test("secret"))) |
  "\(.metadata.name): \(.subjects[]?.name)"'

RBAC: Minimum Necessary Access

# Grant a service account read access to only its own secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: myapp-secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["myapp-db", "myapp-api-keys"]  # specific secrets only
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-secret-reader
  namespace: production
subjects:
  - kind: ServiceAccount
    name: myapp
roleRef:
  kind: Role
  name: myapp-secret-reader
  apiGroup: rbac.authorization.k8s.io

Detecting and Remediating Secret Sprawl

Before improving your secrets management architecture, you need to understand the current state. Secret sprawl in Kubernetes manifests in three ways: secrets embedded in ConfigMaps, environment variables injected directly from secret values, and secrets in container image layers. A sprawl audit finds all three.

# Find all secrets across all namespaces
kubectl get secrets --all-namespaces -o json | jq -r '
  .items[] |
  "\(.metadata.namespace)/\(.metadata.name) [\(.type)] keys: \(.data | keys | join(","))"'

# Find pods using secrets as env vars (check for potential over-exposure)
kubectl get pods --all-namespaces -o json | jq -r '
  .items[] | .spec.containers[]? |
  select(.env != null) |
  .env[] | select(.valueFrom.secretKeyRef != null) |
  "\(.name) from \(.valueFrom.secretKeyRef.name)/\(.valueFrom.secretKeyRef.key)"'

# Find ConfigMaps that contain obvious secret patterns (base64 or tokens)
kubectl get configmaps --all-namespaces -o json | jq -r '
  .items[] |
  select(.data != null) |
  "\(.metadata.namespace)/\(.metadata.name)" as $name |
  .data | to_entries[] |
  select(.value | test("password|secret|key|token"; "i")) |
  "\($name): \(.key)"'

For each secret found, assess whether it should be managed by ESO (pulling from AWS Secrets Manager), Vault Agent (dynamic credentials), or Sealed Secrets (GitOps-committed encrypted values). The decision tree is straightforward:

Secret type	Recommended approach
Database credentials	Vault dynamic credentials (short-lived)
API keys from third-party services	ESO + AWS Secrets Manager
TLS certificates	cert-manager + Let’s Encrypt or private CA
Service-to-service tokens	Vault Agent or workload identity
Config values (non-secret)	ConfigMap, not Secret

Migration from raw secrets to ESO is non-breaking — ESO creates a standard Secret object. Your pods do not need to change; only the secret source changes.

Preventing Secrets in Git History

Even with Sealed Secrets or ESO, developers sometimes accidentally commit raw credentials. Add a pre-commit hook that scans for high-entropy strings and known secret patterns:

# Install detect-secrets
pip install detect-secrets

# Initialize a baseline (commits current state as known)
detect-secrets scan > .secrets.baseline
git add .secrets.baseline

# Add pre-commit hook
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
detect-secrets-hook --baseline .secrets.baseline
EOF
chmod +x .git/hooks/pre-commit

For team-wide enforcement use pre-commit framework:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

If a secret has already been committed, assume it is compromised and rotate it immediately before removing it from history:

# Remove a file from all history (requires force push — coordinate with team)
git filter-repo --path secrets.yaml --invert-paths

# Or use BFG (faster for large repos)
bfg --delete-files secrets.yaml
git reflog expire --expire=now --all && git gc --prune=now
git push --force

Removing from history does not help if the secret was already exposed — rotate it first, then clean history for hygiene.