← Back to Blog
DevOps2026-03-2614 min read

Kubernetes Secrets Management: Complete Guide for Production

Kubernetes Secrets are the standard way to deliver sensitive data to your pods — but the defaults are insecure. This guide covers everything from basic secret creation to production-grade patterns using External Secrets Operator, Sealed Secrets, HashiCorp Vault, and AWS Secrets Manager.

What Is a Kubernetes Secret?

A Kubernetes Secret is an API object that stores sensitive data such as passwords, tokens, and TLS certificates separately from your Pod spec. Unlike hardcoding values in environment variables inside a Deployment YAML, Secrets let you inject credentials at runtime and control access with RBAC.

There are several built-in Secret types:

  • Opaque — arbitrary user-defined key/value data (the default type)
  • kubernetes.io/tls — TLS certificates and keys
  • kubernetes.io/dockerconfigjson — credentials for a private container registry
  • kubernetes.io/service-account-token — automatically mounted service account tokens
  • kubernetes.io/basic-auth — username and password pairs
  • kubernetes.io/ssh-auth — SSH private keys

For most application secrets — database passwords, API keys, JWT signing keys — you will use Opaque.

The Base64 Encoding Gotcha

The most common surprise for developers new to Kubernetes Secrets is that the values stored in a Secret YAML manifest are base64-encoded, not encrypted. Base64 is an encoding scheme, not a security measure. Anyone with access to the YAML file or the cluster API can trivially decode the values.

# Encode a value for use in a Secret manifest
echo -n 'mysecretpassword' | base64
# Output: bXlzZWNyZXRwYXNzd29yZA==

# Decode it back
echo 'bXlzZWNyZXRwYXNzd29yZA==' | base64 --decode
# Output: mysecretpassword

This means you should never commit Secret YAML manifests to version control without additional encryption (more on Sealed Secrets and SOPS below). The base64 encoding exists purely for transport encoding of binary-safe data, not for confidentiality.

Base64 Encode/Decode Online

Need to quickly encode a value for a Kubernetes Secret, or decode one to verify it? Use our free Base64 tool — client-side, nothing leaves your browser.

Open Base64 Encoder

Creating Secrets with kubectl

Imperative (from literal values)

# Create a generic (Opaque) secret from literal values
kubectl create secret generic db-credentials \
  --from-literal=username=mydbuser \
  --from-literal=password='s3cr3tP@ssw0rd!'

# Create a secret from a file
kubectl create secret generic app-tls \
  --from-file=tls.crt=./server.crt \
  --from-file=tls.key=./server.key

# Create a TLS secret directly
kubectl create secret tls my-tls-secret \
  --cert=./server.crt \
  --key=./server.key

# Create a Docker registry secret
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=myuser \
  --docker-password=mypassword \
  --docker-email=myuser@example.com

Declarative (YAML manifest)

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
data:
  # Values must be base64-encoded
  username: bXlkYnVzZXI=
  password: czNjcjN0UEBzc3cwcmQh
---
# Alternatively use stringData — kubectl will base64-encode it for you
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
stringData:
  username: mydbuser
  password: "s3cr3tP@ssw0rd!"

The stringData field is a write-only convenience. When you kubectl apply this manifest, Kubernetes encodes the values and stores them under data. When you kubectl get secret db-credentials -o yaml, you will only see the base64-encoded data field — stringData is never returned.

Mounting Secrets: Environment Variables vs Files

Once a Secret exists, you can expose it to a Pod in two ways. Each has different security trade-offs.

Method 1: Environment Variables

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - name: app
          image: myapp:latest
          env:
            # Inject a single key from a Secret
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: username
          # Or inject ALL keys from a Secret as env vars
          envFrom:
            - secretRef:
                name: db-credentials

Limitation: Environment variables are visible to all processes in the container and can leak via /proc/<pid>/environ. They are also snapshotted at pod start — if you rotate the Secret in etcd, running pods do not see the updated value until they restart.

Method 2: Volume Mounts (Recommended for Sensitive Data)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      volumes:
        - name: secret-vol
          secret:
            secretName: db-credentials
            defaultMode: 0400  # read-only for owner only
      containers:
        - name: app
          image: myapp:latest
          volumeMounts:
            - name: secret-vol
              mountPath: /etc/secrets
              readOnly: true

Each key in the Secret becomes a file under /etc/secrets/. So db-credentials with keys username and password creates /etc/secrets/username and /etc/secrets/password.

Key advantage: When a Secret is updated in etcd, Kubernetes automatically updates the files in the volume (within the kubelet sync period, typically 60 seconds). Your application can watch the file and reload credentials without a pod restart — this is essential for secret rotation.

The Security Problem: etcd Is Not Encrypted by Default

Kubernetes stores Secrets in etcd, its distributed key-value store. By default, the data in etcd is stored in plain text (base64 is still just encoding, remember). This means anyone with access to the etcd backup files or the etcd API can read all your secrets.

To enable encryption at rest, you configure an EncryptionConfiguration on the API server:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}  # fallback for reading unencrypted secrets

You then restart the API server with --encryption-provider-config=/etc/kubernetes/encryption-config.yaml. After enabling, existing Secrets are still unencrypted — you must re-write them: kubectl get secrets --all-namespaces -o json | kubectl replace -f -.

Managed Kubernetes services like EKS, GKE, and AKS offer envelope encryption using their respective KMS services (AWS KMS, Google Cloud KMS, Azure Key Vault) which is the recommended approach for production.

RBAC for Secrets

Even with encryption at rest, you need to control who and what can read Secrets. Kubernetes RBAC lets you grant least-privilege access.

# Role that allows reading only the db-credentials secret in the production namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: db-secret-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-credentials"]  # restrict to specific secret by name
    verbs: ["get"]
---
# Bind it to a service account used by your app
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-reads-db-secret
  namespace: production
subjects:
  - kind: ServiceAccount
    name: myapp-sa
    namespace: production
roleRef:
  kind: Role
  name: db-secret-reader
  apiGroup: rbac.authorization.k8s.io

Important: Avoid using ClusterRoleBindings for secret access unless absolutely necessary. Scope secret access to the specific namespace and specific secret name using resourceNames. Also avoid granting list or watch verbs on secrets — a pod that can list all secrets in a namespace can read every one of them.

External Secrets Operator (ESO)

The External Secrets Operator is the de facto standard for syncing secrets from external secret stores (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault, and others) into Kubernetes Secrets automatically. It runs as a controller in your cluster and keeps Secrets in sync.

# Install ESO via Helm
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace
# 1. Create a SecretStore (defines how to connect to AWS Secrets Manager)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa  # IRSA service account
---
# 2. Create an ExternalSecret (defines which secrets to sync)
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h          # how often to sync
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: db-credentials       # name of the K8s Secret to create/update
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/myapp/db     # path in AWS Secrets Manager
        property: username
    - secretKey: password
      remoteRef:
        key: prod/myapp/db
        property: password

ESO creates and maintains the db-credentials Kubernetes Secret for you. When you rotate the secret in AWS Secrets Manager, ESO automatically updates the Kubernetes Secret within the refreshInterval. Your application picks up the change via the volume mount file update — no pod restart required if coded to watch files.

Sealed Secrets: Encrypting Secrets for Git

Sealed Secrets, from Bitnami, solve the problem of storing Secret manifests in Git. A SealedSecret is a CRD that can only be decrypted by the Sealed Secrets controller running in your cluster. You can safely commit the SealedSecret YAML to a public or private repository.

# Install the Sealed Secrets controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

# Install kubeseal CLI
brew install kubeseal  # macOS

# Create a regular Secret manifest first (do NOT apply it)
kubectl create secret generic db-credentials \
  --from-literal=password='s3cr3tP@ssw0rd!' \
  --dry-run=client -o yaml > /tmp/db-secret.yaml

# Seal it — output is safe to commit
kubeseal --format yaml < /tmp/db-secret.yaml > sealed-db-secret.yaml

# Apply the SealedSecret (the controller decrypts it and creates the Secret)
kubectl apply -f sealed-db-secret.yaml

The generated sealed-db-secret.yaml contains asymmetrically encrypted ciphertext. Only the controller in your specific cluster holds the private key needed to decrypt it. Even if the file leaks, an attacker cannot extract the secret value.

Backup the controller's private key — if you lose it, you cannot decrypt any of your SealedSecrets:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-master-key-backup.yaml

HashiCorp Vault Integration

HashiCorp Vault is a dedicated secrets management platform with features far beyond what Kubernetes Secrets provide: dynamic secrets, automatic lease renewal, audit logging, fine-grained policies, and multiple auth methods. It is the preferred choice for enterprises with complex compliance requirements.

There are two main integration patterns:

Vault Agent Sidecar Injector

Vault's mutating webhook injects a Vault Agent sidecar into your pods. The sidecar authenticates to Vault using the pod's service account token (Kubernetes auth method) and writes secrets to a shared tmpfs volume that the main container reads.

# Annotate your Deployment to enable injection
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "myapp-role"
        vault.hashicorp.com/agent-inject-secret-db.txt: "secret/data/prod/db"
        vault.hashicorp.com/agent-inject-template-db.txt: |
          {{- with secret "secret/data/prod/db" -}}
          export DB_PASSWORD="{{ .Data.data.password }}"
          {{- end -}}
    spec:
      serviceAccountName: myapp-sa
      containers:
        - name: app
          image: myapp:latest

The secret is written to /vault/secrets/db.txt inside the container. Your app can source this file or read it directly. The agent continuously renews leases and re-writes secrets when they are rotated.

Vault via External Secrets Operator

ESO also supports Vault as a backend, which keeps your application code completely unaware of the secrets source. The ESO controller handles Vault authentication and syncs secrets into standard Kubernetes Secrets. This is the cleaner architecture for teams already using ESO.

AWS Secrets Manager with IRSA

On EKS, the cleanest approach is using IAM Roles for Service Accounts (IRSA) with the External Secrets Operator. Your ESO pod assumes an IAM role that has secretsmanager:GetSecretValue permission, without any long-lived credentials stored in the cluster.

# IRSA IAM policy (attach to the ESO service account role)
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789:secret:prod/myapp/*"
    }
  ]
}

# Annotate the ESO service account with the IAM role ARN
kubectl annotate serviceaccount external-secrets-sa \
  -n external-secrets \
  eks.amazonaws.com/role-arn=arn:aws:iam::123456789:role/eso-secrets-role

This approach means no AWS credentials live anywhere in your cluster. The Pod Identity mutating webhook injects a temporary token at pod creation time, and the AWS SDK in ESO exchanges it for temporary STS credentials automatically.

Secret Rotation

Secret rotation is one of the most operationally challenging aspects of secrets management. Here is a practical approach:

  • Rotate at the source first: Update the secret value in AWS Secrets Manager, Vault, or wherever it is stored.
  • ESO syncs automatically: Within the refreshInterval, ESO updates the Kubernetes Secret in etcd.
  • Volume mounts update automatically: Kubelet propagates the change to all pods mounting the Secret as a volume within the kubelet sync period (default 60s).
  • Application must reload: Your application needs to watch for file changes or periodically re-read credentials from the mounted file. Many database connection pool libraries support this natively.
  • Env var pods need restart: Pods using environment variables from Secrets do NOT auto-update. You must trigger a rolling restart: kubectl rollout restart deployment/myapp.

For zero-downtime database password rotation specifically, use a dual-credential pattern: create a new password, update the app to accept both old and new, rotate the database credential, then remove the old password reference.

Production Best Practices Summary

  • Never commit plain Secret YAML to Git — use Sealed Secrets, SOPS, or generate secrets outside CI from a secure store.
  • Enable etcd encryption at rest — use envelope encryption with KMS on managed clusters (EKS, GKE, AKS).
  • Use External Secrets Operator in production — sync from AWS Secrets Manager, Vault, or GCP Secret Manager. This is the source of truth, not etcd.
  • Prefer volume mounts over env vars for sensitive secrets — they support live rotation and are harder to leak accidentally via debug outputs.
  • Apply RBAC with resourceNames — restrict pod access to only the specific secrets they need, using the narrowest possible verbs.
  • Set automountServiceAccountToken: false on pods that do not need to call the Kubernetes API.
  • Audit secret access regularly — enable Kubernetes audit logging and alert on unexpected get operations against secrets.
  • Set a rotation schedule — rotate all production secrets on a cadence (quarterly minimum), and immediately after any team member departure.
  • Use namespaces to isolate secrets per environment — dev, staging, and production secrets should never share a namespace.
  • Back up Sealed Secrets controller keys or your Vault unseal keys — losing these means you cannot decrypt your manifests or unseal your Vault.

Debugging Secrets

# List secrets in a namespace
kubectl get secrets -n production

# Describe a secret (shows metadata, NOT the values)
kubectl describe secret db-credentials -n production

# Decode and view a secret value (use carefully — output to terminal)
kubectl get secret db-credentials -n production \
  -o jsonpath='{.data.password}' | base64 --decode

# Check if a pod can access a secret (impersonation)
kubectl auth can-i get secret/db-credentials \
  --namespace production \
  --as system:serviceaccount:production:myapp-sa

# Check ESO ExternalSecret sync status
kubectl get externalsecret db-credentials -n production
kubectl describe externalsecret db-credentials -n production

# Force ESO to re-sync immediately
kubectl annotate externalsecret db-credentials \
  -n production \
  force-sync=$(date +%s) --overwrite

The Bottom Line

Vanilla Kubernetes Secrets are a starting point, not a production-grade solution. For any serious workload, the recommended stack is: AWS Secrets Manager / HashiCorp Vault as the source of truth + External Secrets Operator to sync into the cluster + volume mounts for live rotation + RBAC with resourceNames for access control + etcd encryption at rest.

If you need to store Secret manifests in Git for GitOps workflows, Sealed Secrets gives you a straightforward path. For base64 encoding secret values manually, use our free Base64 tool. Explore our other DevOps guides and 70+ free developer tools.