Managing Secrets in Kubernetes: Best Practices

In the chaos of deploying applications, managing secrets such as API keys, database passwords, and other sensitive information can easily become a security nightmare. Imagine a scenario where an accidental commit leaks your production database credentials into a public repository—this could lead to catastrophic data breaches.

As organizations move towards containerized deployments at scale, securely handling secrets in Kubernetes becomes increasingly critical. In 2025, with the proliferation of microservices and multi-cloud strategies, ensuring that sensitive information is protected will be paramount for maintaining trust and compliance.

In this blog post, we'll explore best practices for managing secrets in Kubernetes, ensuring your applications remain secure without compromising on performance or developer productivity.

---

Introduction to Secrets Management

Handling secrets securely in cloud-native environments requires a robust strategy. Kubernetes provides native support for managing sensitive information through its Secret resource. However, using it effectively demands careful planning and adherence to best practices.

Kubernetes Secrets allow you to store and manage sensitive data separately from your application code, reducing the risk of exposure. We'll dive into how to create, use, and protect these secrets efficiently.

Understanding Kubernetes Secrets

What are Kubernetes Secrets?

Kubernetes Secrets enable you to securely store and manage sensitive information like passwords, tokens, or keys. These secrets can be mounted as files or exposed as environment variables within your pods.

Types of Secrets

Kubernetes supports different types of secrets:

Creating Kubernetes Secrets

Opaque Secrets Example

To create an opaque secret, you can define it in a YAML file:

# Define the secret object
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: dXNlcm5hbWU= # base64 encoded 'username'
  password: cGFzc3dvcmQ= # base64 encoded 'password'

This YAML file specifies a secret named my-secret containing two key-value pairs.

Using kubectl create secret

Alternatively, you can use kubectl to create secrets directly from the command line:

# Create secret using kubectl
kubectl create secret generic my-secret --from-literal=username='username' --from-literal=password='password'

This command creates a new secret named my-secret with the specified literals.

Mounting Secrets as Volumes

Mounting Example

To mount secrets as files within a pod, you can modify your pod specification:

# Pod definition using secret as volume
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: nginx
      volumeMounts:
        - name: secret-volume
          mountPath: "/etc/secret"
          readOnly: true
  volumes:
    - name: secret-volume
      secret:
        secretName: my-secret

This configuration mounts the my-secret secret into the /etc/secret directory of the container.

Explanation

In this example, the secret my-secret is mounted as a volume in read-only mode at /etc/secret. The keys from the secret (username and password) are stored as files within that directory.

Exposing Secrets via Environment Variables

Environment Variable Example

To expose secrets through environment variables, modify your pod specification:

# Pod definition using secret as environment variable
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: nginx
      env:
        - name: SECRET_USERNAME
          valueFrom:
            secretKeyRef:
              name: my-secret
              key: username
        - name: SECRET_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-secret
              key: password

This configuration sets environment variables SECRET_USERNAME and SECRET_PASSWORD from the my-secret secret.

Explanation

Here, the keys from my-secret are exposed as environment variables in the container. This approach is useful when applications require sensitive information to be passed via environment variables.

Encrypting Secrets with etcd Encryption

Why Encrypt?

By default, Kubernetes stores secrets unencrypted in etcd. Enabling encryption at rest ensures that your secrets remain secure even if someone gains access to the underlying storage.

Configuring Encryption

To enable encryption for secrets, you need to configure a key management service (KMS) provider or use a local provider like AES-CBC:

# Generate an encryption configuration file
cat > encryption-config.yaml <<EOF
kind: EncryptionConfiguration
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: $(head -c32 /dev/urandom | base64)
      - identity: {}
EOF

This configuration file enables AES-CBC encryption for secrets.

Explanation

The encryption-config.yaml file specifies that all secrets should be encrypted using the AES-CBC algorithm. The identity provider allows unencrypted reads and writes, which is useful during a migration to encrypted storage.

Using External Secret Management Tools

Why Use External Tools?

While Kubernetes provides basic secret management capabilities, external tools offer enhanced features like integration with existing secret stores (e.g., HashiCorp Vault), automatic rotation, and fine-grained access control.

Popular Tools

Some popular external secret management tools include:

• HashiCorp Vault: Provides secure secrets management with dynamic secrets, encryption as a service, and identity-based access control.
• AWS Secrets Manager & AWS Systems Manager Parameter Store: Offer secure storage for secrets with integration into the AWS ecosystem.
• Azure Key Vault: Integrates seamlessly with Azure services for managing secrets securely.

Example: HashiCorp Vault Integration

To integrate Kubernetes with HashiCorp Vault, you can use the vault-k8s project:

# Install vault-k8s CLI tool
kubectl krew install vault

# Initialize and configure Vault
vault operator init -key-shares=1 -key-threshold=1 > keys.txt

This setup initializes a new Vault instance with a single key share.

Explanation

The vault-k8s tool integrates HashiCorp Vault with Kubernetes, enabling dynamic secret management. The initialization command generates the necessary keys for accessing the Vault.

Best Practices for Managing Secrets

Regular Rotation

Regularly rotate secrets to minimize the risk of unauthorized access. Automated tools can help manage this process efficiently.

Least Privilege Access

Ensure that only necessary services and users have access to sensitive secrets. Implement role-based access control (RBAC) policies accordingly.

Avoid Hardcoding Secrets

Never hardcode secrets in your application code or configuration files. Use Kubernetes secrets or external secret management tools instead.

Monitor Secret Usage

Implement logging and monitoring for secret usage to detect any unauthorized access attempts. Regularly review audit logs to ensure compliance with security policies.

---

Troubleshooting Common Issues

Issue: Secrets Not Mounting Correctly

Solution: Verify that the secret name matches in both the pod specification and the secrets resource. Ensure proper permissions and volume mount paths.

Issue: Encryption Configuration Errors

Solution: Check the encryption configuration file for syntax errors or invalid keys. Validate that all necessary providers are correctly specified.

Issue: External Tool Integration Failures

Solution: Review the documentation for the external tool you're using. Ensure that all prerequisites are met and configurations are correct.

---

Conclusion

Securing secrets in Kubernetes is essential for maintaining application security and compliance. By following best practices and leveraging native features along with external tools, you can effectively manage sensitive information without compromising performance or developer productivity.

Key Takeaways:

1. Use Kubernetes Secrets to store and manage sensitive data securely.
2. Implement encryption at rest using AES-CBC or an external key management service.
3. Regularly rotate secrets and enforce least privilege access policies.
4. Monitor secret usage and audit logs for unauthorized access attempts.

By adhering to these best practices, you can ensure that your Kubernetes deployments remain secure and resilient in the face of evolving security threats.