Secrets Management in CD Environments with GitOps and SOPS
![Secrets Management in CD Environments with GitOps and SOPS Secrets Management in CD Environments with GitOps and SOPS](/_nuxt/image/a07868.png)
![Secrets Management in CD Environments with GitOps and SOPS Secrets Management in CD Environments with GitOps and SOPS](/_nuxt/image/32d461.png)
![Secrets Management in CD Environments with GitOps and SOPS | Platform Engineers](/_nuxt/image/efa214.png)
Modern software development practices heavily rely on continuous integration and continuous delivery (CI/CD) pipelines for automated deployments. These pipelines automate various tasks, including building, testing, and deploying applications. However, a critical challenge in CD environments is managing secrets securely. Secrets, such as API keys, passwords, and database credentials, are essential for application functionality, but their exposure can lead to security vulnerabilities.
SOPS is a text file editor that automates the encryption and decryption of files. It supports various encryption methods, including GPG, AWS KMS, GCP KMS, AGE and HashiCorp Vault. SOPS integrates with VSCode and other editors, allowing users to edit encrypted files seamlessly.
What is GitOps?
GitOps is a DevOps practice that uses Git as the single source of truth for infrastructure and application configurations. It enables developers to manage infrastructure and applications using familiar tools and workflows. GitOps workflows typically involve using a CI/CD pipeline to automatically deploy changes to infrastructure and applications based on changes to Git repositories.
Approaches for secret Management
When it comes to Continuous Deployment (CD), managing secrets securely is crucial. While GitOps and SOPS are popular approaches, there are other methods worth exploring. In this post, we'll discuss the importance of secrets management and alternative strategies using Vault and cloud providers. There are several approaches to secret management beyond HashiCorp Vault and cloud provider-specific solutions:
- External Secrets Operator (ESO): This is an open-source Kubernetes operator that integrates with external secret stores like AWS Secrets Manager, Google Cloud Secret Manager, and Azure Key Vault. It provides a user-friendly abstraction for managing secrets across multiple environments.
- Hashicorp Vault: HashiCorp Vault is a comprehensive secret management solution that provides granular access controls, audit logs, and dynamic secret generation. It is widely used but can be complex to set up and maintain.
- In-house Tools: Some organizations develop their own in-house tools for managing secrets and configurations. These tools can be tailored to the specific needs of the organization and can integrate with various secret management solutions.
We chose to use SOPS over other secret management options because of its unique approach to encryption and version control. Unlike other solutions that encrypt both the secret values and keys, SOPS only encrypts the values, leaving the keys in plain text. This allows us to track changes to secrets in our Git repository, ensuring that all environment files have the same keys.
This approach also simplifies the management of secrets across different environments, as the keys remain consistent. SOPS integrates seamlessly with our existing Git workflow, making it easy to adopt and maintain. By using SOPS, we can ensure the security of our secrets while also maintaining transparency and control over changes to those secrets.
What is SOPS?
SOPS is an open-source tool that enables developers to encrypt secrets in Git repositories. It uses AWS KMS, GCP KMS, or Azure Key Vault to encrypt secrets and stores the encrypted secrets in Git. SOPS also enables developers to decrypt secrets using the same key management service.
Secrets management is crucial for ensuring the security and integrity of sensitive data and systems within an organization. Here are some key reasons why secrets are important and what can be done for effective secret management:
- Preventing Data Breaches: Proper secrets management helps avoid costly data breaches by ensuring that sensitive information, such as passwords, keys, and tokens, are securely stored and transmitted. This minimizes the risk of unauthorized access to critical systems and data.
- Protecting Critical Assets: Secrets management safeguards critical assets, including user and application accounts, devices, and other network elements, from intrusions and unauthorized access.
- Compliance with Regulations: Effective secrets management helps organizations meet the requirements of security regulations and standards, such as NIST, FIPS, and HIPAA, by ensuring secure storage, transmission, and auditing of secrets.
Why use SOPS with GitOps?
Using SOPS with GitOps provides several benefits: SOPS encrypts secrets before they are committed to Git, ensuring that they are not stored in plaintext. Git provides version control for secrets, enabling developers to track changes and roll back to previous versions if necessary. SOPS enables developers to manage secrets using familiar Git workflows, reducing the complexity of secrets management.
How Does It Work?
SOPS encrypts files automatically when they are written and decrypts them when they are read. It supports various encryption methods, making it highly customizable. SOPS can be used with text files or structured data such as YAML, JSON, INI, and ENV files.
How to use SOPS with GitOps
To use SOPS with GitOps, we will use FluxCD's kustomize-controller. This controller decrypts secrets encrypted with SOPS using in-cluster secrets or cloud provider integrations, ensuring secure management within your GitOps workflow.
SOPS with AWS KMS
Step 1: Install SOPS
1. Download SOPS:
curl -LO https://github.com/getsops/sops/releases/download/v3.7.3/sops-v3.7.3.linux.amd64
2. Copy the Downloaded file to local/bin:
mv sops-v3.7.3.linux.amd64 /usr/local/bin/sops
3. Make SOPS Executable:
chmod +x /usr/local/bin/sops
Step 2: Create a KMS Key
1. Sign in to AWS Management Console: Open the AWS Management Console and navigate to the KMS console.
2. Create a KMS Key:
-
Choose "Create key".
-
Enter an alias for the key (e.g., "sops-key").
-
Choose "AWS KMS generates the key material".
-
Add the AWS users or roles that will manage the key.
-
Set the key usage permissions.
-
Review and create the key.
Step 3: Configure SOPS
Create a Configuration File: Create a file ~/.sops.yaml
with the following content:
creation_rules:
- kms: arn:aws:kms:{region}:{account-id}:alias/{alias}
Replace {region}
, {account-id}
, and {alias}
with your AWS region, account ID, and the alias of your KMS key.
Step 4: Encrypt Secrets
1. Create a YAML File with Secrets: Create a file secrets.yaml
containing your secrets.
2. Encrypt the File:
sops --encrypt --kms arn:aws:kms:us-east-1:001301278159:key/b3f4dd5b-a217-46b5-aef2-152fa66be8f4 --encryption-context Role secrets.yaml
Step 5: Configure FluxCD
1. Create a Kubernetes Secret: Create a Kubernetes secret with the PGP keys on each cluster.
2. Add GitRepository and Kustomization Manifests: Add the GitRepository and Kustomization manifests to the fleet repository.
3. Configure Decryption: Configure the Kustomization manifest to use SOPS for decryption:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: my-secrets
namespace: flux-system
spec:
interval: 10m0s
sourceRef:
kind: GitRepository
name: my-secrets
path: ./
prune: true
decryption:
provider: sops
secretRef:
name: sops-gpg
Step 6: Deploy and Sync
1. Commit and Push Changes: Commit the encrypted secrets and manifests to the Git repository.
2. Sync with FluxCD: FluxCD will pull the changes from the Git repository, decrypt the secrets using SOPS and AWS KMS, and apply them to the cluster.
By following these steps, you can securely manage and deploy Kubernetes secrets using SOPS and AWS KMS with FluxCD.
SOPS with AGE encryption
Step 1: Generate an AGE Key
Generate an AGE key: Use the age-keygen
command to create a key pair. This command generates a public and private key.
age-keygen -o age.agekey
Step 2: Configure SOPS
Create .sops.yaml file: This file configures SOPS to use the AGE key for encryption and decryption.
creation_rules:
- encrypted_regex: '^(data|stringData)$'
age: < public_key >
Step 3: Encrypt Secrets
Encrypt a Kubernetes secret: Use SOPS with the AGE public key to encrypt a secret.
sops --age=< public_key > --encrypt --encrypted-regex '^(data|stringData)$' --in-place basic-auth.yaml
Step 4: Create a Secret in Kubernetes
Create a secret in Kubernetes: Use the AGE private key to create a secret in the flux-system
namespace.
cat age.agekey | kubectl create secret generic sops-age --namespace=flux-system --from-file=age.agekey=/dev/stdin
Step 5: Configure Flux
Update the Flux Kustomization: Add the decryption configuration to the Flux Kustomization.
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 10m0s
path: ./cluster/base
prune: true
sourceRef:
kind: GitRepository
name: flux-system
validation: client
decryption:
provider: sops
secretRef:
name: sops-age
Step 6: Verify and Use
Verify and use the encrypted secrets: Flux will now use SOPS to decrypt the secrets in the Kubernetes cluster.
This setup ensures that your secrets are encrypted and decrypted securely using AGE and SOPS, and Flux can manage these secrets in your GitOps workflow.
SOPS with Text Files
Install SOPS and Age: Install SOPS and Age on the machine running Flux. You can get the binaries from their GitHub release pages.
Create a PGP Key: Generate a PGP key using OpenGPG. Make sure not to specify a passphrase if you plan to use this key with Flux.
Configure SOPS: Create a .sops.yaml
file in your Git repository to specify the encryption settings. This file defines the rules for encrypting and decrypting files.
Encrypt Files: Use the sops
command to encrypt your text files. For example, you can encrypt a file secret.yaml
using sops -e -i secret.yaml.
Store Encrypted Files: Store the encrypted files in your Git repository.
Configure Flux: Update your Flux Kustomization to use SOPS for decryption. This involves specifying the decryption
provider as sops
and referencing the secret containing the decryption key.
Flux will automatically decrypt the secrets when reconciling the cluster state. This ensures that the encrypted secrets are decrypted and applied to the cluster only when needed.
By following these steps, you can securely manage and deploy encrypted text files using SOPS and FluxCD.
SOPS with Structured Data
To use SOPS with structured data using Fluxcd, you can follow these steps:
1. Configure SOPS:
- Install SOPS and age using their documentation.
- Generate a key using
age-keygen
and create a.sops.yaml
file with the public key.
2. Encrypt Secrets:
Encrypt your Kubernetes secrets using SOPS. For example, you can encrypt a secret file like this:
apiVersion: v1
kind: Secret
metadata:
name: mysqlcreds
type: Opaque
data: null
stringData:
DB_USER: bugs
DB_PASSWORD: bunny
Encrypt this file using SOPS:
sops -e mysqlcreds.yaml > mysqlcreds-encrypted.yaml
3. Integrate with Flux:
Configure Flux to use SOPS for decryption. In your gotk-sync.yaml
file, add the decryption
property:
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: flux-system
spec:
interval: 10m0s
path: ./clusters/my-cluster
prune: true
sourceRef:
kind: GitRepository
name: flux-system
decryption:
provider: sops
secretRef:
name: my-private-key
Ensure the secret my-private-key
is created correctly and contains the private key.
4. Apply Encrypted Secrets:
- Push the encrypted secret file to your Git repository.
- Flux will pick up the change and reconcile the cluster. The encrypted values should be decrypted and applied correctly.
If decryption does not work, check the Flux logs
using flux logs to identify any issues.
By following these steps, you can effectively use SOPS with structured data using Fluxcd for secure and automated secret management in your Kubernetes cluster.
SOPS with HashiCorp Vault
HashiCorp Vault is a secrets manager that provides secure storage and retrieval of secrets. SOPS can be used with HashiCorp Vault for encryption and decryption.
A Short Introduction to VaultHashiCorp Vault is a secrets manager that provides secure storage and retrieval of secrets. It can be used for encryption, decryption, and signing of data. Vault can be used with various backends, including AWS KMS, GCP KMS, and local file systems.
To manage Kubernetes secrets using SOPS with HashiCorp Vault and Flux, follow these steps:
1. Configure HashiCorp Vault:
- Set up HashiCorp Vault as your identity-based secrets and encryption management system.
- Enable the transit backend in Vault to use it for encryption.
2. Encrypt Secrets with SOPS: Export the VAULT_ADDR
and VAULT_TOKEN
environment variables to your shell.
Use sops
to encrypt a Kubernetes Secret, specifying the Vault address and token:
sops --hc-vault-transit $VAULT_ADDR/v1/sops/keys/my-encryption-key --encrypt \
--encrypted-regex '^(data|stringData)$' --in-place basic-auth.yaml
3. Create a Secret for the Vault Token: Create a secret for the Vault token using kubectl
:
echo $VAULT_TOKEN | kubectl create secret generic sops-hcvault \
--namespace=flux-system --from-file=sops.vault-token=/dev/stdin
4. Set Up Flux Kustomization: Set the decryption secret in the Flux Kustomization to sops-hcvault
.
5. Authorize Kustomize Controller: Configure the kustomize-controller to use workload identity by adding patches to the flux-system kustomization.yaml file.
6. Integrate with Flux:
- Flux supports decrypting secrets stored in Flux sources using SOPS without additional controllers.
- Flux can synchronize secrets from Vault to a Kubernetes secret using the Vault CSI provider, allowing applications to reference the secret without refactoring.
By following these steps, you can securely manage Kubernetes secrets using SOPS with HashiCorp Vault and Flux.
Conclusion
When using cloud services, it is generally recommended to utilize the cloud provider's key management service for better access control and management of encryption keys. For instance, Google Cloud offers Cloud Key Management Service (Cloud KMS), which allows users to create and manage customer-managed encryption keys (CMEKs) for various Google Cloud services and applications. Similarly, Azure provides Azure Key Vault, which supports customer-managed keys (CMKs) and offers different options for storing and managing keys in the cloud.
If the cloud provider does not offer a key management service, alternatives include using Advanced Encryption Standard (AES) encryption or deploying an independent vault. This approach provides full control over encryption keys and can be used in various environments.
Tools like SOPS can be used to manage secrets securely and scalably in a GitOps environment. SOPS supports multiple encryption methods, including AWS KMS, GCP KMS, and HashiCorp Vault, making it highly customizable.