Secrets GSM GKE

Andi Lamprecht ·2026-05-11· 3 min read· Accepted

References

Managing secrets in GKE

Context and problem statement

As Google Secret Management (GSM) was selected for storing/maintaining and providing secrets for the applciations and infrastructure, the Engineering organization requires an approach for managing secrets in GKE as a standard workload engine. Few options have been reviewed to increase productivity and developer experience, also to improve speed of a feature delivery to the end user, which will result in reduced cost of each individual drone delivery.

Decision drivers

• Scalability
• Maintainability
• Automation
• Security
• GCP-native
• Cost
• Management overhead
• Operational complexity

Decision Outcome

PE will use Kubernetes Secret Store CSI Driver to integrate with the Google Secret Manager service offered by GCP to provide a solution to Engineering for managing their deployment secrets securely as determined in DEVOPS-1915.

Consequences

• Requires implementation work to replace Doppler
• Requires prioritization and effort to structure Google Secret Manager prior to or during implementation efforts
• Cost consideration of GSM - not a result of this specific decision, but is a result of choosing GSM
• Promotes usage of Workload Identity
• Fine-grained access control, leading to removal of read-access to promote better security
• Fine-grained access control, allowing write access to others for all environments
• Continues to require RBAC at the GKE cluster to enforce that production secrets cannot be read using Kubernetes secrets commands
• The chosen option is not explicitly GCP-supported

Considered Options

1. Berglas

   • Recommended by Berglas to not use Berglas
   • Not actively maintained as of 31JUL2023
   • Integrates well with GCP services
   • Not supported by GCP
   • Yet another dependency outside of GKE

2. GSM API

   • Requires using a separate SDK resulting in significant transformative requirements
   • More code to maintain

3. Kubernetes Secret Store CSI Driver

   • Kubernetes native
   • Abstracts secret store
   • Possibility to reload secrets without restarting pods
   • OSS driver requires installation and maintenance

4. Init Container

   • Yet another required pod
   • Every workload requires this added to manifest

5. Kubernetes-External-Secrets

   • Fetches secrets on behalf of all workloads, lacking fine-grained audit access
   • OSS driver requires installation and maintenance
   • Kubernetes native
   • Abstracts secret store

Invalid Image Path

Links

• Medium - Consuming Google Secret Managers Secrets in GKE
• GCP - Access secrets stored outside GKE clusters using Workload Identity
• GCP - Use Secret Manager with other products
• Berglas GitHub
• Access Secrets from Secret Manager using Berglas
• Bridging the Gap: Leveraging Secret Store CSI Drivers to Access Secrets from Google Secret Manager in GKE Cluster
• GCP - Secret Manager best practices
• Using the Compute Engine persistent disk CSI Driver
• Kubernetes and Secrets Management in Cloud: Part 2
• kube-secrets-init