Secret Management

Andi Lamprecht ·2026-05-11· 4 min read· Accepted

Secret Management Solution

Context and problem statement

Providing a quick and secure way for engineers to configure secrets for an application will improve speed of development lifecycle and product delivery to the end user, as a result it will reduce cost of each delivery.

We need to implement a secure and scalable approach for storing/rotating/accessing sensitive configuration for various applications.
Doppler was used as the main secret management solution, but we started reaching the current plan limits which would require switching us to Enterprise subscription, leading to increased usage cost, or discovering different solution. In an effort to support “Cheaper, Farther, Faster” in Q32023, Platform Engineering is working to create a solution to support sunsetting Doppler no later than end of Q12024.

The goal is to design the Secret management solution compatible with our GCP native capabilities.

Decision drivers

• Scalability
• Maintainability
• Automation
• Security
• GCP-native
• Cost

Decision Outcome

We are going to iterate on the solution in stages to make sure that current capabilities and dependencies will be provided, for the cases when engineers need to be able to update secrets manually, because of a lack of automation to support zero-secret approach (when all secrets are non-human generated and saved into a secret management system without a need for engineer to copy and store data locally).

Vision 1 shows current state, where engineers have read/write access into secrets in lower environments and only PE team can access production and cicd pipeline environments to add/update/delete a secret. It creates a dependency on PE team, which isn’t a scalable solution.

Next stage (Vision 1.5) is focused on providing all of Engineering (including PE!) with read/write permissions in non-Production environments and write permission in Production, similar to and has feature parity with native Doppler capabilities. By accounting for local and mobile development, Engineering teams will be empowered to read their secrets from their league-aligned instance of Google Secret Manager in non-Production and not CI/CD pipeline or non-League secrets.

Next stage (Vision 2) is focused on providing only write-only permissions in all environments. Significant work is required to account for local and mobile engineering efforts to achieve this.

Last stage (Vision 3) will prevent humans from reading or changing secrets with the exception of using break-glass permissions. This way the focus will be on improving zero-secret capability across our tools and providing a contract on how new tool/capability that being explored by teams can be covered with similar approach.

Google Secret Manager (GSM) service provides the following list of capabilites that will allow secure, audited and controlled approach for managing secrets:

• Integration with various GCP services:

  • GKE (approach TBD)
  • Cloud Functions
  • Cloud Composer
  • Cloud Run

• Encryption by default (GSM manages encryption at-rest and only accepts connection over HTTPS);

• Audit logging (every secret update will be captured in GCP Logging with a details about who did a change and when it occure);

• Versioning (GSM saves previous states of the secret, when it is updated, that will allow roll back with break glass permission);

• IAM (access will be controlled by GCP groups and provided to the engineers automatically, when they join company, that reduces cost and effort on on-boarding activities).

Consequences

GSM doesn’t support folder structure to separate access for different workloads across folders and environments. Google recommendation is to use the strict naming convention that will provide a level of separation for the pipeline and applications to read necessary secrets.

On highest level secrets will be separated by GCP project, means league and environment. Each league has corresponding prod and dev (non-prod) GCP project where all infrastructure will be hosted as well as secrets.

Naming convention should include repository, environment and variable name to be able to destinguish secrets across different applications and environments.

Suggested naming convention is:

repo_name-env_name-var_name

with - as delimiter between fields

var_name - in uppercase and underscores

env_name - in lowercase and underscores

repo_name - depends on repository name (preferably lower case and dashes)

Considered options

1. Google Secret Manager
2. Doppler
3. 1Password

Links

• Miro board