The Practical Developer

Secrets Management For Real Teams: Vault, SOPS, And The .env File You Should Burn

October 27, 2023 by The Practical Developer

Almost every team starts with a .env file in 1Password and ends with secrets in Slack. Here are the three credible options for production secrets — Vault, SOPS-encrypted-in-git, cloud-native (AWS/GCP) — with the trade-offs, the migration paths, and the rotation policy that survives a year.

security

devops

reliability

A laptop on a desk — the right setting for the careful work of secrets management

The new engineer joins the team. Onboarding instructions: “ask Slack for the .env file.” A senior engineer DMs them a 4 KB file with database passwords, API keys, and the Stripe live secret. They paste it into 1Password “for safekeeping.” Six months later the engineer has left, the file is still in their personal vault, and nobody knows where else copies of those secrets live.

This is how every team’s secrets management starts and how most stay. The fix is not “be more careful.” It is to put secrets in a system where the answer to “who has access” is auditable, and where rotation is a button-press, not a Slack thread.

This post is the three credible options for production secrets — HashiCorp Vault, SOPS-encrypted-in-git, cloud-native (AWS/GCP/Azure) — with the trade-offs and the rotation policy that holds up.

What “managing secrets” actually means

Five operations that any system worth using must support:

  1. Store a secret with a name and access policy.
  2. Read it from production at runtime.
  3. Audit who read it and when.
  4. Rotate it without redeploying.
  5. Revoke access when somebody leaves.

A .env file in a chat tool fails 3, 4, and 5. A “secrets manager” that requires a redeploy to rotate fails 4. A system without per-secret ACLs fails 5.

The three options below all support these operations to different degrees.

Option 1: HashiCorp Vault

Vault is the heaviest, most flexible option. Run it as a service. Apps authenticate to Vault and request secrets at runtime.

# App authenticates with a Kubernetes service account.
$ vault read database/creds/app-readonly
Key                Value
---                -----
lease_id           database/creds/app-readonly/abc123
lease_duration     1h
lease_renewable    true
password           A1b-2C3d-...
username           v-token-app-...-abc

Strengths:

  • Dynamic secrets. Vault can generate per-app DB credentials with a TTL of one hour. The credentials are auto-revoked. There is no static “DB_PASSWORD” anywhere.
  • PKI. Vault can issue and rotate TLS certs.
  • Transit engine. Encrypt arbitrary data with Vault as the key manager.
  • Audit log. Every read is logged.
  • Policy-based access. Fine-grained ACLs.

Costs:

  • You run a service. Vault is a stateful, replicated, hard-to-operate distributed system. If Vault is down, your apps cannot start.
  • Learning curve. Auth methods, policies, secret engines — it is a real product to learn.
  • Total cost. HashiCorp’s commercial pricing is non-trivial; the open-source version is free but requires real ops investment.

Vault is the right answer for organizations large enough to have a dedicated security or platform team. It is overkill for a 5-engineer startup.

Option 2: SOPS-encrypted in Git

SOPS (Secrets OPerationS) encrypts the values of a YAML or JSON file using a KMS key, leaving the keys in plaintext. The encrypted file lives in Git like any other config:

# secrets.enc.yaml
database:
  host: db.example.com
  password: ENC[AES256_GCM,data:abc123,iv:...,tag:...,type:str]
stripe:
  api_key: ENC[AES256_GCM,data:def456,...,type:str]
sops:
  kms:
  - arn: arn:aws:kms:us-east-1:...:key/abc-123

Anyone with access to the KMS key can decrypt; the file is otherwise gibberish. Use AWS KMS, GCP KMS, Azure Key Vault, or age for a key-pair-based approach.

Strengths:

  • Lives in Git. No separate service. Audit log is the git history. Diffs are reviewable.
  • Cheap. KMS keys cost cents per month.
  • Simple. Decryption happens at deploy time; runtime is a plain env var.

Costs:

  • No dynamic secrets. Rotation requires a commit + deploy.
  • Encrypted-at-rest only. Once decrypted, the secret is in your build artifacts, your CI logs (if you’re not careful), or your container’s env.
  • No per-app ACLs. Everyone with the KMS key can decrypt everything.

For most teams under ~50 engineers, SOPS hits the sweet spot. The “secrets are in Git” psychology takes getting used to but the encryption is genuinely strong.

Option 3: Cloud-native (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault)

Each cloud has a managed secrets service. They look similar:

aws secretsmanager get-secret-value --secret-id prod/db/password

Strengths:

  • Managed. No service to run.
  • Native integration. EKS pods can mount secrets via CSI drivers; Lambda has built-in support; ECS tasks read at startup.
  • Per-secret ACLs. IAM policies scope access.
  • Audit logs in CloudTrail / Cloud Audit / Activity Logs.

Costs:

  • Per-secret pricing. AWS Secrets Manager is $0.40/secret/month — adds up to real money at scale.
  • Cloud lock-in. A multi-cloud or hybrid setup needs a different approach.
  • Rotation is supported but uneven. AWS RDS rotation works well; rotation for arbitrary third-party APIs is on you.

For teams already deep in one cloud, the cloud-native option is usually the lowest-effort credible answer.

The decision tree

A realistic flowchart:

  1. Do you need dynamic per-app database credentials with TTL? Yes → Vault.
  2. Are you single-cloud and already using KMS / IAM heavily? Yes → cloud-native.
  3. Are you a small team that wants minimum operational surface? SOPS in Git.
  4. Some combination? SOPS for static long-lived secrets (third-party API keys), cloud-native or Vault for high-rotation database credentials.

Most teams start with #3 (SOPS) and migrate to #2 or #1 when scale demands it. Migration is real work; pick something you can live with for a couple of years.

The rotation policy that holds up

Whatever tool you pick, you need a rotation policy. The realistic version:

  • Database passwords: rotate quarterly. With Vault dynamic credentials, this is automatic. With static creds, schedule a maintenance window per quarter.
  • Third-party API keys: rotate when an employee leaves who had access to them, or annually.
  • Cloud IAM keys: prefer instance roles / workload identity; long-lived keys rotated quarterly.
  • Signing keys (JWT, etc.): rotate every 6-12 months with a grace period.
  • TLS certs: automated via cert-manager / Let’s Encrypt; verify the renewal works.

The frequency matters less than the practice of rotating. A team that has rotated zero secrets in two years has secrets they cannot rotate (because they don’t remember which apps use them).

What to do when somebody leaves

The “engineer leaves” checklist is the test of your secrets management. The right answer is:

  1. Revoke their personal access (SSO, GitHub, cloud accounts) — automated.
  2. Rotate any secret they had access to — possibly automated.

If step 2 is “ask around what they had access to,” you have a problem. The secrets-management system should answer “what could this user have read in the last 90 days?” from audit logs.

Things to never put in a secrets manager

A few patterns that look like secrets but aren’t:

  • Configuration that is not sensitive (port numbers, feature flags). Put in plain config files.
  • Public-facing API keys (frontend to identify your account). They are public; don’t pretend.
  • Secrets that change every request (per-user JWTs). Mint at runtime, don’t store.

The secrets manager is for things that are sensitive and long-lived. Anything else belongs elsewhere.

Common antipatterns

Storing secrets in environment variables in CI logs. A set -x in a shell script and you’ve leaked. Use the CI’s secret-injection mechanism (GitHub Actions secrets, GitLab masked variables) and never echo them.

Embedding secrets in container images. Multi-stage builds have helped, but a RUN apt-get && SECRET=abc... baked into a layer means anyone with the image has the secret. Use build secrets (docker build --secret) or runtime injection.

Single shared key for everything. “Master API key” with full access to everything. Should be split: read-only for analytics, write for billing, etc.

No backup of root keys. If you lose your KMS key or Vault root token, secrets become unrecoverable. Have an offline backup with a clearly-documented recovery procedure.

The takeaway

A .env file in 1Password is a starting point, not a destination. Pick a real secrets management system (Vault, SOPS-in-Git, cloud-native) before you have to. Establish a rotation rhythm. Make “who can read this secret” a query, not a Slack thread.

The next time an engineer leaves, the response should be one button-press, not a six-hour audit. The next time you need to rotate a database password, it should be a Friday afternoon, not a sprint.

A note from Yojji

The kind of security hygiene that turns “we know who has access to what” from a wish into a query — secrets management, rotation, audit logs — is the kind of long-haul engineering discipline Yojji’s teams put into the products they hand back to clients.

Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem, cloud platforms (AWS, Azure, GCP), and full-cycle product engineering — including the security and operations work that decides whether your secrets are auditable or scattered across team chats.