Secrets management

How a secret in the repo leaks

The classic incident is not a break-in. Someone commits a .env with a live database password, pushes, and the key is now in the git history of every clone and fork. Deleting it in the next commit changes nothing: the old commit is still there, and the platform keeps it in caches you cannot reach.

Docker makes this worse quietly. A secret placed with ENV or ARG, or copied in as a file, becomes part of an image layer, so anyone who can pull the image or run docker history reads it back. Build logs and crash dumps catch secrets the same way.

Environment variables are the usual "fix," and they do beat hardcoding, but they are not a boundary. The value is still one long-lived shared secret, readable through a process listing, a stack trace, a child process that inherits the environment, or a logging line that prints the whole config.

Wrong "It is in an environment variable, not in the code, so it is safe." Right an env var relocates the same permanent secret. It can still leak, and every service that reads it holds the identical credential, so one leak burns them all. The real reductions come later in this note: fetch secrets at runtime, and prefer credentials that expire.

A secret that reached a shared remote or a shared log is compromised. Not "probably." You cannot prove who read it, so you revoke and rotate it, then clean up.

Envelope encryption: encrypt data with a data key, wrap the data key

You will meet KMS the first time you try to encrypt something larger than a config value. AWS KMS Encrypt only takes 4096 bytes of plaintext, because a KMS key is meant to wrap other keys, not to encrypt your files directly.

The pattern that scales is envelope encryption. Ask KMS for a data key, get back two things, and use them in a specific order:

GenerateDataKey(KeyId, KeySpec=AES_256)
  -> Plaintext:      the raw data key (DEK), use it locally
  -> CiphertextBlob: the same DEK encrypted under the KMS key (the KEK)

encrypt your 20 MB document with the plaintext DEK   # locally, not in KMS
wipe the plaintext DEK from memory
store CiphertextBlob next to the ciphertext          # the "envelope"

To read it back, you send only the CiphertextBlob to KMS Decrypt, get the plaintext DEK, decrypt the document, and wipe the key again. The bulk data never travels to KMS; only the small wrapped key does.

Two terms interviewers expect: the DEK (data encryption key) encrypts your data, and the KEK (the KMS key, a wrapping key) encrypts the DEK. The DEK's plaintext form lives only briefly in your process; the durable copy on disk is always encrypted.

Wrong "Just call KMS Encrypt on the whole payload." Anything past 4096 bytes fails, and even if it fit, you would be making a network round trip per object and paying KMS to move your data. Envelope encryption keeps the heavy work local and sends KMS only the key.

Fetch secrets at runtime, and prefer dynamic ones

Instead of shipping secrets inside the build, the service fetches them at startup from a secrets manager (Vault, AWS Secrets Manager, GCP Secret Manager) using its own identity. The secret lives in one managed place with access control and an audit trail, and it is never in the image or the repo.

The stronger version is a dynamic secret. Rather than storing one shared database password, Vault's database engine generates a fresh credential each time the app asks:

app reads: database/creds/app-role
Vault -> connects to Postgres, creates a unique user + password
      -> returns { lease_id, lease_duration (TTL), lease_renewable }
when the lease TTL expires, Vault drops that database user

Because "dynamic secrets do not exist until read," a stolen backup or leaked config has nothing to steal, and each credential is tied to a lease with a time-to-live. When the lease expires, Vault revokes it automatically.

Wrong "We keep the DB password in Vault, so it is a dynamic secret." Storing a static value in Vault's key-value store is still a static secret with all the sharing problems; it just moved into a nicer box. Dynamic means Vault generates a new, short-lived credential per request and revokes it on its own.

Rotation, and the first five minutes after a leak

Rotation is changing a secret's value on a schedule so a stolen copy stops working. Yearly, by hand, is the pattern that fails: the window between leak and rotation is the whole point, and a year is a long window. Automate it, and where you can, use short-lived or dynamic credentials that rotate by design so there is little to rotate manually.

Catch secrets before they land. Secret scanning with push protection blocks a commit that contains a recognized key "before it reaches your repository," which is far cheaper than cleaning history afterward.

When a secret does leak, the order is fixed:

1. revoke  - invalidate the exposed credential now
2. rotate  - issue a new one and deploy it
3. delete  - remove the exposed copy (log line, commit, file)

Wrong "I deleted the commit and force-pushed, so we are clean." Removing the copy is step three, not step one. Until you revoke and rotate, the leaked value still works, and forks, clones, and CI caches you cannot reach may already hold it. Delete last, and only after the credential it exposed is already dead.

Workload identity instead of a stored key

The most valuable secret in most pipelines is the one that authenticates the pipeline itself: a long-lived cloud access key sitting in the CI secret store. It never expires, it usually has broad permissions, and it is copied into every job. The senior move is to stop storing it at all and let the workload prove who it is.

On compute this is already normal: an EC2 instance uses an instance role, an ECS task uses a task role, a Kubernetes pod uses a service account bound to a cloud role (IRSA on EKS, Workload Identity on GKE). The runtime hands the process short-lived credentials it never sees at rest. CI is the piece teams forget, and OIDC federation closes it.

The workflow grants itself id-token: write, GitHub's OIDC provider signs a JWT describing the run, and AWS is configured to trust token.actions.githubusercontent.com as an identity provider. The role's trust policy is where the security lives: it pins the token's aud to sts.amazonaws.com and its sub claim to a specific repository and ref, such as repo:octo-org/octo-repo:ref:refs/heads/main or :environment:production. sts:AssumeRoleWithWebIdentity then returns credentials that expire within the hour. Nothing long-lived is stored anywhere.

The failure mode to name in an interview: a trust policy that matches sub with a wildcard, or omits the repo, lets any repository or any branch in that GitHub org assume the role. The token is only as good as the condition you check it against, so the sub and aud conditions are the actual control, not the federation itself.

Envelope encryption and what key rotation does not do

A KMS key is a wrapping key. It rarely encrypts data directly; it encrypts the data keys that encrypt data. That framing explains KMS rotation, which trips people up.

When you enable automatic rotation, KMS generates new key material on a schedule (default 365 days, configurable, with on-demand rotation available any time). Two properties matter. The KMS key is the same logical resource: its key ID and ARN do not change, so applications and stored ciphertext need no updates. And KMS retains every previous version of the key material, choosing the right one automatically on decrypt, so ciphertext encrypted last year still decrypts.

Wrong "We rotate the KMS key monthly, so a leaked data key is now useless and old data is re-encrypted." The AWS docs are blunt here: rotation "does not rotate the data keys" and "will not mitigate the effect of a compromised data key." Your stored data is not touched, the DEKs that actually protect it are unchanged, and a leaked DEK keeps working until you re-encrypt the data under a fresh one yourself. KMS key rotation limits reuse of the wrapping key; it is not incident response.

One detail worth carrying: an encryption context is non-secret key-value data bound as additional authenticated data. Supply it on encrypt and you must supply the exact same context on decrypt, which lets you tie a wrapped key to, say, a specific tenant so a blob cannot be replayed under another. (The crypto primitives underneath, AES-GCM and AAD, belong to encryption-fundamentals; here the point is operational.)

Vault leases, renewal, and revocation

Every dynamic secret and every service token in Vault carries a lease: metadata with a lease_id, a duration (the TTL), and whether it is renewable. This is what makes dynamic secrets bounded rather than just convenient.

Renewal is a request, not a guarantee. A client asks to extend with an increment, which is the desired remaining TTL measured from now, and the backend may cap or ignore it. So the mental model is not "renew forever"; it is "renew up to whatever ceiling the mount enforces, then the credential dies." When the TTL lapses with no renewal, Vault revokes the secret on its own, dropping the generated database user or cloud credential.

Revocation is also a lever you pull during an incident. vault lease revoke <lease_id> kills one credential immediately and blocks further renewals. Because a lease ID is prefixed with the path it came from, you can revoke a whole blast radius at once:

vault lease revoke -prefix database/creds/app-role   # every cred from this role
vault lease revoke -prefix aws/                       # every dynamic AWS cred

The operational cost is honest: dynamic secrets need Vault reachable on the request path, Vault has to track outstanding leases, and a mass revocation or a flood of short TTLs pushes load onto both Vault and the downstream (Postgres has to create and drop users). You trade a standing shared secret for a dependency and some churn, which is usually the right trade for a credential that grants real access.

Leak response is a lifecycle

The reflex to delete the offending commit is the wrong first step and the tell of someone who has not run an incident. The order is revoke, then rotate, then delete, and the reason is epistemic: once a secret is on a shared remote or in a shared log, you cannot enumerate who pulled, forked, or scraped it, so you assume the value is burned and make it worthless before you tidy up.

Detection has two layers. Push protection stops a recognized secret before it enters history, which is the cheap win. Scanning of existing history and of issues, PRs, and gists catches what already landed; for partner patterns the platform notifies the provider directly so they can revoke the credential out from under an attacker. Neither replaces rotating the key you own.

The part that separates senior answers is propagation. A single leaked database password may have been copied into a read replica's config, a cached CI artifact, a teammate's local .env, and a downstream service that embedded it. Revoking the one you can see does not help if four copies still authenticate. So the response includes finding every place the secret flowed, which is exactly the argument for dynamic, short-lived credentials: there is far less to chase when the credential was only ever going to live for an hour.