Feature flags & progressive delivery

What a feature flag decouples

A feature flag is a runtime switch: a rule set decides, on each evaluation, which code path runs. The payoff is separating two things that used to move together. Deployment puts your code on the servers. Release turns it on for users. Merge behind a flag that defaults off, deploy any day you like while the code sits dormant, then flip the flag when you (or product) decide to expose it. Rolling back a bad release becomes flipping the switch, not shipping a new build.

The pipeline that builds and ships the artifact is a separate concern (that is the ci-cd-pipelines module). Flags are the layer on top that controls who reaches the code once it is already in production.

Wrong "The code went live on Tuesday, so it was released Tuesday." Deploy is not release. Code behind an off flag is in production and exposed to nobody.

Four kinds of flag, four lifespans

Martin Fowler's taxonomy sorts flags by how long they live and how often the decision changes. Interviewers ask for these four by name:

• release toggles hide in-progress work so the trunk stays shippable. Short-lived, a week or two, then deleted.
• experiment toggles route users into A/B variants to measure an effect. They live as long as the experiment needs to reach a significant result.
• ops toggles control operational behavior. Most are short, but some are long-lived kill switches: a flag you flip to shed load or disable a failing dependency mid-incident, with no deploy.
• permission toggles gate a feature by user segment (beta testers, paying customers). These can live for years.

Getting the category wrong bites later. A kill switch someone deleted is missing when the incident hits. A release toggle nobody removed is dead weight in every future test run.

Same user, same variation

A 10% rollout does not flip a coin per request. If it did, a user would get the new checkout on one page load and the old one on the next. Instead the flag system hashes a stable key for the user (their id) and maps it to a bucket. Same key, same bucket, same variation, every time, and the server stores nothing per user. Widen the rollout later and the users already inside it stay inside.

Wrong "A 10% rollout means each request has a 10% chance of the new path." That is a per-request coin flip: the same user flickers between variations across page loads. Bucketing on a stable key is what keeps a user pinned to one variation.

Flags are inventory you pay to carry

Every flag is a branch in your code and a line in your config. A release flag earns its keep for a week or two, then turns into debt: an extra path to test, an extra way to misconfigure. Plan the removal when you plan the rollout. A release flag's life should reach an end:

The failure mode is the flag that never reaches "removed": it sits at full on forever, the old code path rots beside it, and a year later nobody can say whether deleting it is safe.

Reading a canary instead of watching a dashboard

The ci-cd module covers how a rollout ships the artifact. The senior skill is judging one while it runs. Google's SRE workbook defines a canary as "a partial and time-limited deployment of a change in a service and its evaluation," where the unchanged remainder of the fleet is your control.

Two ways teams get the evaluation wrong. The first is before/after: watch the fleet's error rate for an hour before the rollout and an hour after. Time is the dominant confounder, since 3pm traffic does not look like 2pm traffic, and 10% of the errors disappear inside a fleet-wide average. Compare the canary against a concurrent control running the old version in the same window, so the change is the only difference between the two groups.

The second is a bad metric. The signal has to proxy user harm and be attributable to the change. HTTP error codes and request latency qualify. System-wide CPU does not: a batch job on the same host moves it, so it fails the "clearly attributable" test. Start from your SLIs and take the handful that indicate a user is hurting.

Size and duration trade against each other. A smaller canary limits blast radius but takes longer to accumulate a representative sample, and some defects only show under real load or after a cold cache warms. Too strict a pass threshold gives false positives that teach people to ignore the canary; too loose lets real regressions through. When analysis fails, a computer rolls back. A human paging another human is the slow path you are trying to design out.

How stable bucketing works, and how it breaks

A percentage rollout is deterministic hashing, not sampling. flagd's fractional operator hashes the bucketing value (by default the targeting key joined with the flag key) with MurmurHash3 and drops the result into weighted buckets. LaunchDarkly hashes the context key plus the flag into 100,000 partitions and serves a variation to the partitions below your cutoff. Two consequences follow:

• The same targeting key always lands in the same bucket, so a user keeps their variation with no per-user record stored anywhere. Raising 10% to 20% only appends partitions above the old cutoff, so nobody already in the feature drops out.
• The flag key is part of the hashed input, so two flags both at 10% pick different users. You cannot reuse one flag's audience by copying its percentage onto another.

What breaks it: changing the salt or the bucketing key rehashes everyone. Mid-experiment that destroys the result, because assignment has to stay stable for the experiment's whole life (sticky bucketing). And if a client SDK and a server SDK hash the same user differently, one surface serves the new UI while another serves the old. Consistent targeting needs the same key and the same hash at every evaluation point.

Wrong "A per-request random draw at 10% is fine, it still hits 10% overall." The aggregate is right and the individual experience is broken. A user flips between variants on every page load, and any experiment reading conversion per user is measuring noise.

Kill switches and flag debt

An ops kill switch is a long-lived flag you never plan to remove. During an incident you flip it to disable a failing feature or shed load in seconds, no deploy and no pipeline run. It gets treated differently from a release toggle: it has a named owner (usually SRE), it is tested in both the on and off state, and it stays.

Release toggles are the opposite, inventory with a carrying cost. Fowler's guidance is to cap that inventory and force removal. The mechanics that hold up in practice: file the cleanup ticket the moment the rollout starts, stamp an expiration date in the flag's metadata, and for the strict version add a time bomb, a test or startup check that fails once a flag outlives its date. Skip this and you end up in a war room reading a config file with 300 flags where nobody can name which ones are load-bearing.

What rollback means once flags are in play

Two rollback paths at different speeds. If the change is behind a flag, rollback is a flag flip: near-instant, and it reverts exposure without touching what is deployed. If it is a bad artifact with no flag, rollback is redeploying the previous known-good version through the pipeline, minutes at best. Decide which one you are relying on before the incident. Reaching for a kill switch that was never wired is a bad time to learn it does not exist.