Feature Flags: How to Deploy Safely to Production at Big Tech Scale

It’s 2pm on a Tuesday. Your team just deployed a backend refactor that looked clean in staging. Within three minutes, the error rate graph goes vertical. Orders are failing. Users are dropping. Someone types “rollback?” in the incident channel and the next 45 minutes are the most stressful of your quarter — reverting commits, redeploying, waiting for the canary to stabilize, watching the metrics crawl back to baseline while the on-call phone keeps ringing.

This is the “big bang” deployment. Everything goes out at once, and when something breaks, you can’t separate the what from the when from the who. Feature flags exist to make this scenario structurally impossible — by decoupling deployment from release. Code ships to production dark. You turn it on deliberately, incrementally, and always with a kill switch.

The gap between how juniors implement flags and how they actually work at 10,000 TPS is wider than most engineers expect. Let’s close it.

The Naive Approach — And Why It Breaks

The first version of a feature flag almost always looks like this:

# The "good enough for now" implementation

def get_checkout_price(user_id: str, cart: Cart) -> Decimal:
    flag = db.query("SELECT enabled FROM feature_flags WHERE name = 'new_pricing_engine'")
    if flag.enabled:
        return new_pricing_engine.calculate(cart)
    return legacy_pricing_engine.calculate(cart)

This works fine at 10 requests per second. At 10,000 TPS, this is a database bomb. Every single request — checkout, search, product page render — fires a synchronous SQL query before it can do anything else. Your feature flag table becomes the most-read table in your entire database, and it contains exactly one row. You’ve added a mandatory round-trip to your critical path for a value that changes maybe once a week.

The slightly smarter version caches the result in memory. But now you have a different problem: how do you propagate a flag change to 200 application servers within a second? You can’t tell them to invalidate their local cache. And if you set a short TTL — say, 5 seconds — you’re still hitting the database 200 times every 5 seconds across your fleet, for every flag, across every service.

The naive approach treats feature flags as a config key. They’re not. At scale, they’re a distributed real-time state propagation problem.

The Big Tech Architecture: The Evaluation Engine

The architecture that solves this has three components working together: a low-latency flag store, a local in-process cache with push-based invalidation, and a stateless evaluation engine that applies targeting rules at call time.

flowchart LR
    A[Request] --> B[Evaluation Engine]
    B -->|nanosecond lookup| C[In-Process Cache]
    C -->|cache miss only| D[Redis]
    B --> E[Flag Decision]

The Flag Store: Redis with Push-Based Invalidation

The flag store is not your primary database. At companies like Meta (Gatekeeper), Google, and Amazon, flag state lives in Redis, with sub-millisecond read times. But the real trick is that application servers don’t read from Redis on every request either.

Each server maintains an in-process cache — a hash map in memory — populated at startup and kept fresh via a pub/sub invalidation channel. When an operator flips a flag, the flag service writes to Redis and simultaneously publishes an invalidation event. Every subscribed server receives it within milliseconds, drops its cached value for that flag, and re-fetches from Redis exactly once.

flowchart LR
    A[Operator] -->|flip flag| B[Flag Service]
    B -->|write new state| C[Redis]
    B -->|publish invalidation| D[Pub/Sub Bus]
    D -->|notify| E[App Server Fleet\nin-process cache]
    E -->|re-fetch once| C

The result: flag reads are a hash map lookup in nanoseconds. Flag updates propagate fleet-wide in under a second. Your database is never involved.

The Evaluation Engine: Context-Aware Targeting

A boolean enabled/disabled flag is the simplest case. Production feature flags need to answer more nuanced questions: is this flag enabled for this specific user, in this country, on this device, with this account tier?

This is the evaluation engine’s job. Rather than storing a simple boolean, each flag stores a ruleset — an ordered list of targeting conditions evaluated against the caller’s context. The engine walks the rules in order and returns the first match.

# feature_flags/engine.py

from dataclasses import dataclass

@dataclass
class EvaluationContext:
    user_id: str
    country_code: str
    account_tier: str        # e.g. "free", "pro", "enterprise"
    app_version: str | None = None

class FeatureFlagEngine:
    def __init__(self, flag_store: FlagStore):
        self._store = flag_store

    def is_enabled(self, flag_name: str, ctx: EvaluationContext) -> bool:
        flag = self._store.get(flag_name)  # nanosecond in-process cache read
        if flag is None:
            return False  # fail closed -- unknown flags are always off

        for rule in flag.rules:
            if self._matches(rule, ctx):
                return rule.enabled

        return flag.default_enabled

    def _matches(self, rule: Rule, ctx: EvaluationContext) -> bool:
        if rule.target_user_ids and ctx.user_id not in rule.target_user_ids:
            return False
        if rule.target_countries and ctx.country_code not in rule.target_countries:
            return False
        if rule.target_tiers and ctx.account_tier not in rule.target_tiers:
            return False
        if rule.rollout_percentage:
            # Deterministic hash: same user always lands in the same bucket
            bucket = hash(f"{rule.flag_name}:{ctx.user_id}") % 100
            if bucket >= rule.rollout_percentage:
                return False
        return True

The call site stays clean — all the complexity lives inside the engine:

# checkout/pricing.py

def get_checkout_price(user_id: str, country: str, tier: str, cart: Cart) -> Decimal:
    ctx = EvaluationContext(
        user_id=user_id,
        country_code=country,
        account_tier=tier,
    )
    if flags.is_enabled("new_pricing_engine", ctx):
        return new_pricing_engine.calculate(cart)
    return legacy_pricing_engine.calculate(cart)

The rollout_percentage deserves special attention. By hashing flag_name + user_id instead of using a random number, you get deterministic bucketing — the same user always lands in the same bucket. This means a user at 10% rollout who refreshes the page doesn’t flip between old and new experiences. It also means you can reproduce any evaluation result exactly given the same inputs, which makes debugging and auditing tractable.

The Tradeoffs: What Can Still Go Wrong

Getting the architecture right doesn’t make feature flags free.

Flag debt is real. Every flag you add is a branch in your code. After the rollout is complete, that branch is dead weight — but it stays in the codebase forever unless someone removes it. At scale, services accumulate hundreds of stale flags, the code becomes increasingly hard to reason about, and the evaluation engine runs rules for flags that have been at 100% for six months. The discipline of treating flag removal as part of the rollout process — scheduled at the time you create the flag — is something most teams learn the hard way.

Pub/sub delivery is not guaranteed. If a server misses an invalidation event — due to a network partition, a restart, or a transient Redis outage — it will serve a stale flag value until its TTL expires. You need a fallback polling mechanism with a reasonable TTL (30-60 seconds is common) to bound your maximum staleness window. This is the tradeoff you accept for nanosecond reads.

Targeting complexity becomes a maintenance burden. A rule that says “users in DE, on Pro tier, on version >= 4.2, except user IDs in this exclusion list” is technically valid and operationally terrifying. The evaluation engine will happily execute it. Keeping targeting rules simple and well-documented is a process discipline problem, not a technical one.

Flags are not a substitute for quality. The kill switch gives you confidence to deploy. It doesn’t give you permission to skip testing. Shadow traffic and dark reads — the same techniques that made the Amazon Checkout migration safe — are still the gold standard for validating new behaviour before you trust a flag to protect you at 100%.

Takeaways

Deployment and release are not the same thing. Feature flags decouple when code ships from when users see it. That separation is the foundation of safe deploys at any scale.
Never read flags from your primary database on the hot path. Redis with in-process caching and pub/sub invalidation is the standard pattern. Anything slower adds latency to every request for a value that changes rarely.
Deterministic bucketing is non-negotiable for percentage rollouts. Hashing on flag_name + user_id gives you stable, reproducible targeting. Random sampling does not.
Fail closed. An unknown or unreachable flag should always default to false. Fail open — defaulting unknown flags to enabled — is how you accidentally release half-finished features during an outage.
Treat flag removal as part of the rollout. Create the ticket to remove the flag on the same day you create the flag. Flag debt compounds silently and is expensive to pay down later.

The engineering behind feature flags looks simple from the outside — it’s just an if/else, right? The implementation that survives production is a distributed system in its own right: a low-latency store, a push-based invalidation bus, a deterministic evaluation engine, and the operational discipline to clean up after yourself.

If you’re thinking through your own flag infrastructure or want to dig into any of these tradeoffs, connect with me on GitHub or LinkedIn.