Load and performance testing

Open vs closed load models (VUs vs arrival rate)

Most load tools default to a closed model: a fixed number of virtual users (VUs), each one looping "send a request, wait for the response, send the next." The number of concurrent requests is capped by the VU count, and a VU that is waiting on a slow response is not sending anything. That last part is the trap.

When the server gets slower, iterations take longer, so new requests start later, so the offered load drops. The test eases off the system at the exact moment you wanted to push it. Real users do not behave this way: they keep arriving whether or not your server is keeping up.

An open model fixes the arrival rate instead of the VU count. You say "start 800 iterations per second" and the tool launches them on that schedule no matter how long each one takes. In k6 the closed executors are constant-vus and ramping-vus; the open ones are constant-arrival-rate and ramping-arrival-rate. Gatling draws the same line: constantConcurrentUsers/rampConcurrentUsers are closed, constantUsersPerSec/rampUsersPerSec are open.

Wrong "200 VUs means I tested 200 requests per second." VUs are concurrency, not throughput. Their request rate depends on how fast each response comes back, so it drifts down under load. If your target is a request rate, drive an arrival-rate executor and read throughput from the actual requests sent. Gatling's own guidance is that most real systems, especially websites, are open workloads.

Which test type answers which question

The type is a property of the whole test, and each shape answers a different question. Pick by what you need to learn.

Two of these get confused. A stress test runs hot but briefly, so it will not catch a leak that takes hours to matter; that is a soak test's job. A spike test is about the transition (does autoscaling react in time, does the queue drain) rather than the steady peak a stress test holds.

Thresholds turn an SLO into a pass or fail

A load test that just prints numbers needs a human to judge it. A threshold makes the run self-grading. In k6 you declare pass/fail conditions on metrics, and if any fails, k6 exits non-zero (exit code 99), which fails the CI step and can block a deploy.

export const options = {
  thresholds: {
    http_req_duration: ['p(99)<300'],  // 99th percentile under 300ms
    http_req_failed:   ['rate<0.01'],  // under 1% errors
  },
};

Write the threshold to match the SLO you promised rather than a rounder number that is easier to pass. If the SLO says p99 under 300ms at 800 rps, the threshold is p(99)<300 and the scenario runs at 800 rps. A threshold on avg would pass a run whose tail badly misses the SLO, which is the next section.

Why averages hide the latency tail

Latency distributions are skewed: a wall of fast responses and a long slow tail. The average sits down among the fast majority and barely moves when the tail gets ugly, so it is the wrong number to gate on.

Wrong "Average response time is 90ms, we are fine." An average of 90ms is consistent with 1% of requests taking 5 seconds, and that 1% is often your most active users (more requests means a higher chance of hitting the tail). Report and gate on percentiles: p95, p99, sometimes p99.9. The SLO is almost always written as a percentile for this reason, and your threshold should quote the same one. How percentiles get aggregated across shards and time windows (you cannot average them) is covered in the backend-performance notes.

Coordinated omission in a load generator

Coordinated omission is a measurement bug: the load tool stops issuing requests during the exact window when the system is slowest, so the recorded latencies omit the worst cases, and they omit them in a coordinated way (every stalled client backs off together). The deep version of this, and the HdrHistogram correction that repairs an already-recorded dataset, lives in the profiling-and-benchmarking notes. Here the point is narrower: how it hides inside a load tool and how the workload model prevents it.

A closed executor produces coordinated omission by construction. Each VU sends the next request only after the current response returns, so when the server freezes for 200ms, that VU issues nothing for 200ms. The arrival of requests that a real open workload would have queued behind the stall never fires, so those omitted requests are precisely the slow ones, and the p99 comes out flattering.

Switching to an arrival-rate executor is the structural fix: k6 schedules iterations on a fixed clock regardless of responses, so a stall produces a pile of requests whose latencies climb, which is what you wanted to see. But "we use arrival-rate, so we are safe from coordinated omission" is too strong.

Wrong "Arrival-rate executors measure the tail correctly no matter what." They only measure the tail of requests they actually send. When the generator cannot free a VU in time to start a scheduled iteration, k6 does not send it and increments dropped_iterations instead. Those dropped iterations are precisely the ones that would have carried the worst latency, so a run with rising dropped_iterations has quietly reintroduced the omission. Treat dropped_iterations > 0 as "this run is not valid evidence for a latency claim yet," size the generator up, and rerun.

Sizing the generator with Little's Law

Before you trust an arrival-rate test, prove the generator can deliver the rate. Little's Law gives the bridge between throughput and concurrency:

L = λ · W
concurrency (VUs in flight) = arrival rate · response time

At 800 req/s with a 300ms response, you have about 800 × 0.3 = 240 requests in flight at any instant. Your VU pool has to cover that plus headroom, because the moment latency rises the required concurrency rises with it. In k6 that means preAllocatedVUs and maxVUs sized above the expected concurrency; set them too low and the executor runs out of VUs, drops iterations, and under-delivers the load without failing loudly.

The same formula is your back-of-envelope capacity check before any test runs. Read it in whichever direction you know two of the three terms:

• Known rate and latency: how much concurrency (VUs, threads, pool slots) must exist.
• Known concurrency cap and latency: the ceiling on throughput. A pool capped at 200 connections serving 400ms requests tops out near 200 / 0.4 = 500 req/s, so a 700 rps target is impossible without more slots or lower latency, and no amount of load generation changes that.
• Known rate and concurrency budget: the latency you must hold to fit.

This is why latency is a throughput multiplier: a 2x latency regression doubles the concurrency your system needs at the same request rate, and it doubles the VUs your test needs to keep offering that rate.

Pass/fail tied to SLOs, and where thresholds mislead

A threshold is only meaningful if it encodes the SLO you promised. The mapping is literal: an SLO of "p99 < 300ms and error rate < 0.1% at 800 rps" becomes a scenario at 800 rps with http_req_duration: ['p(99)<300'] and http_req_failed: ['rate<0.001']. A failed threshold sets k6's exit code to 99, which is what turns the test into a deploy gate.

thresholds: {
  http_req_duration: ['p(99)<300', 'p(99.9)<1000'],
  http_req_failed:   ['rate<0.001'],
},

Three failure modes turn a green run into a lie:

1. Gating on avg instead of the SLO percentile. The average is dominated by the fast majority and can pass while p99 misses by an order of magnitude.
2. A green latency threshold on top of dropped_iterations or a high http_req_failed rate. Fast responses are cheap when half the load never landed or errored out early, so always read the percentile next to the delivered rate and the error rate, never alone.
3. Percentiles computed by the tool per-instance and then averaged across load generators. Percentiles do not average; the aggregation method is in the backend-performance notes.

abortOnFail on a breakpoint or stress threshold stops the run the moment the ceiling is crossed, which saves you from hammering an already-broken system, but use delayAbortEval so a single transient sample does not abort a valid run.

Pre-prod vs prod: fidelity and testing in production

A staging pass is worth exactly as much as staging resembles production, and usually that is not much. The gaps that move the tail:

• Data volume. A table with 100 rows seq-scans instantly; the same query on 100M rows lives or dies on an index and a good plan. Query latency, and therefore your whole SLO, can invert between the two.
• Topology and limits. Pod or instance count, connection-pool caps, autoscaling reaction time, and rate limiters all shape behavior under load and are rarely mirrored in a one-pod staging box.
• Neighbors and network. Shared-tenant CPU, real CDN and TLS termination, cross-AZ hops. A local or single-node test has none of it.

Wrong "It passed in staging, so the SLO holds in prod." Staging validates the script, the logic, and gross regressions. It does not certify a production SLO unless it matches production scale, and matching it exactly is often too expensive to be real.

When parity is unaffordable, test in production with guardrails instead of pretending staging is enough. The safer forms: a canary that takes a small percentage of real traffic while you compare its p99 against the baseline, or shadow (mirror) traffic that replays real requests against the new version with responses discarded. Both need a kill switch, isolation of side effects (a shadowed write must not double-charge anyone), and a clear abort metric. This is the only way to load-test against real data distributions, real cache states, and real traffic mix, which is where the surprises actually are.