Webhooks

A webhook is an HTTP callback

A webhook lets one system send an HTTP request to another system when a subscribed event occurs. The receiver exposes a URL. The sender calls that URL with an event payload, then uses the HTTP response to decide whether that delivery attempt succeeded.

This is push delivery. Polling reverses the responsibility: the interested system calls an API periodically to ask whether anything changed. GitHub recommends webhooks for near real-time updates and notes that polling many resources can consume API rate limits. Polling still fits one-off or intermittent reads.

Wrong "A webhook is a permanent connection between two servers."

Right A webhook is an event-triggered HTTP delivery. A WebSocket is a different mechanism that opens a two-way communication session between client and server.

Why duplicate deliveries happen

Consider this sequence:

1. The receiver applies an event.
2. Its response is lost or arrives after the sender's timeout.
3. The sender cannot tell whether the work completed.
4. A retry sends the event again.

Under an at-least-once delivery guarantee, each message is delivered one or more times. Google Cloud Pub/Sub documents this guarantee and warns that a message can be redelivered even after acknowledgement. A webhook provider's contract may be narrower. Stripe documents possible duplicates and retries live deliveries for up to three days with exponential backoff, but a bounded retry window can expire while an endpoint remains unavailable. Read the provider's contract before claiming at-least-once delivery.

An event ID lets the receiver recognize a repeat. The check must survive restarts, so an in-memory set is insufficient for durable processing.

CREATE TABLE webhook_receipts (
  provider TEXT NOT NULL,
  event_id TEXT NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (provider, event_id)
);

Wrong "A retry is a new business event because it is a new HTTP request."

Right Delivery attempts and business events are separate identities. A sender can make several attempts for one event.

Acknowledge accepted work

Stripe recommends asynchronous processing and a quick 2xx response. The important boundary is durable acceptance. If the receiver returns success before saving the event or handing it to durable processing, a crash can lose work that the sender now considers delivered.

receive request
verify sender
save or enqueue the event durably
return 2xx
process the accepted event asynchronously

A non-success response can trigger a retry only when the provider's delivery policy says so. Receiver code has to follow that provider's documented status-code and timeout rules.

Arrival order is not event order

Stripe does not guarantee that webhook events arrive in the order in which they were generated. A later state change can therefore arrive before an earlier one. Processing every payload as "the latest state" can move local data backward.

When the provider contract offers no ordering guarantee, retrieve the current object when reconciliation is needed:

receive invoice.paid
fetch the referenced invoice and subscription from the provider API
apply the current provider state

Deduplication solves repeated delivery of one event. It does not solve two distinct events arriving out of order.

Verify before parsing or enqueueing

An HTTPS connection protects data in transit, but the endpoint still needs application-level evidence that the provider created the payload. Stripe signs webhook deliveries with HMAC-SHA256. The shared endpoint secret and the exact signed message produce the expected message authentication code.

Provider formats differ. Stripe signs a string containing its timestamp and the original JSON request body. Its documentation requires the raw body because parsing and serializing JSON can change the bytes.

app.post(
  "/webhooks/stripe",
  express.raw({ type: "application/json" }),
  async (req, res) => {
    const event = stripe.webhooks.constructEvent(
      req.body,
      req.headers["stripe-signature"],
      process.env.STRIPE_WEBHOOK_SECRET,
    );

    // Persist or enqueue event before returning success.
    await acceptDurably(event);
    res.sendStatus(200);
  },
);

Stripe's manual verification instructions construct the provider-defined signed payload and compare the expected signature in constant time. The official library call above performs provider-specific verification without duplicating that format in application code.

Wrong "I can verify the signature after JSON.parse because the object has the same values."

Right Signature verification is over the provider-defined message. For Stripe, that message contains the original request body.

A valid signature can still be a replay

An attacker can resend an intact payload with its intact signature. Stripe includes a signed timestamp and its libraries use a default five-minute tolerance. A receiver can reject a correctly signed payload when its signed timestamp is outside the accepted window.

Freshness is only one check. Stripe creates a new timestamp and signature for every retry, so a legitimate retry can be recent and correctly signed. Durable event-ID deduplication still has to prevent the business effect from running twice.

verify HMAC over the provider-defined signed payload
reject a signed timestamp outside the allowed tolerance
atomically claim the stable event ID
if already claimed, return 2xx without repeating the effect

Wrong "HMAC verification makes the handler idempotent."

Right HMAC authenticates the signed message and detects modification. An idempotency record tracks whether this receiver has accepted the event before.

The check and the state change must agree

A read followed by a write has a race when two delivery attempts run concurrently. Both requests can observe "missing" before either inserts the event ID. A unique key lets the database choose one winner.

BEGIN;

WITH claimed AS (
  INSERT INTO webhook_receipts (provider, event_id)
  VALUES ('stripe', 'evt_123')
  ON CONFLICT DO NOTHING
  RETURNING provider, event_id
)
INSERT INTO webhook_jobs (provider, event_id, payload)
SELECT provider, event_id, '{"type":"invoice.paid"}'::jsonb
FROM claimed;

COMMIT;

When the receipt already exists, claimed returns no row and this statement creates no job. The scope of the key matters. Stripe notes that separate Event objects can sometimes represent duplicates; for those cases it recommends combining the object ID with the event type. Use the identity rule documented by the provider rather than assuming every repeated business fact has the same event ID.

Keep the HTTP path short

The ingress handler should verify, durably accept, and acknowledge. Stripe recommends an asynchronous queue because delivery spikes can overwhelm synchronous endpoint processing. The worker can then retry local processing without extending the provider's HTTP timeout.

HTTP handler: verify -> deduplicate -> durable queue -> 2xx
worker: dequeue -> apply idempotent effect -> record outcome

If durable acceptance fails, a success response would hide the failure from the sender. Return the response prescribed by the provider contract and let its documented delivery policy decide whether another attempt occurs.

Delivery semantics belong to the product contract

There is no single retry contract shared by all webhooks. Stripe retries live deliveries for up to three days with exponential backoff. AWS EventBridge defaults to a 24-hour event age and up to 185 attempts for retriable target failures. A sender must publish its own success condition, retriable failures, attempt limits, event-age limit, and redelivery controls.

An at-least-once contract guarantees that each message is delivered one or more times, so receivers must also accept duplicate deliveries. A bounded webhook retry policy does not meet that guarantee when an endpoint can remain unavailable until the retry window expires. In that case, publish bounded retries and possible duplicates as separate properties.

Wrong "At least once means eventual delivery is guaranteed."

Right It guarantees one or more deliveries and therefore permits duplicates. A bounded retry policy can exhaust, so it should not be mislabeled.

Backoff needs a stopping condition

Exponential backoff increases the delay after successive failures. Jitter randomizes that delay so many failing deliveries do not all retry on the same schedule. AWS EventBridge provides a concrete documented policy: by default it retries retriable target failures for up to 24 hours and 185 attempts with exponential backoff and jitter.

A sender policy should make the bounds explicit:

attempt = 1
while attempt <= max_attempts and event_age <= max_event_age:
  result = deliver(event)
  if result is accepted:
    stop
  if result is permanent under the published policy:
    send_to_dead_letter_handling(event, result)
    stop
  if attempt == max_attempts:
    break
  delay = capped_exponential_delay_with_jitter(attempt)
  if event_age + delay > max_event_age:
    break
  wait(delay)
  attempt += 1

send_to_dead_letter_handling(event, last_result)

The distinction between retriable and permanent failure is provider-specific. Publishing it matters more than copying one provider's status-code table.

Dead-letter handling is part of delivery

AWS documents a dead-letter queue as the destination for events that remain undelivered after retries. Without one, EventBridge drops an event after attempts are exhausted. The stored failure gives operators a place to inspect, repair, and later process the event.

{
  "eventId": "evt_123",
  "destination": "https://receiver.example/webhooks",
  "attempts": 12,
  "lastFailure": "timeout",
  "firstAttemptedAt": "2026-07-12T10:00:00Z",
  "lastAttemptedAt": "2026-07-12T13:20:00Z"
}

Dead-letter storage does not repair the destination. The operating path needs inspection, alerting, a retention policy, and controlled replay after the cause is fixed. Replays must retain an identity that the receiver can deduplicate.

Ordering has to be narrower than "webhooks are ordered"

Stripe explicitly declines to guarantee generation order. Concurrent delivery, independent retries, and receiver workers can each change observed order. A receiver should use only ordering metadata that the provider defines. When the payload points to a mutable resource, Stripe recommends retrieving related objects if earlier events are missing.

event arrives for resource R
if the event can be applied under the provider's version contract:
  apply it conditionally
else:
  fetch current R from the provider
  reconcile local state to that response

An event timestamp alone does not create an ordering guarantee. Its meaning and tie behavior must come from the provider contract.

Webhooks, polling, and WebSockets answer different needs

GitHub describes webhooks as event-triggered deliveries that avoid repeatedly polling its API. It recommends API calls when information is needed once or intermittently. MDN defines WebSockets as a two-way interactive session in which either side can send messages without polling.

These mechanisms can coexist. A webhook can trigger fast processing, while periodic polling reconciles anything missed after the sender's retry window. A WebSocket can deliver live UI messages while a webhook handles server-to-server integration.