Reliable webhook integration: retries, idempotency, and recovery.

Verify and acknowledge

Validate the provider signature against the raw request body, reject stale or invalid requests, and return the required success response quickly. Move slow work out of the request path when the platform supports it.

import { createHmac, timingSafeEqual } from "node:crypto";

const TOLERANCE_SECONDS = 300;

export async function POST(request: Request) {
  // Verify the exact bytes that were signed — before JSON parsing.
  const rawBody = await request.text();
  const timestamp = request.headers.get("x-webhook-timestamp") ?? "";
  const signature = request.headers.get("x-webhook-signature") ?? "";

  const age = Math.abs(Date.now() / 1000 - Number(timestamp));
  if (!Number.isFinite(age) || age > TOLERANCE_SECONDS) {
    return new Response("stale or missing timestamp", { status: 400 });
  }

  const expected = createHmac("sha256", process.env.WEBHOOK_SECRET!)
    .update(`${timestamp}.${rawBody}`)
    .digest("hex");
  const valid =
    expected.length === signature.length &&
    timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
  if (!valid) return new Response("invalid signature", { status: 401 });

  const event = JSON.parse(rawBody);
  await recordAndEnqueue(event); // idempotent insert, then background work
  return new Response(null, { status: 204 });
}

Header names and the exact signed string differ by provider. Where the provider ships an SDK helper — for example Stripe’s constructEvent — use it instead of hand-rolled verification.

Make processing idempotent

Use the provider event ID or a stable business key to prevent duplicate effects. Store processing state and design database writes so a retry cannot charge, notify, or mutate the same record twice.

create table webhook_events (
  provider     text        not null,
  event_id     text        not null,
  event_type   text        not null,
  received_at  timestamptz not null default now(),
  status       text        not null default 'received', -- received | processed | failed | dead_letter
  attempts     int         not null default 0,
  last_error   text,
  primary key (provider, event_id)
);

-- Record the event once. A duplicate delivery inserts nothing.
insert into webhook_events (provider, event_id, event_type)
values ($1, $2, $3)
on conflict (provider, event_id) do nothing
returning event_id;
-- No row returned → already seen: acknowledge and skip side effects.

Apply the business change and mark the row processed in the same transaction, or make the downstream write idempotent too; otherwise a crash between the two can still double-apply.

Expect delay and reordering

Fetch current provider state when an event alone is insufficient. Model transitions explicitly and avoid assuming that creation, update, and completion events arrive once or in sequence.

Build an operator recovery path

Record sanitized payload metadata, attempt count, error, and next action. Use bounded retries, a dead-letter state, alerting, replay controls, and reconciliation for events that cannot be processed automatically.

Retry and backoff

An example schedule, not a rule — add jitter, and tune the numbers to the provider’s own retry window.

Handle provider outages

When a provider is down or stops sending events, nothing arrives to fail — the risk is silence. Track the age of the last event per provider and alert when it exceeds what normal traffic allows.

Retry behavior differs by provider: Stripe documents automatic retries for up to three days, while GitHub does not automatically redeliver failed deliveries. Check the current documentation for each provider you depend on.

After recovery, backfill by listing recent objects or events through the provider API and passing them through the same idempotent path, so replayed and live events cannot double-apply. Keep the product usable in the meantime and mark provider-dependent data as possibly stale.

Failure scenarios

Implementation checklist

• Verify the signature against the raw body with a constant-time comparison
• Reject stale timestamps to limit replay attacks
• Return a 2xx quickly and move slow work to a job or queue
• Record the event ID under a unique constraint before side effects
• Make downstream writes idempotent as well
• Treat delivery order as unknown; compare versions or fetch current state
• Bound retries with exponential backoff and jitter
• Keep a dead-letter state with payload metadata and the last error
• Give an operator a way to replay one event or a batch
• Alert on failure spikes and on silence (last-event age)
• Log sanitized metadata — never secrets or full payment data
• Test with provider test events, duplicated deliveries, and reordered events

Common webhook mistakes

• Parsing and re-serializing JSON before verifying, which breaks the signature
• Doing slow work before responding, which causes timeouts and duplicate retries
• Using delivery order as business order
• Treating a 2xx response as proof that the work succeeded
• Retrying forever with no dead-letter state
• Having no way to replay events after fixing a bug
• Logging full payloads that contain personal data or secrets

Primary references

Provider behavior changes; check these before relying on a specific number.

• Stripe: Receive Stripe events in your webhook endpoint ↗
• GitHub: Best practices for using webhooks ↗
• GitHub: Handling failed webhook deliveries ↗
• PostgreSQL: INSERT … ON CONFLICT ↗

See the approach in working products

www.useresolve.streamRESOLVECircle webhooks are signature-verified and de-duplicated, and funding states and testnet USDC escrow are explicit records rather than hidden side effects.View RESOLVE case study →

circle-arc-net.vercel.appAgora ForgeLI.FI routes are compared and executed from quotes, and Circle CCTP transfers run via Circle App Kit on testnets, with each hop shown before signing.View Agora Forge case study →