A successful HTTP response can hide a lost event, a duplicated action, or an update applied in the wrong order.
Webhook testing verifies that an HTTP event receiver authenticates requests, accepts work durably, and produces the intended result despite retries, concurrency, and interrupted processing. A response status establishes what the sender observed. It does not establish that your application completed the work correctly.
A webhook sends an HTTP request to a registered webhook URL when an event occurs. A WebSocket maintains a connection for bidirectional communication; testing that connection is a different task.
This guide proposes receiver integration tests for a fictional workspace provisioning service. A synthetic workspace event should create one workspace and one welcome notification for the correct tenant. Examples are proposed scenarios, not experiments or measured results. Provider documentation supplies specific contracts; the inbox and recovery design below is an engineering recommendation to test against your architecture.
Define the receiver contract and observable outcome
Separate delivery from application completion
Write down the accepted event types, trusted provider accounts, environment, signature scheme, payload limits, acknowledgement deadline, and supported schema versions. Specify what durable acceptance means and who owns processing failures after acknowledgement. Include the provider's retry and manual redelivery behavior rather than assuming every webhook behaves alike.
The fictional outcome is one workspace belonging to tenant A, one welcome notification, and no changes to tenant B. Record separate states for received, authenticated, accepted, processing, completed, quarantined, and ignored. A queued event is accepted but unfinished. A discarded event must have an explicit reason.
Establish independent assertions
Read the database and fake notification destination directly after each scenario. Assert counts, ownership, resource state, and pending work. Also assert that rejected requests produced no business effect. A receiver log saying processed cannot prove these conditions; it may have been emitted before a failed commit.
Build a repeatable webhook testing workflow
Isolate the complete receiver path
Use a staging tenant, dedicated database namespace, fake outbound services, and separate endpoint secrets. Freeze clocks for local boundary tests and synchronize clocks for provider deliveries. Give parallel runs distinct fixtures. Reset inbox rows, resource records, queues, and notification captures before each case.
Exercise the HTTP route through its actual middleware, authentication adapter, database, and worker. Calling the business handler directly misses body parsing and acknowledgement defects. Keep provider credentials outside fixtures and version control.
- Restore the synthetic tenant and verify its initial state.
- Select the provider contract and schema fixture version.
- Send the request with controlled bytes and delivery metadata.
- Capture the response and inspect durable acceptance.
- Run or interrupt the worker at the specified boundary.
- Read final state and assert effects and prohibited changes.
Verify the harness catches lost work
Use a negative control that returns success without persisting anything. The test must fail. Another control sends an event for tenant B while the expected outcome belongs to A. Verify that the checker detects the ownership mismatch instead of treating any created workspace as success.
Authenticate raw bytes using the provider contract
Verify before interpreting business data
Stripe's signature troubleshooting documentation explains that verification receives the raw body, signature header, and endpoint secret. Capture the original bytes before JSON middleware changes whitespace, encoding, or property order. Prefer the provider's recommended SDK and test the deployed middleware configuration.
GitHub's validation documentation specifies its HMAC header and recommends constant time comparison. For a custom verifier, follow that provider's exact algorithm, encoding, and header format. Reject missing headers, malformed encodings, mismatched lengths, and invalid signatures without throwing an uncontrolled server error. Never compare digests with ordinary string equality.
Test a valid request, one changed byte, the wrong endpoint secret, Unicode content, and a signature valid for another environment. Check secret rotation against the provider's overlap rules and confirm retired keys eventually fail.
Separate replay freshness from duplicate processing
The Standard Webhooks specification signs the message identifier, attempt timestamp, and body together. It recommends timestamp tolerance checks and constant time HMAC comparison. Adopt that scheme only for a compatible sender; it is not the contract of every provider.
Where a signed attempt timestamp exists, test stale requests, permitted clock skew, and future timestamps against your configured policy. Do not substitute an unsigned header or payload creation time for authenticated freshness. Some provider schemes authenticate only the body. Document that limitation rather than inventing a universal freshness guarantee. Deduplication still needs durable event identity even when freshness passes.
Acknowledge only after durable acceptance
Make the persistence boundary explicit
Persist authenticated, validated work in a durable inbox or durable queue before sending the success status required by the provider. A memory buffer or detached promise does not satisfy this boundary. Stop the receiver immediately after acknowledgement and confirm another worker can recover the event.
Queue failure at the acknowledgement boundary
If the queue is your only durable store and enqueue fails, return a non success response such as 503. Assert no success escaped. If an inbox transaction already committed and a recoverable dispatcher will enqueue later, queue unavailability can still permit acknowledgement. Test that dispatcher after restart; the inbox must contain everything required to resume.
Define policies for signed unusable requests
For invalid authentication, reject without processing. For correctly signed malformed JSON, choose a documented rejection policy, commonly 400, and verify alerting even if the sender retries it. For a valid but unsupported schema, explicitly choose rejection or durable quarantine followed by acknowledgement.
For an authenticated unknown event type, an explicit ignore policy can acknowledge after recording its disposition. An event that should have a handler must instead raise a detectable configuration failure. Test each policy separately so accidental catch all success responses cannot swallow required work.
Deduplicate logical events and survive worker crashes
Claim an inbox record atomically
Use a unique inbox key composed of trusted tenant, provider account namespace, and an identifier stable across retries for the intended effect. Exclude identifiers that change per attempt, not every field called a delivery ID. Derive tenant ownership from trusted endpoint configuration and validated account mapping, not an arbitrary payload field.
Shopify's delivery verification guide distinguishes webhook delivery IDs from event IDs shared across subscriptions for the same merchant action. For this proposed single provisioning effect, map deliveries to the stable event identity. Keep subscription routing explicit when different handlers intentionally need separate effects.
Check which identity fields the signature actually authenticates. Shopify's HMAC covers the body, not its event ID header. Propose a replay fixture retaining valid signed bytes while changing that unsigned ID. Independently enforce business idempotency from authenticated resource data, or verify an identity binding, so a new header cannot authorize another effect. Treat unsigned metadata as a correlation hint, not proof of a new event.
GitHub's webhook best practices state that its delivery ID remains unchanged on redelivery. Add a legitimate GitHub redelivery fixture asserting reuse of the existing inbox record. That stability supports retry deduplication, but does not establish that the header itself is signed.
Send concurrent copies and assert one inbox row and one logical effect. A separate lookup followed by insert is vulnerable to races. Enforce uniqueness in storage and handle the conflict atomically. A duplicate of unfinished work should acknowledge existing durable acceptance without deleting or prematurely marking that work complete.
Preserve effects across interrupted completion
Crash after the business effect commits
Interrupt the worker after creating the workspace but before recording completion. A retry must reconcile the committed result. Where possible, commit the resource mutation, inbox completion, and notification outbox row in one database transaction. Test rollback and restart around every commit boundary.
An external notification cannot share that transaction. Use a transactional outbox and a stable downstream idempotency key, then crash after sending but before marking the outbox row delivered. If the destination lacks idempotency or reconciliation, document the remaining duplicate risk. Inbox deduplication alone cannot guarantee one external effect.
Retain event identities for the supported replay horizon. Separate events describing the same resource may still need business level uniqueness; event deduplication does not settle that contract.
Exercise retries timeouts and redelivery semantics
Lose the response after accepting the event
Commit the inbox row, then drop the HTTP connection before the sender receives acknowledgement. Redeliver the same event and assert no second effect. Also test failure before commit, delayed responses, retry exhaustion, and a duplicate arriving while the first worker is active. Use bounded waits that inspect durable state rather than arbitrary sleeps.
Stripe's webhook delivery documentation describes automatic retries, unordered delivery, and fresh signatures and timestamps for each delivery attempt. A repeated payload therefore need not have the same signature. Its signature freshness check and your retained event identity solve different problems. Test those behaviors for Stripe without applying its retry schedule to other senders.
Include providers without automatic retries
GitHub's failed delivery guidance states that failed deliveries are not automatically redelivered. Test an authorized manual or scheduled recovery path where applicable. A non success response communicates failure but cannot ensure another attempt from every provider. Record who detects and recovers missed deliveries.
Handle event ordering and schema evolution
Reconcile using resource semantics
Deliver workspace version three before version two and assert that the older update cannot overwrite the newer state. Use a provider documented resource version or sequence when available, enforcing comparison atomically. An event timestamp is not a universal resource version and may be shared by different events.
If no ordering token exists, retrieve authoritative current state with scoped credentials and apply a defined reconciliation rule. Test missing resources, delayed reads, rate limits, and concurrent updates during that fetch. Decide whether the application needs the current snapshot or every historical transition; those requirements need different handling.
Pin schema expectations independently
Version fixtures with event type, payload format, endpoint API version, and receiver adapter. Cover required fields, unknown optional fields, nulls, changed enums, and unsupported versions. Older accepted events can remain in your inbox across a deployment, so run them through the candidate worker as well as testing freshly generated payloads.
Keep contract validation separate from business assertions. A valid schema can still target the wrong tenant or produce an invalid state transition.
Combine fixtures with real provider staging flows
Use local requests for controlled faults
Test webhooks during local development with synthetic signed fixtures to control body corruption, concurrency, and crash timing. Inspect the payload, HTTP headers, and status codes to debug failures, then assert persistence and effects behind the webhook endpoint.
The Stripe CLI documentation supports local event forwarding and fixtures that issue API request sequences. Forward requests to localhost with the provider listener, or use an approved tunnel for a reachable development URL. Use the appropriate signing secret and isolated provider environment. CLI generated examples exercise useful paths but do not represent every subscription, schema, or application journey.
Verify the registered staging destination
Trigger a real provider action using synthetic staging resources. Check destination registration, event subscriptions, TLS, proxy routing, environment secrets, and the resulting worker behavior. Compare the provider's delivery record with your inbox and final state. Keep a small suite of real flows alongside faster fixtures, with explicit cleanup and permitted outbound destinations.
Rehearse recovery and define release gates
Preserve evidence without exposing payloads
OWASP's Logging Cheat Sheet recommends excluding secrets and sensitive data and sanitizing untrusted log values. Record correlation IDs, redacted tenant references, validation outcome, response status, attempt count, queue age, and completion state. Restrict payload archives separately from ordinary logs and test redaction with synthetic secret markers.
Authorized replay from a dead letter queue
After bounded processing failures, preserve the event and reason in a dead letter queue or equivalent quarantine. Fix the cause, inspect the expected effect, and rehearse replay in staging with authorized operators and fake destinations. Keep original event identity and record operator, reason, and outcome. Never disable public endpoint verification to admit an expired captured signature; use provider redelivery or an authenticated internal recovery path over previously verified records.
Gate technical correctness and visible experience
Use a release checklist requiring passing authentication, durable acceptance, concurrent duplicate, crash recovery, schema, ordering, and redaction checks before promotion. Run backend regression automation on receiver changes. Set acknowledgement and backlog budgets from the provider contract and expected load. Rehearse recovery and alert ownership. These are proposed release criteria, not claims about tested performance.
Use the bug report guide for reproducible steps and redacted evidence. If an agent consumes the resulting workspace, the AI agent testing guide covers that separate behavior. For human feedback, create a BetaRoast testing round asking people to inspect the visible pending, ready, and recovery experience using synthetic accounts. Keep receiver fault injection and internal integration checks with engineering.
Frequently asked questions
What should webhook testing verify beyond a successful response?
Verify authenticated input, durable acceptance, correct tenant ownership, completed resource state, and intended downstream effects. Include duplicates, interruptions, and prohibited changes. Success at the HTTP boundary does not establish application completion.
Can signature validation prevent duplicate events?
No. A valid request can be delivered repeatedly. Where supported, signed timestamp checks constrain replay freshness. Atomic inbox identity and idempotent effects control repeated processing across legitimate retries and recovery.
Should a receiver return success when its queue is unavailable?
Only if the work is already durably stored and has a tested recovery path. If enqueueing is the sole persistence mechanism and it fails, return a non success response and invoke the provider specific recovery policy.
Are fixture requests enough to approve a release?
They cover controlled faults efficiently. Also exercise registered staging destinations with real provider actions and verify resulting state. Retain both layers because subscriptions, network routing, middleware, and environment configuration can fail independently.
Verify the provider contract against untouched request bytes, acknowledge only durable acceptance, and test logical effects through duplicates, timeouts, and crashes. Separate signature freshness from deduplication, reconcile ordering with resource semantics, and rehearse authorized recovery before release.