Why webhook payloads break pipelines, and what to check

A webhook payload looks like a JSON document, so people treat it like a file. It behaves nothing like one, and every property that makes it different is a property that breaks a pipeline built on batch assumptions.

The shape depends on the event

The first surprise for most people: orders/create and orders/updated from the same vendor do not have the same fields. Neither does orders/create from a POS versus an online channel. If you're landing all events into one table, you're unioning several schemas and calling the result one thing.

The failure mode is subtle: a field that's mandatory for one event type shows 60% presence across the mixed stream, so it looks optional, so nobody asserts on it, so nobody notices when it goes missing from the event type where it was required.

Profile each event type separately before you decide anything is optional. If a field is 100% present within orders/create and absent from orders/updated, that's not a 60%-present field — that's two different contracts sharing a table.

Optional means genuinely optional

Vendor APIs omit keys rather than sending nulls, and they omit far more than the docs suggest. discount_code, shipping_address, customer.phone — all absent whenever they don't apply, which is most of the time for some of them.

The trap is bootstrapping your checks from a small sample. Twenty payloads from your test store, all with shipping addresses, and you write not_null on shipping_address.zip. Then a digital-only order arrives and your ingest quarantines a perfectly valid payload.

A rule that holds up: a field observed present in 100% of a small sample is not a required field, it's an unremarkable field. Confidence in "always present" should scale with how much data you've seen — 100% of 30 payloads is weak evidence, 100% of 30,000 is strong. Treat anything in between as a prompt to go look rather than a fact.

Delivery is at-least-once

Every major webhook provider retries on non-2xx, and several retry on timeouts they couldn't confirm. You will receive duplicates. This means a raw webhook table cannot have a uniqueness constraint on the event id, and any downstream aggregate that assumes one row per event will overcount during a retry storm — which is exactly when you're least able to think clearly about it.

Deduplicate on the delivery id at the boundary, keep the raw log append-only, and assert uniqueness on the deduplicated view rather than the landing table. Uniqueness checks belong downstream of dedup, not upstream.

Ordering isn't guaranteed either

A related and less discussed one: orders/updated can arrive before orders/create. Retries, parallel delivery workers, and network variance all reorder events. Any logic that assumes the create event landed first will occasionally build state from an update to a row that doesn't exist yet.

Sort by the payload's own timestamp rather than receipt time, and make your state transitions idempotent and order-independent where you can.

Nested arrays change what presence means

line_items[] is where the counting gets subtle. line_items[].sku present in 100% of line items tells you nothing about how many orders have line items at all. Both numbers matter and they answer different questions:

• Is the array populated? — a per-order question. Empty or missing line_items on an order is usually an anomaly worth flagging.
• When an item exists, does it have this field? — a per-element question. This is the one that belongs in a validation rule about SKUs.

Conflating them produces checks that pass when they shouldn't. A tool that reports one number for "presence" on an in-array field is giving you the wrong denominator half the time.

The vendor will change it without telling you

Not maliciously — additively, which is worse, because additive changes don't break anything immediately. A new field appears. An enum gains a value. A numeric field starts arriving as a string from one code path. None of it errors; all of it eventually produces wrong numbers.

The defence is a stored profile of what the payload looked like, diffed on a schedule. Not the declared schema — the observed one, including type mix and value distributions. This is the only reliable way to find out that a vendor changed something in week one instead of week seven.

Where to put each check

Two layers, because they catch different things and have different failure costs.

At the door — JSON Schema. Structural validation on the raw payload, before it lands. It's the only common validator that reads nested JSON as-is, so it can assert on line_items[].sku without anything being flattened first. Keep it strict about structure and loose about values: required keys, types, array bounds. Route failures to a quarantine table rather than dropping them — a rejected payload is evidence, and you'll want it when you're diagnosing why the vendor changed something.

After it lands — SQL-based checks. Soda, dbt tests, or Great Expectations on the flattened tables, for everything that needs to compare rows to each other: uniqueness after dedup, referential integrity, volume and freshness, business rules like "total equals the sum of line items". None of that is expressible in JSON Schema, which validates one document at a time and can't see across records.

The split matters because the costs differ. A structural failure at the door is cheap — quarantine one payload, alert, move on. A business-rule failure after landing is a data incident with a blast radius. Catching structure early keeps the expensive layer focused on the things only it can see.