Skip to content
HookDeploy
← All posts

Engineering5 min read

How We Cut Our Cold-Path Latency From 2600ms to 650ms

How HookDeploy cut cold-path latency from a 2600ms regression down to 650ms: parallelizing sequential Supabase calls, adding a missing cache layer, moving R2 writes off the response path, and consolidating three config lookups into one cached RPC.

Michael Bernstein

Founder, HookDeploy

Fast is easy when a product does one thing. HookDeploy started simple: capture a webhook, write it down, forward it. Then we added custom response configuration, response delay simulation, and platform verification handshakes for Meta, Slack, and Zoom. Each feature was a real customer need. Together, they quietly turned our cold-path latency from roughly 277ms into something closer to 2600ms.

That’s not a rounding error. That’s the difference between “fast enough nobody notices” and “slow enough people file a support ticket.” Here’s what actually happened, and how we fixed it.

How a fast path gets slow, one feature at a time

No single feature we shipped was the obvious villain. Each one, on its own, added a small, defensible cost:

  • Verification config lookups, so a Meta or Slack handshake could be checked before we even captured the request
  • Response config lookups, so a custom status code and body could be applied instead of the default
  • Sanitization config lookups, so header sanitization rules could be read

Each of these needs its own data, and each was implemented the straightforward way: fetch it from Supabase when needed. Individually reasonable. Collectively, we’d added three sequential round trips to a hot path that used to have almost none.

Root cause one: sequential calls where parallel calls belonged

The first problem was structural. getVerificationConfig was running before a Promise.all, with getResponseConfig and fetchEndpointSanitizationConfig running inside it. That meant one full round trip happened, completed, and only then did the other two start, in parallel with each other but not with the first. Three lookups that could have cost the price of the slowest one were instead costing the sum of two.

The fix here was straightforward once we saw it clearly: pull all three into a single Promise.all so they race each other instead of queuing. That alone recovers a meaningful chunk of latency, but it wasn’t close to enough on its own.

Root cause two: a lookup with no cache at all

fetchEndpointSanitizationConfig had never been given a KV cache layer. Every single request was paying a full Supabase round trip for it, uncached, every time. In a synchronous request path, an uncached database call is one of the most expensive things you can leave in place, and it’s exactly the kind of gap that’s easy to miss when you’re adding a feature under time pressure and planning to “add caching later.”

We added the cache. Predictable win, but again, not the whole story.

Root cause three: a write that had no business blocking the response

The R2 write, storing the actual payload, was happening synchronously, in the response path, before we returned anything to the caller. That’s backwards. The customer sending the webhook doesn’t need us to finish writing to object storage before we acknowledge receipt. They need a fast, honest “got it,” and the actual persistence can happen a moment later without them ever knowing the difference.

We moved the R2 write into ctx.waitUntil, the same background-execution pattern we already used for Supabase metadata inserts, usage counters, and fanout to forwarding destinations and subscribers. Anything that doesn’t change what we tell the caller doesn’t belong in the sequence that decides how fast we tell them.

The real fix: stop asking three questions when one will do

Parallelizing and caching individually was progress, but the deeper fix was architectural: why were we making three separate round trips (verification, response config, sanitization) for data that’s all keyed off the same endpoint and needed at the same moment in the same request?

We consolidated all three into a single RPC, get_endpoint_configs(p_endpoint_id), that returns verification, response, and sanitization configuration in one round trip. The result is cached under one unified key, configs:v1:{endpointId}, replacing what used to be three separate cache keys read independently. One RPC, one cache entry, one lookup cost, instead of three.

This is also where we found a real bug, not just an optimization opportunity. Our cache invalidation logic was deleting the old resp_config:v1: key on save, but the hot path had already moved on to reading the new unified configs:v1: key. We were invalidating a cache nobody was reading anymore, which meant saved changes weren’t reliably showing up until the TTL happened to expire on the key that actually mattered. Consolidating the read path forced us to consolidate the invalidation path too, and that fixed a real correctness bug alongside the performance one.

One more dial: how long a cache entry gets to live

Our KV TTLs were sitting at 60 seconds across the board, which is Cloudflare KV’s practical minimum. That’s fine when a cache miss is cheap. It’s not fine when every miss now costs a consolidated but still real Supabase round trip. We bumped TTLs to 3600 seconds everywhere, on the understanding that TTL is a safety net now, not the primary mechanism keeping the cache honest. Explicit invalidation on save is what actually keeps the cache correct; the longer TTL just means we’re not paying the cache-miss cost sixty times an hour for data that rarely changes.

Where we landed

Hot path: roughly 50ms. Cold path: roughly 650ms, down from the 2600ms regression, and meaningfully better than our original 277ms baseline once you account for the fact that we now do considerably more per request than we did back then. For reference, Hookdeck, one of the more established players in this space, publishes 97ms as their number. We’re not claiming to have solved latency once and for all. We’re saying that adding real functionality doesn’t have to mean accepting a 10x regression, if you’re willing to go find out exactly where the cost actually is instead of guessing.

We also added pipeline_ms directly to the response body on every request, so this number isn’t just something we say in a blog post. It’s something every customer can see on every single webhook we process.

Not one big rewrite fixed this. Three specific, boring fixes did: run independent lookups in parallel instead of in sequence, cache the thing that was never cached, and get anything off the response path that doesn’t need to be there. The consolidation into a single RPC and cache key was the piece that turned “we made three things individually faster” into “we removed two-thirds of the round trips entirely,” and it’s also the piece that surfaced a real invalidation bug we wouldn’t have found any other way. If there’s a lesson worth taking from this, it’s that latency regressions from feature growth aren’t a sign you built the features wrong. They’re a sign it’s time to go measure, instead of assume, where the actual cost went.