Log drain feedback loop caused ~$1,700 in overages — support ticket unanswered since July 6

Help
1

Posting this partly as a warning and partly because I’m stuck on a support ticket.

Back in mid-June I set up a Log Drain to feed an AI-crawler analytics feature I was building. I pointed the drain at an API route living inside the same project the drain was watching. Didn’t think twice about it at the time.

Every delivery hit that endpoint, which generated its own request and function logs, which the drain then shipped right back to the same endpoint. It just fed itself. Ran that way for about two weeks.

At its steady state it was sitting around 370 requests/second, day and night. Roughly 30 million requests a day, 60+ million function invocations, and about 66 GB of drained logs daily. All of it the loop talking to itself — zero real traffic.

Damage: ~$595 Edge Requests, ~$366 Function Invocations, ~$344 Drains Volume, ~$179 Fast Origin Transfer, ~$176 Fluid Active CPU. About $1,700 total. My normal bill for this team is under $100/month.

I caught it and killed the drain on July 2. Logs show traffic dropped back to baseline within minutes. I’ve since rebuilt it with the drain destination hosted outside Vercel entirely, so there’s no way to recreate the loop. Turning on Spend Management too, which I should have had on from the start.

Two things I’m hoping someone can weigh in on:

Is there anything planned to catch this? Even a warning when a drain’s destination URL points at a domain inside the same project would have saved me. Or some kind of circuit breaker when drain volume goes vertical. I’m not trying to pass the buck — this was my mistake — but it fails silently and you don’t find out until the invoice shows up two weeks later.

Second, I opened a billing ticket on July 6 asking about a one-time courtesy credit and haven’t heard back. The invoice is past due now. If any staff here can nudge it toward whoever handles those, I’d really appreciate it. Happy to DM the ticket number and team ID.

Anyway — mostly posting so the next person who searches “log drain runaway usage” actually finds something.

2

Hi Michael, glad you caught and fixed the loop. Since it is already contained, the useful takeaway is how to prevent the same failure mode by design.

A safer architecture for this pattern is:

Log Drain
    |
    v
Isolated external receiver
    |
    v
Bounded queue
    |
    v
Processing and storage

The receiver should sit outside the project being observed, return a fast 2xx response, and enqueue the payload instead of processing it synchronously.

Additional safeguards:

  • Deduplicate entries using the log id.
  • Collect only the required sources, environments and paths.
  • Apply sampling before enabling full traffic.
  • Limit queue size, ingestion rate and retry volume.
  • Alert on sudden request-rate or drain-volume growth.
  • Use Spend Management as the final circuit breaker.

The validation rule is simple: processing a delivered log must not generate another log that can enter the same drain.

If that condition is not guaranteed by the topology, the feedback path still exists.

3

I’m sorry I didn’t see this post until now. I found your ticket and see the support team issued a refund last week.

In the future, you can catch this kind of thing earlier using spend management with the option to automatically pause all deployments or trigger a webhook for custom action when your preferred spend limit is reached.