Malicious bots and firewall : charged for denied?

Help
1

Hello,
I’m using Vercel on Hobby plan (for now) but I’m a bit nervous about the firewall.

Recently I’ve ben hammered by malicious bots (worst : alibaba crawler) and could get up to 200k requests/30days for one single bot (bing bot).
Vercel’s firewall neither identified this as DDOS nor challenged it, even though all those bots represented about 95% of my traffic.

I started to mitigate with better robots.txt but… it only works for bots respecting it. Alibaba, for example, does not.

I tried to create a custom firewall rule to deny Alibaba’s bots and saw the calls being denied : good. But it hit me : if a call is denied by a custom rule, will I be billed for it ?
If I build good custom rules to deny malicious traffic, will I still be in trouble regarding Vercel usage ?

This documentation mentions “persistent actions” with timeframe, but I can’t see them. Is it because I’m on Hobby plan ? (while it says custom rules are on all plans) :

I feel that if I have to upgrade to pro plan to reduce my traffic so it goes below Hobby plan, that’d be a silly move… Better host myself in this case ? What would be the benefit from paying so I can properly block bots ?
220k requests a day by a single user agent looks like an attack and I’m a bit disappointed that the firewall doesn’t catch it by itself.

Project : NextJS15 with mostly SSR and ISR.

2

Great questions, @iliaschaumont-gmailc. We do not bill for usage where requests were actively blocked by our Vercel Firewall, but you you may incur Edge Requests and incoming Fast Data Transfer usage for active WAF custom rule challenged or denied requests. The docs have all the details about usage & pricing for Vercel WAF.

We also have a live event from the Vercel Security Series happening this week that I recommend attending :slightly_smiling_face:

1 Like
3

Thanks for your answer Amy.
I am though a bit confused and worried.
I guess I am not the only case you’ve been seeing this with, but crawler and AI crawler started to get crazy, and the easiest way to block them is UserAgent for the cleanest, and IP for the dirtiest.
For the IP, I wish I could do that every time, but those dirty bots don’t provide their full IP range. That means that I’d have to block xxx.yyy.0.0 and risking blocking some legit IPs within this range.
For the user agent, it has to go through custom rule so I’d get charged for this anyway.

First, am I correct on this?

Now, if I get to pro plan because these bots make me go FAR above hobby plan, then I’d be able to “survive” (ie not be above free plan) but that won’t solve my issue : I’d still be vulnerable to bots and my only way of blocking them would be custom rules, for which I’d be charged for.

It that correct again?

Then I wonder if there is a way to fix this issue with Vercel?
I can’t do anything in my Next app (because that’d mean Vercel has already been hit).
I could put a proxy in front of Vercel, but then I hardly see the interest of using Vercel as the stats would be diluted by the proxy?

Is there a solution I missed here (except self hosting) that would allow me to filter out malicious bots by user agents without being charged for it?
The event you point seems interesting, but Vercel firewall definitely considered 200k requests from from same user agent as legit (not even challenged) and there is no further configuration I can do on the firewall (unless there are hidden features for pro plan?).
I see there is rate limiting for pro plan but again, if what I want to do is pure blocking, not rate limiting, what are my options?

Don’t get me wrong : the features seem awesome, really. They just didn’t deliver in my case, so I’m a bit frustrated with the options left.

I’d really appreciate your input here. I enjoy using Vercel but I get messages every week telling me “warniiiiiing reaching limit” and there doesn’t seem to be anything I can do, which is frustrating because I have about 300-400 unique active users a month, and I can assure you they don’t visit thousands of pages each (would be fun though).
(I already applied ISR and cache and compression at least to reduce what I could reduce)

Thanks

4

It all depends on how you set up your project and custom rules. I recommend taking a look at persistent actions and reading through this guide: How to block bots from OpenAI GPTBot

I really recommend joining that live session with the Vercel security product team as well. They’re the experts so they have all the background knowledge to answer in a lot more detail than I can :smiley:

5

Thanks for your feedback.

We just reached the exact point I have an issue with.

First documentation mentions persistent actions. Great, I’d like to do that! But what’s described in the documentation doesn’t exist in the product (select a time frame). Seems like an interesting solution (get charged for first hit then everything else with same pattern blocked for a while).

Second document talks about the nasty bots, perfect! But it says that if i totally block a user agent, it shouldn’t reach my functions or static or whatever, and so it shouldn’t count as charging.
See this part :

Persistent Blocking (No Charge)

If you want to fully prevent GPTBot from crawling your site—and avoid incurring data transfer or function usage for these requests—you can persistently block it. Requests that match your block rule won’t reach your Vercel Functions or static pages, so you won’t be charged.

I also tried your firewall templates (great feature by the way) but they use custom rules. Sounds like the most convenient solution, but nothing mentions the financial aspect and Vercel usage limit here.

I’m interested in the event but I also need to be able to find a clear answer on that. No offense, but our relation is contract based and I want to make sure which actions I should/could/can take to avoid having my app “paused” because of a misunderstood documentation.

I’m really sorry to insist, but before engaging funds or committing to the product I need to know what will cost me money and what won’t.
I’m currently on hobby plan and could get my app pause.
If I go to pro plan and some bots go crazy over night, I might wake up being poor and I’d like to avoid that.
Hope you understand and don’t mind my insistance.

1 Like
6

Here this highlights perfectly the situation.
Traffic from yesterday :
I blocked half of the malicious traffic (not caught by the firewall) with a custom rule by denying Alibaba JA4.
Now when looking at the remaining Allowed traffic, I still have 2/3 of traffic coming from a bot (still not caught by the firewall).

Am I charged for the traffic from Alibaba ?
If I go for paid plan and keep blocking malicious crawlers based on user-agent or JA4, will I get charged for this ?
It feels wrong being charged for something I can’t fight against. Any help would be much appreciated

image
image2468×1304 232 KB

image
image2514×1272 288 KB

1 Like
7

i am having similar issues and concerns although not as far along as you are with creating the custom rules. I am getting hit sometimes with 20k hits a day from various bots. I added the ones I know about to my robots and perhaps it is being reduced a little bit?

it’s extra frustrating b/c I am aso using a paid API and burned through a few hundred dollars before realizing what was going on. (no shade please, this is my first deployment of a full stack app!).

it’s also hard to see what bots are hitting my deployment except for one by one right lcicking the user agent, copying it and pasting it into a blank texxt file. is this b/c I am on the hobby plan?

Like the first poster, I don’t want to pay for bot / spam traffic but havn’t found any great solutions for this, and this has to be a universal problem for everyone, not only vercel.

I don’t see thse issues however with wordpress hosting on cpanels.. .bots seem to be under control maybe by the hosting plan?

Thanks for any tips.

8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.