New to Vercel, having some issues with load testing blob fetches

Hi, I'm writing a web app to help coordinate things during a few hackathons coming up in November.  I'm new to Next.js and Vercel.  The app is working well in manual testing, but when I tried load testing it to make sure that it would work for the largest of the hackathons (300 participants) I started getting failures fetching blob contents.  

I'm not seeing any ddos hits in the firewall, and the page load itself works fine.  It's just the blob fetch by its URL that's failing when the usage ramps up.  I'm working on the Hobby plan at the moment.

What I'm simulating is the point in the hackathon when students in an auditorium will scan a projected QR code and enter their code to see what room they need to go to and who their team members are.  Every student will be fetching the same blob by URL over the course of a few minutes.

I see that there's a limit of 20 simple blob operations per second, and that a fetch by URL is counted as a simple operation if it's a cache miss.  The blob being fetched is the same for all the students.  I had expected that the first fetch would be a cache miss and the rest would find the blob in the local region cache, and not count against the limit.

But that doesn't appear to be happening, since I'm getting failed fetches when I ramp up the load testing.  

Am I thinking about this right that the fetch failures are likely due to exceeding the Hobby plan limit of 20 simple operations per second?  Or should the cache be avoiding that and I'm seeing failures for some other reason?

Vercel Staff

Hi Jay,

Load testing is available only on Enterprise plan currently: https://vercel.com/guides/what-s-vercel-s-policy-regarding-load-testing-deployments.

Umm, okay? My question wasn’t about how to do load testing. It was about blob fetch failures during load. I asked some specific questions about how the region cache affects blobs, and how it didn’t appear to be working the way I thought it would, and whether I was thinking right or not. Your reply does not answer any of those questions, and doesn’t help with the issue (the link you posted also has no mention of blobs and caching).

Can you go back and reread the original post and help with the questions in it? Or, if I’ve totally misunderstood the content at the link you posted, explain it to me in simpler terms.

My main concern is to ensure that under the real conditions of a Hackathon (also stated in my original post) that we don’t get 75% blob fetch failures.

Vercel Staff

Yes, Blob storage also has Firewall protection. If you try to fetch the same Blob within a short period, it will trigger a security challenge that only a web browser can solve.

Thanks!

How would I be able to verify that what you’re saying is happening is actually what’s happening? I go to the Firewall section in the Vercel dashboard for the project, and I see zero in the Challenged and Denied categories. Is that where blob related challenges should be counted? Or is there a separate section for the blob firewall?

Assuming it’s the blob related challenges that are causing the issue, can you confirm my understanding of these?

Blob related challenges do not happen if the content is fetched from cache
One of the site staff doing a student team lookup before the students are asked to do so will make sure it’s in the region cache before the students try to access it
If a blob related challenge does happen, the students will all be using mobile web browsers and will be able to navigate the challenge to get to the content

Edit: I ran the test with more logging, and the failed fetches are returning an HTTP 503 error. That doesn’t suggest a blob challenge, right? I’m still not seeing anything in the firewall about challenged, denied, or DDOS mitigation requests.

Vercel Staff

If you use cache-control header and access from cache, it indeed shouldn’t return 403 by Firewall.

I ran the test with more logging, and the failed fetches are returning an HTTP 503 error

Do you have any screenshot / link of logs where we can check this?

Okay, I suspect I’ve been using blobs for the wrong reason. I was using them to store dynamic data that would be updated as users interacted with the app. Then I ran into caching issues, and went down a rabbit hole of trying to make blobs work for that purpose. Where blobs seem to be intended for longer-lived data. I’ve decided to move to Upstash’s KV database since it seems more intended for the sort of dynamic data I’m working with, where multiple people can change data at the same time.

I still don’t know why I was getting the 503 errors on the blobs when testing at scale, but it doesn’t seem like a problem I’ll need to solve now. Thanks for your help.

This topic is closed. It no longer accepts new replies.

Hi, I'm writing a web app to help coordinate things during a few hackathons coming up in November. I'm new to Next.js and Vercel. The app is working well in manual testing, but when I tried load testing it to make sure that it would work for the largest of the hackathons (300 participants) I started getting failures fetching blob contents. I'm not seeing any ddos hits in the firewall, and the page load itself works fine. It's just the blob fetch by its URL that's failing when the usage ramps up. I'm working on the Hobby plan at the moment. What I'm simulating is the point in the hackathon when students in an auditorium will scan a projected QR code and enter their code to see what room they need to go to and who their team members are. Every student will be fetching the same blob by URL over the course of a few minutes. I see that there's a limit of 20 simple blob operations per second, and that a fetch by URL is counted as a simple operation if it's a cache miss. The blob being fetched is the same for all the students. I had expected that the first fetch would be a cache miss and the rest would find the blob in the local region cache, and not count against the limit. But that doesn't appear to be happening, since I'm getting failed fetches when I ramp up the load testing. Am I thinking about this right that the fetch failures are likely due to exceeding the Hobby plan limit of 20 simple operations per second? Or should the cache be avoiding that and I'm seeing failures for some other reason?

Thanks!

Assuming it’s the blob related challenges that are causing the issue, can you confirm my understanding of these?

Blob related challenges do not happen if the content is fetched from cache
One of the site staff doing a student team lookup before the students are asked to do so will make sure it’s in the region cache before the students try to access it
If a blob related challenge does happen, the students will all be using mobile web browsers and will be able to navigate the challenge to get to the content