Vercel Nextjs Googlebot soft 404 issues

[admin-finlywealth (Abid)]

First off, other Vercel customers also reporting this problem:

https://www.reddit.com/r/nextjs/comments/1ijmhqx/soft_404_on_nextjs_webpages_hosted_on_vercel/

Hi everyone, I’ve been dealing with a major indexing issue using the latest versions of Nextjs (15.1.6), React 19.0.0 as well as the PayloadCMS integrated in my site hosted on Vercel (for reference https://finlywealth.com/ is the home page).

This issue is practically impossible to reproduce on your own. I’ve tried almost everything you can think of, like the Chrome Canary with disabling javascript, changing user agent to Googlebot Phone, disabling caching, etc. Even when you use the “live test” of Google search console, the page is displaying just fine.

I’ve even bought a license to ScreamingFrog to crawl my entire site with the “googlebot” user agent, and so far everything works just fine. I ended up setting up Sentry on our Nextjs website to be able to narrow things down to what errors the googlebot crawler might be encountering, and pretty much it all comes down to this:

Error: Connection closed.
  at t(./node_modules/.pnpm/next@15.1.6_@babel+core@7.24.5_@opentelemetry+api@1.9.0_react-dom@19.0.0_react@19.0.0__react@19.0.0_sass@1.77.4/node_modules/next/dist/compiled/react-server-dom-webpack/cjs/react-server-dom-webpack-client.browser.production.js:1498:1)

Here’s a publicly sharable link on Sentry:

Error: Connection closed.

I’ve been trying to figure out what I’m doing wrong on our website for 2+ weeks now. The soft 404 issue is persistent on googlebot for most pages, even though a lot of these pages are static content. I’m using ISR and SSG for producing a lot of these pages, as well as integrated unstable_cache for DB calls via PayloadCMS to fetch data from our MongoDB Cloud.

I’ve even increased the maxDuration flag on all our pages to 60 as well as the settings on Vercel for functions durations to be 45. I’ve even boosted our CPU utilization to max on Vercel, hoping to get this resolved but still not successful.

[system]

There’s another community post with 404 debugging tips that might be helpful. Please give these solutions a try and let us know how it goes.

Debugging 404 Errors

Sometimes things don’t go as expected when deploying a website. If you see a 404 on your site, that means the page or resource couldn’t be found. There are many possible causes, and the absence of something can be tricky debug if you don’t know where to look. Here are some things you can do to find the cause and fix it. Debugging Tips Check the error code If you see a mostly white screen with 404: NOT_FOUND along with a Code and and ID then you can click the blue info box below the error deta…

A human should be around soon to offer more advice. But you can also get helpful information quickly by asking v0.

[pawlean (Pauline P. Narvas)]

Hey, Abid! Welcome to the Vercel Community.

I’ve not dug deep into the issue yet, but we did have a question about SEO crop up recently. @dom-vercel shared a solution that seemed to work for the OP. Cross-posting here in case it’s helpful.

Page not visible on Google Search despite SEO optimizations

Current versus Expected behavior: I have completed all necessary steps to make my website visible on Google Search, but my page still doesn’t appear in search results. The link and domain are working perfectly. I have submitted the sitemap, checked the robots.txt, and optimized the meta tags for SEO. I expected the site to appear on Google Search after following these steps, but it hasn’t. Steps taken: Verified and submitted the domain and sitemap in Google Search Console. Successfully teste…

[admin-finlywealth (Abid)]

Hi Pauline and thanks for responding. I believe that’s a very separate problem that we’re not affected by. Our main issue revolves around soft 404s coming from googlebot’s crawling of our nextjs pages. We use Server Side rendering on pretty much all of our pages and haven’t had any problems until more recently. It’s become quite bad now and we’ve been debugging for over 2 weeks now. We’re not able to pinpoint whether it’s a Vercel issue, a Nextjs issue or we’re doing something wrong.

[pawlean (Pauline P. Narvas)]

Ah, apologises. I dug a bit deeper and you’re right - it looks like a wider issue. Let me flag with the team internally