Summary
A monorepo backend service (Express + Prisma, deployed via vercel.json's services schema) started failing 100% of requests with Cannot find module 'express' immediately after a routine git push to main — no dependency, build script, or vercel.json change was involved. Isolated the regression to a Vercel platform-side change between CLI 58.1.0 (working) and 58.9.5 (broken), both auto-selected by Vercel's build infrastructure, not pinned by us.
Environment
- Project: monorepo with vercel.json services schema (not legacy builds/routes, not npm/pnpm workspaces)
- Backend service config: "backend": { "root": "backend", "entrypoint": "src/index.ts" }
- Backend stack: Express 4.18.3 (declared directly in backend/package.json, not hoisted from any workspace root), TypeScript compiled via tsc to CommonJS, Prisma 5.22.0
- Project's configured Node.js runtime: nodejs24.x
Symptom
Every request to every backend route (/api/curriculum/grades, /api/otp/email/send, /api/auth/providers, etc.) returned:
A server error has occurred FUNCTION_INVOCATION_FAILED
Function logs showed:
Cannot find module 'express' Require stack:
- /var/task/index.js
Did you forget to add it to "dependencies" in
package.json? Node.js process exited with exit status: 1.
What was ruled out (each independently verified)
- Not a missing dependency — express is correctly declared in backend/package.json dependencies and present in package-lock.json.
- Not a build failure — every deploy's build log showed ✓ Compiled successfully / ✓ Typecheck complete with zero errors, npm run build exiting 0.
- Not a local reproduction — ran the exact production build command locally (VERCEL=1 npm run build, then node dist/index.js with only dist/ + node_modules/ present, i.e. the same artifact Vercel would package). It ran perfectly: require('express') resolved, server started, /health returned 200.
- Not our own migration/DB change — diffed the last known-healthy commit against the first known-broken commit; the only file difference was a single additive Prisma migration .sql file (no JS/TS/config change of any kind).
- Not a stray root package.json — found and removed an unrelated, accidentally-committed root-level package.json/package-lock.json (leftover from ~3 weeks earlier) as a precaution. Confirmed via a fresh no-cache deploy that this was not the cause — the identical error persisted after removal.
- Not the build cache — reproduced on multiple vercel deploy --prod --force (skips build cache) and after vercel cache purge (clears CDN + Data cache). Same failure every time.
- Not our runtime config — attempted to pin "runtime": "nodejs20.x" on the backend service as a diagnostic; Vercel rejected it outright: Error: Service "backend" has invalid runtime "nodejs20.x". Reverted; the account's available runtime appears to be nodejs24.x only (or at least what's currently active on this project).
Root cause, isolated
Compared full build logs of the last confirmed-healthy deployment vs. the first confirmed-broken deployment (build logs available in Vercel dashboard, deployment IDs below). The only difference in the build log is the Vercel CLI version used internally by the build machine:
┌─────────┬────────────────────────────────────────────────────────────────┬─────────────────────────┬───────────────────────────────────────────────────────┐ │ │ Deployment │ Vercel CLI (build-time) │ Result │ ├─────────┼────────────────────────────────────────────────────────────────┼─────────────────────────┼───────────────────────────────────────────────────────┤ │ Healthy │ dpl_AFgz42vsc8n5GAQiNwUKyPpd6o5o │ 58.1.0 │ All routes 200 │ ├─────────┼────────────────────────────────────────────────────────────────┼─────────────────────────┼───────────────────────────────────────────────────────┤ │ Broken │ dpl_H2b21km1qVFnr2cZyeSEKCw1DN3J (and every subsequent deploy) │ 58.9.5 │ Every backend route 500, Cannot find module 'express' │ └─────────┴────────────────────────────────────────────────────────────────┴─────────────────────────┴───────────────────────────────────────────────────────┘
No project setting, package.json, or vercel.json change controls this CLI version — it's selected by Vercel's platform per-build. Something shipped between 58.1.0 and 58.9.5 broke /nft's (or an equivalent internal step's) dependency tracing for this service's Node.js function, causing express (and likely any other function-local node_modules dependency) to be silently excluded from the deployed bundle despite the build step completing successfully.
Workaround currently in place
Manually aliased production (vercel alias set sophoz.com) to the last known-good deployment built under CLI 58.1.0. vercel rollback to the same deployment was blocked by the account's plan tier (To rollback further than the previous production deployment, upgrade to pro (402)), but direct alias set was not restricted and worked.
This is a stopgap — the project cannot currently accept any new deploy without risking hitting the same regression, since we have no way to pin the build-time CLI version from our side.
Ask
- Confirm this is a known regression in the 58.x build pipeline (Node File Trace / function bundling step) for the services-schema monorepo format.
- If a fix has already shipped past 58.9.5, confirm it's safe to redeploy.
- If not yet fixed, is there any project-level way to pin/downgrade the build-machine's internal CLI/tracing version until a fix ships?
Deployment IDs for reference
- Last healthy: dpl_AFgz42vsc8n5GAQiNwUKyPpd6o5o (commit 503eab9, built with CLI 58.1.0)
- First broken: dpl_H2b21km1qVFnr2cZyeSEKCw1DN3J (commit 695e164, built with CLI 58.9.5)
- Several subsequent deploys, all broken, all under CLI 58.9.5 (available on request)