Cronjob stucks without error

Help
,
1

I have a cronjob that runs once a day and per each row of a table
it fetches external api and updates our db
(in parallel).
Cronjob fails on Vercel production (stucks without final error logs).
It works perfect If running as api route from my PC, all 600 rows are updated.

It only works good on Vercel production if we try to run it maximum for 50 rows,
this way all is good and we see the logs.

What we try to do:

  1. Show ‘Starting cronjob’ log
  2. map through N db rows (currently 650) and create promises, in each we fetch ext. api and update db
  3. await Promise.allSettled(promises)
  4. Show ‘Ending cronjob’ log

What we see on Vercel:

  1. ‘Starting cronjob’ log
  2. updating N db rows with errors*
  3. stucks on some row without final log (‘Ending cronjob’)
    errors* - fetch failed at node:internal/deps/undici/undici:13178:13 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async T (/var/task/.next/server/chunks/83.js:1:10610) at async /var/task/.next/server/app/api/cron-update-cashtags-prices/route.js:1:3870 at async Promise.allSettled (index 499) at async /var/task/.next/server/app/api/cron-update-cashtags-prices/route.js:1:4367 at async /var/task/.next/server/chunks/197.js:13:50753 at async /var/task/node_modules/next/dist/compiled/next-server/app-route.runtime.prod.js:6:34666 at async eS.execute (/var/task/node_modules/next/dist/compiled/next-server/app-route.runtime.prod.js:6:25813) at async eS.handle (/var/task/node_modules/next/dist/compiled/next-server/app-route.runtime.prod.js:6:35920) { [cause]: AggregateError [EMFILE]: at internalConnectMultiple (node:net:1118:18) at internalConnectMultiple (node:net:1186:5) at internalConnectMultiple (node:net:1186:5) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1522:7) at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:132:8) at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) { code: 'EMFILE', [errors]: [ [Error], [Error] ] } }

We can not get what’s actually happening here. Why don’t we see the final log, why it works locally and doesn’t on Vercel, why we see no clear error.

Thanks in advance! Please let me know if more details needed

2

Hi, @maxskurski! Welcome to the Vercel Community! :smile:

Thank you for providing a detailed description of your issue. Based on the information you’ve shared, it appears that you’re encountering limitations related to Vercel’s serverless function execution environment. Let’s break down the problem and explore some solutions.

Vercel has a maximum execution time limit for serverless functions. For Serverless Functions, this limit is 10 seconds on the Hobby plan, 60 seconds on the Pro plan, and 900 seconds (15 minutes) on the Enterprise plan .

Your cron job is likely exceeding this time limit when processing 600+ rows. This could explain why it works for 50 rows but fails for the full dataset.

Vercel also has memory limits for serverless functions. Exceeding these limits can cause the function to terminate abruptly without proper error logging.

Additionally, the EMFILE error you’re seeing suggests that you’re hitting a limit on the number of open file descriptors, which often translates to a limit on concurrent network connections. This is why you’re seeing fetch failures.

Here are some potential solutions to address these issues:

  1. Set up multiple cron jobs to process smaller batches of data at different intervals. This can help you stay within the execution time limits.
  2. Implement batching logic within your cron job.
  3. Use Vercel’s built-in cron job feature for more reliable execution.
  4. Implement continuation logic to resume processing from where it left off if the function times out. You can use Vercel’s KV store or your database to keep track of progress.
  5. Implement more robust error handling and logging to capture detailed information about failures. This will help you diagnose issues more effectively.

Let us know how you get on!

4

Hi, @maxskurski!

I know you marked my answer as the solution, but in case it’s helpful for anyone else reading this, we have a troubleshooting Vercel Cron Jobs guide we recently published.