Hi, @maxskurski! Welcome to the Vercel Community! 
Thank you for providing a detailed description of your issue. Based on the information you’ve shared, it appears that you’re encountering limitations related to Vercel’s serverless function execution environment. Let’s break down the problem and explore some solutions.
Vercel has a maximum execution time limit for serverless functions. For Serverless Functions, this limit is 10 seconds on the Hobby plan, 60 seconds on the Pro plan, and 900 seconds (15 minutes) on the Enterprise plan .
Your cron job is likely exceeding this time limit when processing 600+ rows. This could explain why it works for 50 rows but fails for the full dataset.
Vercel also has memory limits for serverless functions. Exceeding these limits can cause the function to terminate abruptly without proper error logging.
Additionally, the EMFILE error you’re seeing suggests that you’re hitting a limit on the number of open file descriptors, which often translates to a limit on concurrent network connections. This is why you’re seeing fetch failures.
Here are some potential solutions to address these issues:
- Set up multiple cron jobs to process smaller batches of data at different intervals. This can help you stay within the execution time limits.
- Implement batching logic within your cron job.
- Use Vercel’s built-in cron job feature for more reliable execution.
- Implement continuation logic to resume processing from where it left off if the function times out. You can use Vercel’s KV store or your database to keep track of progress.
- Implement more robust error handling and logging to capture detailed information about failures. This will help you diagnose issues more effectively.
Let us know how you get on!