Cannot download zip file into /tmp

chihiro-yabuta · September 28, 2024, 9:47am

Current versus Expected behavior

I am considering deploying an app that scrapes a certain page.
I am trying to download Chromium to the /tmp folder using puppeteer-core.
However, when I attempt to download the zip file, the process stops without any errors.
I would like to ask about this issue.

Code, configuration, and steps that reproduce this issue

github.com

chihiro-yabuta/opas/blob/main/pages/api/scrape.ts#L19


      
          
          export async function scrape(cli: RedisClientType, reqRegion: string, reqGenre: string) {
            let browserInit: Browser;
            let pageInit: Page;
            let resObj: Response = {};
            const key = `opas?region=${reqRegion}&genre=${reqGenre}`;
            const logObj = (l: string): Log => { return { region: reqRegion, genre: reqGenre, log: l } };
          
            try {
              if (!fs.existsSync('/tmp/chromium.zip')) {
                await new Promise((resolve, reject) => https.get(url,
                  (res) => https.get(res.headers.location, (r) => {
                    const writableStream = fs.createWriteStream('/tmp/chromium.zip');
                    r.pipe(writableStream);
                    r.on('end', resolve);
                    r.on('error', reject);
                  })
                ));
          
                const zip = new AdmZip('/tmp/chromium.zip');
                zip.extractAllTo('/tmp', true);

The process after the pipe in this promise is stopping.

github.com

chihiro-yabuta/opas/blob/main/pages/api/index.ts#L36


      
          
            if (cond?.status === 'in-progress') {
              return res.status(200).json(resObj ? resObj : cond);
            } else {
              if (!resObj || updt) {
                await cli.set('opas', JSON.stringify({ status: 'in-progress', key: key }));
          
                !process.env.VERCEL && await mkdir('log', { recursive: true });
                !process.env.VERCEL && await writeFile(`log/${reqRegion}${reqGenre}.log`, '');
          
                scrape(cli, reqRegion, reqGenre);
                return res.status(200).json({ status: 'in-progress', key: key });
              } else {
                return res.status(200).json(resObj);
              }
            }
          }

As additional information, this function is being executed asynchronously because it takes a long time.

Project information

Deployment URL: https://vercel.com/chihiro-yabutas-projects/opas/FsmSNi1ENmowwoCygN8LcFw3ZZyd
Environment (local, preview, production): production
Project Framework: nextjs
Build Settings:
  Framework Preset: nextjs
  Build Command (if not default):
  Output Directory (if not default):
  Install Command (if not default):
Node/Runtime Version:
Package Manager:
Relevant Packages: https, fs

pawlean · October 1, 2024, 5:27pm

Hi, @chihiro-yabuta! Welcome to the Vercel Community.

Vercel uses a serverless architecture, which has certain limitations . The /tmp directory in serverless environments is typically limited in size and not persistent between function invocations.

Installing Chromium or Puppeteer in a serverless environment is generally not recommended or supported. The installation process requires more resources and time than typically available in a serverless function execution.

It’s also worth pointing out from our our Terms of Service:

You may not use the Services or Vercel’s infrastructure for proxying, scraping, to create virtual private networks, or to create virtual private servers.

chihiro-yabuta · October 2, 2024, 3:33am

thank you. I’ll consider another way.

system · October 4, 2024, 3:33am

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
V0: Upload a Project v0 v0	19	286	May 7, 2025
Command line and zipfile not working in ide Help nextjs	1	7	March 20, 2025
"Could not find Chromium" error on Vercel Help	2	45	January 12, 2025
🛡️ CSRF Protection & Scraping Prevention Issue Help nextjs	4	35	December 18, 2024
External package build configuration to works with nextjs Help nextjs , turborepo	2	34	October 24, 2024

Cannot download zip file into /tmp

Current versus Expected behavior

Code, configuration, and steps that reproduce this issue

Project information

Related topics