Implementing upload progress tracking with Vercel Blob

Discussions
,
1

I want to share an interesting technical challenge I faced while working on tracking file upload progress for Vercel Blob (initially shared on Twitter/X) Sounds straightforward, right? Well…

I quickly discovered that tracking upload progress with fetch using .addEventListener('progress') wasn’t possible. Instead, I had to devise a solution combining Readable and Transform streams, with XMLHttpRequest as a fallback for older browsers.

The Solution

While there’s no way to .addEventListener('progress') to fetch calls, you can pass a ReadableStream (the “web” ones) to fetch. This is possible thanks to streaming requests and the duplex fetch option.

source: https://developer.chrome.com/docs/capabilities/web-apis/fetch-streaming-requests

fetch("https://vercel.com", {
  body: stream,
  duplex: "half",
});

Why use a stream? Because you can track its consumption via a TransformStream. Like this:

function trackStream() {
  let bytesRead = 0;

  return new TransformStream({
    transform(chunk, controller) {
      bytesRead += chunk.byteLength;
      console.log(`Read ${bytesRead} bytes`);
      controller.enqueue(chunk);
    },
    flush() {
      console.log(`Read ${bytesRead} bytes`);
    }
  });
}

Going back to our fetch example, you can now track its progress:

fetch("https://vercel.com", {
  body: stream.pipeThrough(trackStream()),
  duplex: "half",
});

That’s it!

Well, almost…

Browser compatibility challenges

I discovered that not all browsers support streaming requests yet:

For browsers that don’t support streaming requests, you have to fall back to XMLHttpRequest. Here’s an example:

const xhr = new XMLHttpRequest();
let bytesRead = 0;

xhr.upload.addEventListener('progress', (event) => {
 bytesRead = event.loaded;
 console.log(`Progress: ${bytesRead}/${event.total} bytes`);
});

xhr.addEventListener('load', () => {
 console.log('Upload complete! Total sent:', bytesRead);
});

xhr.open('POST', 'https://vercel.com');
xhr.send('Hello World');

To detect whether or not your environment supports streaming requests, you would need this code:

import { isNodeProcess } from 'is-node-process';

const supportsRequestStreams = (() => {
  if (isNodeProcess()) {
    return true;
  }

  let duplexAccessed = false;

  const hasContentType = new Request('https://vercel.com', {
    body: new ReadableStream(),
    method: 'POST',
    get duplex() {
      duplexAccessed = true;
      return 'half';
    },
  }).headers.has('Content-Type');

  return duplexAccessed && !hasContentType;
})();

Advanced implementation

The TransformStream code is not enough because when you get a body, it may not be chunked at all the way you want it to be to track progress, so you have to re-chunk it yourself for the upload progress to be smooth.

function trackProgress(
  chunkSize: number = 64 * 1024,
  onProgress?: (bytes: number) => void,
): TransformStream<ArrayBuffer | Uint8Array> {
  let buffer = new Uint8Array(0);

  return new TransformStream<ArrayBuffer, Uint8Array>({
    transform(chunk, controller) {
      // Use microtasks otherwise this is synchronous
      queueMicrotask(() => {
        // Combine the new chunk with any leftover data
        const newBuffer = new Uint8Array(buffer.length + chunk.byteLength);
        newBuffer.set(buffer);
        newBuffer.set(new Uint8Array(chunk), buffer.length);
        buffer = newBuffer;

        // Output complete chunks
        while (buffer.length >= chunkSize) {
          const newChunk = buffer.slice(0, chunkSize);
          controller.enqueue(newChunk);
          onProgress?.(newChunk.byteLength);
          buffer = buffer.slice(chunkSize);
        }
      });
    },
    flush(controller) {
      queueMicrotask(() => {
        // Send any remaining data
        if (buffer.length > 0) {
          controller.enqueue(buffer);
          onProgress?.(buffer.byteLength);
        }
      });
    }
  });
}

Why I didn’t use axios

You might be wondering, “Why not just use axios?” Well, I’ll admit our code is heavily inspired by it. However, we wanted to keep our bundle small and modern. This approach allowed us to create a more tailored solution specific to Vercel Blob’s needs.

Want to learn more?

If you’re interested in digging deeper, check out the Vercel Blob documentation and our source code on GitHub.

Feel free to share your thoughts or ask any questions!

2 Likes