Seeing other accounts blob storage data

(edited)

There seemed to have been a very troubling bug this morning coming from one of our customers accounts. Their site started reporting 500 errors and when we went to check; it was coming from the application pulling data from the Blob service. When we checked on the data it was because the blob storage data we were pulling was not our own but instead data from presumably another customers account.

Is this a known issue on Vercels side and if so was this reported anywhere and are there any steps to fix it ASAP. I'd like to understand how this happened and what has been done to ensure this never happens again so i can explain it to our clients.

Vercel Staff

Thank you for reaching out about this. I'm checking with the storage team now and I'll let you know what I learn ASAP

reply to #2

Thanks! If possible I'd just like to understand what might have happened and how it was mitigated so I have something to give back to the client.

Vercel Staff

What I know so far is there was a bug introduced at Oct 28 12:16 PM UTC and fixed at Oct 28 1:33 PM UTC. The team is still investigating the full scope of its impact. I'll follow up with the latest as soon as I learn more the team

Is there a reason this didn’t show up within the status page either? We incorrectly told our client there were no issues on the platform side because the status page said no incidents reported and everything was operational

Is there an update for this issue?

Vercel Staff

At this time, we don't have new information to share, but we will follow up with you as soon as we do. We understand the importance of this matter and are treating it with the highest priority.

reply to #8

Vercel Staff

Thank you for your patience. We introduced additional unit tests and made changes within the API design to prevent recurrence.

The URLs provided were invalid and if clicked would have returned a 404.
Our investigation did not uncover any activity related to malicious intent, and we have no evidence of the contents of your blob store being accessed by unauthorized parties.
No account or personal information was accessed. The information exposed within the URL path consisted solely of store ID and filename.

We apologize for any inconvenience this may have caused, and please let us know if you have any additional questions. We would be happy to assist.

Is there a reason it wasn't published to the status site? We looked a bit silly to the client since and the fact that a security issue like this wasn't even documented in https://www.vercel-status.com comes off as if it was swept a bit under the rug.

I understand that issues happen, but ones like this where blob data is revealed to other accounts,I'd expect a bit more public exposure so folks who don't know at least have a record of it for the future. No one from Vercel reached out via email or messaged us about it until we brought it up.

Looking for a bit more feedback from previous question.

reply to #12

Vercel Staff

Thank you for waiting. Multiple factors inform the decision to update Vercel’s status page, such as product availability (beta, pre-release, etc.), response obligations (i.e. service level agreements), level of severity and impact. Our investigation did not indicate an update to our status page was necessary in this instance.

I see that someone else from your team also reached out in a support case. You can find the details on the Support tab of your team dashboard. Please follow up with the Customer Success Engineer there if you have more questions. The team has more direct knowledge about the bug and will be better able to provide any additional info you need.

reply to #14

Will do thanks for the information

This topic is closed. It no longer accepts new replies.