Production error - Resetting PGBouncer pool connections - Help

Hi, I'm facing an error in production where all database write operations fail due to a read-only connection being in the connection pool:

I believe this connection was created by a developer using a database client like dbeaver and connecting to the connection pooler (pgbouncer) using "read only" mode. After that the connection stayed in the pool and is completely breaking the application.

While researching this issue I came across this issue in the pgbouncer repository where someone mentions that restarting pgbouncer or running a "DISCARD ALL" command that can be done to reset the session status:

https://github.com/pgbouncer/pgbouncer/issues/176

Does anyone know if this can be done with the pgbouncer instance provided by vercel? I would appreciate any help since this is completely breaking the application in production.

Hey . I don't know the answer to this one and don't want to lead you astray. I reached out to our storage team for advice and we'll keep you updated.

Hey could you confirm a few things:

Are you using https://vercel.com/storage/postgres?
Was this error triggered by your Vercel Postgres database?
What's your setup (npm modules, versions) to connect to your Vercel Postgres database?

Thanks!

HI .

Yes, we are using Vercel's Postgre solution (connetion string is something like "postgres://default:PcM4eLXXXXXX@ep-twiXXXXt-XXXX-29XXXXXX-pooler.us-east-1.postgres.vercel-storage.com/verceldb?pgbouncer=true&connect_timeout=15").

On the application we connect to it only using Prisma ORM (/client": "^5.17.0).

I made sure to turn off my "read only" toggle for connecting with outside tools like Dbeaver and to not connect to the database using the connection pooler URL and it stopped happening, so I'm sure that was causing the issue. My theory is that after creating a "read only" connection through the pooler, the connection was reused by other pool users (like Prisma's client).

It's kind of wild that a read operation on the database could break the entire application in production, but it is what it is.

Hey thanks for the details, I am reaching out to get even more information 👍

Finally, could you send us your team id? You can find it here: https://vercel.com/docs/accounts/create-a-team#find-your-team-id

Hi . My team ID is team_Zd6wDWaSI2Soh3RgyKejdruM

Hi again would you be able to provide me with a time when this error occurred? And the timezone if relevant, thanks!

Hi . It happened on July 23rd 2024 at 16:56:10 UTC for sure, and quite frequently around that time frame as well as the day before.