My next.js app is deployed on Vercel and connecting to GC PostgreSQL DB via Google cloud SQL connector using an IAM service account with OIDC tokens. I have the latest version “@google-cloud/cloud-sql-connector”: “^1.8.3” and set up connection according to Vercel and google documentation. My preview/dev deployment works fine, but my prod gets intermittent connection errors several times a day that look like this:
SSL alert number 42
library: ‘SSL routines’,
reason: ‘ssl/tls alert bad certificate’,
code: ‘ERR_SSL_SSL/TLS_ALERT_BAD_CERTIFICATE’
I have not found anything in documentation or troubleshooting, could it be certificate rotation? Postgres logs don’t show any errors aroudn the time it happens.
I have checked my Vercel environment variables and they are the same for preview and production, except DB instance and name of course, there are no SSL settings there. The prod and dev DBs in GC seem to also have the same network configuration, enforce SSL and use the same Workload Identity Pool with OIDC and the same service account.
In my Vercel configuration, there should be no egress restrictions, and there are no errors in external calls to google api as per Vercel observability.
Nothing suspicious in the SQL logs, just alert about terminating connection due to idle-session timeout, because my pool keeps connections open even idle.
This is what my db.ts looks like (now I am trying min 1 pool connection hoping it will catch the SSL problem and reset instead of the user, but before I had min 0 and allowExitOnIdle: false and it had the same problem):
import { Pool, QueryResult, PoolClient } from ‘pg’;
import { Connector, AuthTypes, IpAddressTypes } from ‘@google-cloud/cloud-sql-connector’;
import { ExternalAccountClient, AuthClient } from ‘google-auth-library’;
import { getVercelOidcToken } from ‘@vercel/functions/oidc’;
import { logger } from ‘./logger’;
let pool: Pool | null = null;
declare global {
// eslint-disable-next-line no-var
var getPool: Promise | undefined;
}
function createAuthClient(): AuthClient {
return ExternalAccountClient.fromJSON({
type: ‘external_account’,
audience: //iam.googleapis.com/projects/${process.env.GCP_PROJECT_NUMBER}/locations/global/workloadIdentityPools/${process.env.GCP_WORKLOAD_IDENTITY_POOL_ID}/providers/${process.env.GCP_WORKLOAD_IDENTITY_POOL_PROVIDER_ID},
subject_token_type: ‘urn:ietf:params:oauth:token-type:jwt’,
token_url: ‘https://sts.googleapis.com/v1/token’,
service_account_impersonation_url: https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${process.env.GCP_SERVICE_ACCOUNT_EMAIL}:generateAccessToken,
subject_token_supplier: { getSubjectToken: getVercelOidcToken },
}) as AuthClient;
}
async function getPool(): Promise {
if (global.getPool)
return global.getPool;
global.getPool = (async () => {
logger.info(‘Database pool connecting’, ‘getPool’, { instance: process.env.DB_INSTANCE, user: process.env.DB_USER, database: process.env.PGDATABASE });
const connector = !!process.env.VERCEL ? new Connector({ auth: createAuthClient() }) : new Connector();
const clientOptions = await connector.getOptions({ instanceConnectionName: process.env.DB_INSTANCE || ‘’, authType: AuthTypes.IAM, ipType: IpAddressTypes.PUBLIC });
pool = new Pool({ …clientOptions, user: process.env.DB_USER, connectionTimeoutMillis: 10000, idleTimeoutMillis: 60000, min: 1, allowExitOnIdle: true, application_name: ‘seonali-app’ });
pool.on(‘error’, (e: Error) => logger.error(‘Database pool error’, ‘getPool’, e));
pool.on(‘connect’, (client) => client.query(‘SET statement_timeout = 30000; SET idle_in_transaction_session_timeout = 120000; SET idle_session_timeout = 600000’));
setInterval(() => logger.info(Pool health check: total:${pool?.totalCount}, idle:${pool?.idleCount}, ‘poolMonitor’), 60000);
logger.info(‘Database pool initialized’, ‘getPool’);
return pool;
})().catch((e) => {
global.getPool = undefined;
logger.error(‘Database connection failed, resetting pool’, ‘getPool’, e);
throw e;
});
return global.getPool;
}
export async function tryQuery(query: string, callsite: string ,params?: unknown): Promise {
try {
const pool = await getPool();
const result = await pool.query(query, params);
return result;
} catch (error: unknown) {
let errorMessage = ‘Database query failed’;
const err = error as { message?: string; code?: string };
if (err?.message?.includes(‘ssl’) || err?.message?.includes(‘certificate’) || err?.message?.includes(‘connection’) || err?.code?.startsWith(‘ECONN’)) {
global.getPool = undefined;
errorMessage = ‘Database SSL/connection failed, resetting pool’;
}
logger.error(errorMessage, callsite, error);
throw error;
}
}
I would appreciate any help, I’m at the point where I’m considering moving to GC with my app as well to get rid of this, I can’t have my app useless for minutes regularly…
Martin