Built a real-time AI bot tracker on Vercel + Neon in a weekend
Most site owners look at Google Analytics and think they understand their traffic.
They don’t. The most interesting visitors to your site never show up in GA at all.
I’m William. Retired construction worker. Self-taught programmer. I built https://aeofix.com — an Answer Engine Optimization platform — using Vercel, Neon, and Claude as my pair programmer.
A few months in, I noticed something: AI crawlers were hitting my site constantly and I had zero visibility into which ones, how often, or what pages they cared about. ChatGPT’s crawler. Anthropic’s. Perplexity. Grok. Google’s AI training bots. All invisible.
So I built a tracker. Here’s how it works.
The stack
- Vercel Edge Middleware — runs on every request, classifies the bot before the page loads
- Vercel Serverless Functions — /api/bot-pixel logs the visit; cron jobs run classification and digests
- Vercel KV — rate limiting (3 req / 2s; scrapers always trip this)
- Neon PostgreSQL — stores every visit with UA, ASN, country, page, confidence score, tier
- Gemini 2.5-flash — weekly narrative intelligence digest on crawl patterns
The pixel
Every page gets a 1×1 transparent GIF:
When it fires, the serverless function fingerprints the UA, looks up the ASN via ip-api.com, runs reverse DNS if it looks like Google, and writes the classified visit to Neon in under 50ms.
What it detects
50+ named bots across categories:
┌────────────────────┬──────────────────────────────────────────┐ │ Category │ Examples │ ├────────────────────┼──────────────────────────────────────────┤ │ AI Training │ GPTBot, ClaudeBot, Google-Extended, │ │ │ Mistral │ ├────────────────────┼──────────────────────────────────────────┤ │ AI Search │ OAI-SearchBot, PerplexityBot, │ │ │ ChatGPT-User, Grok │ ├────────────────────┼──────────────────────────────────────────┤ │ AI Assistant │ Amazonbot, Applebot-Extended │ ├────────────────────┼──────────────────────────────────────────┤ │ Search Index │ Googlebot, Bingbot, DuckDuckBot │ ├────────────────────┼──────────────────────────────────────────┤ │ SEO Tool │ AhrefsBot, SemrushBot, SE Ranking │ ├────────────────────┼──────────────────────────────────────────┤ │ Social Media │ Slackbot, LinkedInBot, TwitterBot │ ├────────────────────┼──────────────────────────────────────────┤ │ Vulnerability │ Nikto, SQLMap, ZGrab │ │ Scanner │ │ └────────────────────┴──────────────────────────────────────────┘
The honeypot
Unrecognized bots get silently redirected to a fake services page with wrong pricing. They never know they’re in it. HubSpot’s domain crawler goes straight there. So does anything that trips the rate limiter.
Surprising insight #1: When Slackbot-LinkExpanding hits your site, it means someone just pasted your URL into a Slack channel. That’s a positive signal — someone shared your content. I track it as Social Media, tier 3.
Surprising insight #2: “Googlebot” from a Google ASN ≠ verified Googlebot. The middleware does a reverse DNS lookup and checks the hostname ends in .googlebot.com or .google.com. Spoofers fail this. Real crawlers pass it and get logged as Googlebot (Verified) with 97% confidence.
The classification pipeline
Bots that slip through as Unknown Bot get a second chance. A cron job runs every 6 hours and re-processes them through 4 layers:
- ASN mapping (Google, Meta, Microsoft, Amazon IPs)
- UA substring patterns (50+ known signatures)
- Accept header analysis (image-only = rendering bot)
- Behavioral heuristics (10+ hits in 1 hour = aggressive scraper)
The dashboard
Six tabs: Overview, Bots, Pages, Countries, AI Funnel, and an Intelligence tab in progress. The AI Funnel tab cross-references bot crawl frequency against actual ChatGPT-referred sessions.
Gemini weekly digest
Every Monday at 9am UTC, Gemini 2.5-flash gets the week’s crawl data and writes a narrative report — which bots increased activity, which pages AI crawlers are focused on, anomalies worth watching. It lands in my inbox automatically.
I named my internal analysis bot Archimedes/1.0. My external site scanner uses aeofixtbot/1.0 and declares itself in robots.txt so site owners can allow or block it — because respecting robots.txt matters.
The whole thing runs on Vercel’s free/hobby tier + Neon’s free tier. Cold start on the pixel endpoint is around 80ms. Warm is under 20ms.
If you’re building anything in the AEO / AI visibility space and want to know which AI engines are actually crawling your site — this is the answer.
Happy to answer questions on the architecture, the Neon schema, or the Gemini prompt structure. The live dashboard is at https://aeofix.com/ai-bot-tracker if you want to see what it looks like.