Symptom
The Facebook Sharing Debugger returns 403 for https://www.devinterface.ai/it,
with the message "This response code could be due to a robots.txt block. Please
allowlist facebookexternalhit on your sites robots.txt config."
Because it never reads our HTML, Facebook falls back to inventing the preview from
the hostname: it reports og:title = www.devinterface.ai and an empty
og:description, even though the page serves a complete set of OG tags.
I have pressed "Scrape Again" more than three times in a row, over a period of hours. The result does not change.
LinkedIn's Post Inspector works correctly on the same URL.
The origin is healthy
All checks below were run against www.devinterface.ai resolved to 216.198.79.1,
at 2026-09-09 14:52 UTC.
Page, requested with Facebook's exact user agent:
HTTP/2 200x-matched-path: /itx-vercel-cache: HITx-vercel-id: fra1::bm9fw-1788965536822-be019f107accrobots.txt, requested with Facebook's exact user agent:
HTTP/2 200x-vercel-id: fra1::tk2m6-1788965537548-9488b4a016b4Other crawlers on the same URL: LinkedInBot 200 · Twitterbot 200 · Slackbot-LinkExpanding 200 · Googlebot 200.
I could not reproduce a 403 in any of eight request shapes: HTTP/1.1,
Accept: */*, no Accept-Encoding, HEAD, with a Range header, and four
variants of the Facebook user-agent string, plus a request with no user agent at
all. All returned 200, from a residential connection and from a datacenter
connection.
What I have already ruled out
robots.txt — served with HTTP 200, contains no Disallow at all, and now
names the crawler explicitly:
User-Agent: *Allow: /
User-Agent: facebookexternalhitUser-Agent: facebookcatalogUser-Agent: meta-externalagentUser-Agent: TwitterbotUser-Agent: LinkedInBot...Allow: /
Sitemap: https://www.devinterface.ai/sitemap.xmlAdding that explicit group changed nothing, which is expected: the file was already permissive.
Deployment Protection — not the cause. The custom production domain answers 200 to unauthenticated requests from any user agent, and if protection were the issue LinkedIn would fail too, which it does not.
Project firewall — nothing configured, and no record of any action taken:
Firewall Not configuredMitigations ActiveAttack Mode OffBot Protection OffAI Bots Allow
vercel firewall persistent-actions ls → No persistent actions in this window.vercel firewall alerts ls → No alerts in this window.Our application — the route is statically prerendered; x-vercel-cache: HIT
confirms the edge is serving it from cache rather than invoking a function.
What I am asking
Since nothing on the project side is returning or recording a 403, I believe the requests from Facebook's ASN (AS32934) are being dropped or challenged upstream of our project, by platform-level mitigation that does not appear in the project firewall log.
Could you please check:
- Whether platform-level mitigation is challenging or blocking AS32934 for this
domain, and if so, whether it can be excluded for
www.devinterface.ai. - Whether the fact that this domain was added and certified today
(Let's Encrypt certificate
notBefore = 2026-09-09 11:57:10 UTC) puts it in any state that treats unverified crawler traffic differently.
I am aware of the earlier community thread where this was described as a transient Facebook-side cache issue that cleared on its own. In our case repeated re-scrapes over several hours have not cleared it, which is why I am asking whether something on the platform side is involved.
I do not have IP Bypass available on our plan (vercel firewall system-bypass
returns 402, "Pro and Enterprise plans include it"), so I cannot apply the ASN
bypass workaround suggested in that thread.