# https://www.robotstxt.org/robotstxt.html # AI crawlers are named explicitly so they are never blocked by accident. They # share this one group: a crawler that matches its own group ignores the "*" # group, so separate groups would let them into the app paths below. User-agent: * User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: CCBot Allow: / # Logged-in app. Retired pages aren't listed: they return 404, and crawlers # need to be able to see that to drop them from the index. Disallow: /commenting Disallow: /focus Disallow: /inbox Disallow: /analytics Disallow: /posted-comments Disallow: /preferences Disallow: /settings Disallow: /platform Disallow: /posts Disallow: /qr-login Disallow: /login-extension Disallow: /onboarding Disallow: /checkout Disallow: /health Sitemap: https://commenter.ai/sitemap.xml Sitemap: https://commenter.ai/blog/sitemap_index.xml