# One group for every crawler (iter 142). A crawler named in its own group # ignores the "*" group, so listing the AI crawlers here keeps the same # Disallow lines for them. Public pages stay open to search and AI crawlers. User-agent: * User-agent: OAI-SearchBot User-agent: GPTBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: ClaudeBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot User-agent: Applebot-Extended Disallow: /admin Disallow: /dashboard Disallow: /settings/ Disallow: /bookmarks Disallow: /checkout-success Disallow: /new/product Disallow: /p/*/edit Disallow: /waitlist/ # /login and /signup are public, indexable entry pages: intentionally allowed. # /forgot-password, /reset-password, /verify-email are kept out of the index via # X-Robots-Tag: noindex (next.config.js). They must NOT be Disallowed here, or # Googlebot could not crawl them to see the noindex header. # One sitemap index at the site root; it lists one sitemap per page kind plus # KittyDir's (/directories/sitemap.xml). Sitemap: https://kittylaunch.com/sitemap-index.xml # LLM guidance files for AI assistants (crawlable, nothing above disallows them) # Short version: https://kittylaunch.com/llms.txt # Full context: https://kittylaunch.com/llms-full.txt