User-agent: * # /app/ is the Vue SPA. It used to be blocked here, which is why some # URLs showed up in Google with "No information is available for this # page" — the bot could see the URL via inbound links but couldn't # fetch the body. Allowed now so Googlebot can render the SPA and pick # up per-route meta tags injected by Pinia + the baseline ones in # spa_base.jinja2. # Public read-only data endpoints the SPA needs at render time — without # these Googlebot renders /app/lp/* and the catalog as an eternal spinner # (the category tree and survey steps are fetched client-side). Longest # match wins, so the rest of /api/ stays blocked. Allow: /api/v1/structure/ Allow: /api/v1/jobs/processes/ Disallow: /api/ Disallow: /cms/ Disallow: /swagger/ Disallow: /redoc/ # Authenticated SPA routes — nothing useful to index, and they 302 to # /login anyway when crawled unauthenticated. Disallow: /app/chats/ Disallow: /app/profile/ Disallow: /app/settings/ Disallow: /app/notifications/ Disallow: /app/balance/ # Password-reset URLs carry a secret hash in the path # (/app/reset-password///). Belt-and-suspenders with the # SPA-side — keep the # token out of any crawl log. Disallow: /app/reset-password/ # Auth landing pages — no useful content for the search index, and the # SPA already serves noindex on these routes. Disallow: /app/signin Disallow: /app/signup Disallow: /app/forgot-password Allow: / # --------------------------------------------------------------------------- # AI / LLM crawlers — explicit policy. Public marketing surface (blog, # Wagtail landings, service landings, the how-it-works explainers, the # licence tool) is allowed for discovery and citation in AI answers; the # rest of the SPA, the API and the admin stay blocked. # # One group for every AI bot: RFC 9309 lets a group name several # User-agent lines, and the bots all get the same rules — ten copies of # the block only invited drift. A bot named here obeys ONLY this group and # does not inherit anything from `*` above. User-agent matching is # case-insensitive, so one spelling per bot is enough. # # Rules are ordered most-specific first so a first-match parser (e.g. # Python's urllib.robotparser) reaches the same verdict as the # longest-match rule of RFC 9309. # --------------------------------------------------------------------------- # OpenAI ChatGPT browsing / search User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User # Anthropic Claude: training crawler, search indexing, user-requested # fetches. Claude-Web and anthropic-ai are retired names, kept for old # clients. User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Claude-Web User-agent: anthropic-ai # Perplexity User-agent: PerplexityBot User-agent: Perplexity-User # Google-Extended is a control token, not a crawler: it governs use of the # content for Gemini training and grounding. Search and AI Overviews fetch # with Googlebot, which follows the `*` group above. User-agent: Google-Extended # Common Crawl (foundation dataset for many LLMs) User-agent: CCBot # Apple Intelligence User-agent: Applebot-Extended # Meta AI User-agent: Meta-ExternalAgent User-agent: FacebookBot # Mistral User-agent: MistralAI-User # You.com User-agent: YouBot # Bytedance / Doubao User-agent: Bytespider # The public corners of the SPA: service landings (/app/lp/), # the how-it-works explainers (llms.txt links both), and the licence-check # tool. Allow: /app/lp/ Allow: /app/how-it-works/ Allow: /app/license-check Disallow: /app/ Disallow: /api/ Disallow: /cms/ Disallow: /swagger/ Disallow: /redoc/ Allow: / Sitemap: https://everlook.ae/sitemap.xml