# ============================================================================ # ROBOTS.TXT FOR SOUNDPROOFWINDOWS.COM # ============================================================================ # V2: This file ALLOWS all AI crawlers on the main site and PRESERVES the # site's original robots.txt rules for regular crawlers. # # Last updated: April 2026 # ============================================================================ # ============================================================================ # AI CRAWLERS - ALLOWED (V2) # ============================================================================ User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: anthropic-ai Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: GoogleOther Allow: / User-agent: GoogleOther-Image Allow: / User-agent: GoogleOther-Video Allow: / User-agent: Google-CloudVertexBot Allow: / User-agent: Google-Firebase Allow: / User-agent: Google-NotebookLM Allow: / User-agent: GoogleAgent-Mariner Allow: / User-agent: Gemini-Deep-Research Allow: / User-agent: Gemini-AI Allow: / User-agent: Bard-AI Allow: / User-agent: Copilot Allow: / User-agent: FacebookBot Allow: / User-agent: facebookexternalhit Allow: / User-agent: meta-externalagent Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: Amazonbot Allow: / User-agent: Amazon-Route53-Health-Check-Service Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: ByteDance Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / User-agent: cohere-training-data-crawler Allow: / User-agent: DeepSeekBot Allow: / User-agent: DuckAssistBot Allow: / User-agent: Diffbot Allow: / User-agent: AI2Bot Allow: / User-agent: Ai2Bot-Dolma Allow: / User-agent: Brightbot Allow: / User-agent: omgili Allow: / User-agent: omgilibot Allow: / User-agent: webzio Allow: / User-agent: Webzio-Extended Allow: / User-agent: YouBot Allow: / User-agent: YouAgent Allow: / User-agent: Scrapy Allow: / User-agent: img2dataset Allow: / User-agent: iaskspider Allow: / User-agent: ICC-Crawler Allow: / User-agent: ISSCyberRiskCrawler Allow: / User-agent: Kangaroo Bot Allow: / User-agent: Meltwater Allow: / User-agent: NewsNow Allow: / User-agent: news-please Allow: / User-agent: PanguBot Allow: / User-agent: PetalBot Allow: / User-agent: Pinterestbot Allow: / User-agent: Seekr Allow: / User-agent: SemrushBot Allow: / User-agent: Sidetrade indexer bot Allow: / User-agent: Timpibot Allow: / User-agent: VelenPublicWebCrawler Allow: / User-agent: sentibot Allow: / User-agent: AcademicBotRTU Allow: / User-agent: Cloudflare-AutoRAG Allow: / User-agent: CloudVertexBot Allow: / User-agent: Cotoyogi Allow: / User-agent: Crawl4AI Allow: / User-agent: Crawlspace Allow: / User-agent: Crawlspace Allow: / User-agent: Devin Allow: / User-agent: Echobot Bot Allow: / User-agent: EchoboxBot Allow: / User-agent: Factset_spyderbot Allow: / User-agent: FirecrawlAgent Allow: / User-agent: FriendlyCrawler Allow: / User-agent: AwarioBot Allow: / User-agent: AwarioRssBot Allow: / User-agent: AwarioSmartBot Allow: / User-agent: DataForSeoBot Allow: / User-agent: magpie-crawler Allow: / User-agent: Nicecrawler Allow: / User-agent: peer39_crawler Allow: / User-agent: peer39_crawler/1.0 Allow: / User-agent: Teoma Allow: / User-agent: BLEXBot Allow: / User-agent: Linguee Bot Allow: / User-agent: MJ12bot Allow: / User-agent: Sogou Allow: / User-agent: sogou spider Allow: / User-agent: Yandex Allow: / User-agent: Yandex-AI Allow: / User-agent: LinkedInBot Allow: / User-agent: HubSpot Allow: / User-agent: HubSpot Crawler Allow: / # ============================================================================ # ORIGINAL ROBOTS.TXT (PRESERVED) # ============================================================================ # (No original robots.txt content was fetched.) # ============================================================================ # SITEMAPS # ============================================================================ Sitemap: https://ai.soundproofwindows.com/sitemap.xml