deadvey
3b47701c18
and the footer was a footer type with no id, so now they are both both. Also added some margins at the top and bottom of the page in CSS
100 lines
2.4 KiB
Plaintext
100 lines
2.4 KiB
Plaintext
User-agent: AddSearchBot
|
|
User-agent: AI2Bot
|
|
User-agent: Ai2Bot-Dolma
|
|
User-agent: aiHitBot
|
|
User-agent: Amazonbot
|
|
User-agent: Andibot
|
|
User-agent: anthropic-ai
|
|
User-agent: Applebot
|
|
User-agent: Applebot-Extended
|
|
User-agent: Awario
|
|
User-agent: bedrockbot
|
|
User-agent: bigsur.ai
|
|
User-agent: Brightbot 1.0
|
|
User-agent: Bytespider
|
|
User-agent: CCBot
|
|
User-agent: ChatGPT Agent
|
|
User-agent: ChatGPT-User
|
|
User-agent: Claude-SearchBot
|
|
User-agent: Claude-User
|
|
User-agent: Claude-Web
|
|
User-agent: ClaudeBot
|
|
User-agent: CloudVertexBot
|
|
User-agent: cohere-ai
|
|
User-agent: cohere-training-data-crawler
|
|
User-agent: Cotoyogi
|
|
User-agent: Crawlspace
|
|
User-agent: Datenbank Crawler
|
|
User-agent: DeepSeekBot
|
|
User-agent: Devin
|
|
User-agent: Diffbot
|
|
User-agent: DuckAssistBot
|
|
User-agent: Echobot Bot
|
|
User-agent: EchoboxBot
|
|
User-agent: FacebookBot
|
|
User-agent: facebookexternalhit
|
|
User-agent: Factset_spyderbot
|
|
User-agent: FirecrawlAgent
|
|
User-agent: FriendlyCrawler
|
|
User-agent: Gemini-Deep-Research
|
|
User-agent: Google-CloudVertexBot
|
|
User-agent: Google-Extended
|
|
User-agent: Google-Firebase
|
|
User-agent: GoogleAgent-Mariner
|
|
User-agent: GoogleOther
|
|
User-agent: GoogleOther-Image
|
|
User-agent: GoogleOther-Video
|
|
User-agent: GPTBot
|
|
User-agent: iaskspider/2.0
|
|
User-agent: ICC-Crawler
|
|
User-agent: ImagesiftBot
|
|
User-agent: img2dataset
|
|
User-agent: ISSCyberRiskCrawler
|
|
User-agent: Kangaroo Bot
|
|
User-agent: LinerBot
|
|
User-agent: meta-externalagent
|
|
User-agent: Meta-ExternalAgent
|
|
User-agent: meta-externalfetcher
|
|
User-agent: Meta-ExternalFetcher
|
|
User-agent: meta-webindexer
|
|
User-agent: MistralAI-User
|
|
User-agent: MistralAI-User/1.0
|
|
User-agent: MyCentralAIScraperBot
|
|
User-agent: netEstate Imprint Crawler
|
|
User-agent: NovaAct
|
|
User-agent: OAI-SearchBot
|
|
User-agent: omgili
|
|
User-agent: omgilibot
|
|
User-agent: OpenAI
|
|
User-agent: Operator
|
|
User-agent: PanguBot
|
|
User-agent: Panscient
|
|
User-agent: panscient.com
|
|
User-agent: Perplexity-User
|
|
User-agent: PerplexityBot
|
|
User-agent: PetalBot
|
|
User-agent: PhindBot
|
|
User-agent: Poseidon Research Crawler
|
|
User-agent: QualifiedBot
|
|
User-agent: QuillBot
|
|
User-agent: quillbot.com
|
|
User-agent: SBIntuitionsBot
|
|
User-agent: Scrapy
|
|
User-agent: SemrushBot-OCOB
|
|
User-agent: SemrushBot-SWA
|
|
User-agent: ShapBot
|
|
User-agent: Sidetrade indexer bot
|
|
User-agent: TerraCotta
|
|
User-agent: Thinkbot
|
|
User-agent: TikTokSpider
|
|
User-agent: Timpibot
|
|
User-agent: VelenPublicWebCrawler
|
|
User-agent: WARDBot
|
|
User-agent: Webzio-Extended
|
|
User-agent: wpbot
|
|
User-agent: YaK
|
|
User-agent: YandexAdditional
|
|
User-agent: YandexAdditionalBot
|
|
User-agent: YouBot
|
|
Disallow: /
|