pre.hn/priv/static/robots.txt

User-agent: *
Allow: *
Sitemap: https://pre.hn/sitemap.xml.gz

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /
feat: Add sitemaps 2023-10-26 13:10:53 +00:00			`User-agent: *`
			`Allow: *`
			`Sitemap: https://pre.hn/sitemap.xml.gz`
feat: Disable crawling for AI 2023-12-10 16:51:57 +00:00
			`User-agent: Google-Extended`
			`Disallow: /`
fix: Fuck ChatGPT 2024-01-31 16:01:08 +00:00
			`User-agent: GPTBot`
			`Disallow: /`