prehnra/pre.hn
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pre.hn/priv/static/robots.txt

7 lines
103 B
Text
Raw Normal View History

feat: Add sitemaps
2023-10-26 13:10:53 +00:00
User-agent: *
Allow: *
Sitemap: https://pre.hn/sitemap.xml.gz
feat: Disable crawling for AI
2023-12-10 16:51:57 +00:00
User-agent: Google-Extended
Disallow: /
Reference in a new issue Copy permalink