5 commits

Author	SHA1	Message	Date
Robert Prehn	20dcf4f265	feat: Revise robots.txt again	2024-03-04 11:47:35 -06:00
Robert Prehn	d3cee29532	feat: Add more scrapers to robots.txt	2024-03-01 13:21:51 -06:00
Robert Prehn	b123de5e1e	fix: Fuck ChatGPT	2024-01-31 10:01:08 -06:00
Robert Prehn	daa1abdca1	feat: Disable crawling for AI	2023-12-10 10:51:57 -06:00
Robert Prehn	94d34abc93	feat: Add sitemaps	2023-10-26 13:10:53 +00:00