aboutsummaryrefslogtreecommitdiffstats
path: root/fatcat_scholar/static/robots.allow.txt
Commit message (Expand)AuthorAgeFilesLines
* robots.txt: block /search for large crawlersBryan Newbold2022-01-111-5/+13
* robots: add commented out reference to sitemap-index-access.xmlBryan Newbold2021-06-301-0/+3
* web: use absolute URLs to sitemaps and in citation_pdf_urlBryan Newbold2021-04-301-2/+2
* web: sitemap.xml servingBryan Newbold2021-04-291-2/+14
* web robots.txtBryan Newbold2020-09-151-0/+5