aboutsummaryrefslogtreecommitdiffstats
path: root/fatcat_scholar/static/robots.allow.txt
Commit message (Expand)AuthorAgeFilesLines
* robots: more blocks to reduce load on search clusterBryan Newbold2022-10-311-2/+8
* robots: block yacybot from search pageBryan Newbold2022-10-301-0/+1
* robots: deny SemanticScholarBot search result pagesBryan Newbold2022-06-061-0/+1
* robots.txt: block /search for large crawlersBryan Newbold2022-01-111-5/+13
* robots: add commented out reference to sitemap-index-access.xmlBryan Newbold2021-06-301-0/+3
* web: use absolute URLs to sitemaps and in citation_pdf_urlBryan Newbold2021-04-301-2/+2
* web: sitemap.xml servingBryan Newbold2021-04-291-2/+14
* web robots.txtBryan Newbold2020-09-151-0/+5