aboutsummaryrefslogtreecommitdiffstats
path: root/fatcat_scholar/static/robots.allow.txt
Commit message (Collapse)AuthorAgeFilesLines
* robots.txt: block /search for large crawlersBryan Newbold2022-01-111-5/+13
|
* robots: add commented out reference to sitemap-index-access.xmlBryan Newbold2021-06-301-0/+3
|
* web: use absolute URLs to sitemaps and in citation_pdf_urlBryan Newbold2021-04-301-2/+2
|
* web: sitemap.xml servingBryan Newbold2021-04-291-2/+14
|
* web robots.txtBryan Newbold2020-09-151-0/+5
Two variants depending on environment.