diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2021-06-30 18:30:24 -0700 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2021-06-30 18:30:24 -0700 |
| commit | 4e2bfc1fc7720849125af221e9afee7d1e5197af (patch) | |
| tree | 4c00644007119e0ab493ba4d52ae6da35452b37a | |
| parent | 0d2d8ae4a020c6d34c65bbc5de8513f04b5d722a (diff) | |
| download | fatcat-scholar-4e2bfc1fc7720849125af221e9afee7d1e5197af.tar.gz fatcat-scholar-4e2bfc1fc7720849125af221e9afee7d1e5197af.zip | |
robots: add commented out reference to sitemap-index-access.xml
| -rw-r--r-- | fatcat_scholar/static/robots.allow.txt | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/fatcat_scholar/static/robots.allow.txt b/fatcat_scholar/static/robots.allow.txt index fb0c251..21a343e 100644 --- a/fatcat_scholar/static/robots.allow.txt +++ b/fatcat_scholar/static/robots.allow.txt @@ -15,3 +15,6 @@ Crawl-delay: 5 Sitemap: https://scholar.archive.org/sitemap.xml Sitemap: https://scholar.archive.org/sitemap-index-works.xml + +# same info as sitemap-index-works.xml plus following citation_pdf_url +#Sitemap: https://scholar.archive.org/sitemap-index-access.xml |