start notes/proposal about 'crawlability' improvements

1 files changed, 68 insertions, 0 deletions

diff --git a/proposals/2021-04-02_crawlability.md b/proposals/2021-04-02_crawlability.md
new file mode 100644
index 00000000..6b9ef66c
--- /dev/null
+++ b/proposals/2021-04-02_crawlability.md
@@ -0,0 +1,68 @@
+
+status: wip
+
+Crawlability Improvements
+--------------------------
+
+We are interested in making the fatcat corpus more crawlable/indexable by
+aggregators and academic search enginges. For example, CiteseerX, Google
+Scholar, or Microsoft Academic (when themselves get used by other projects).
+
+Some open questions:
+
+- is the web.archive.org iframe for PDFs ok, or should we redirect to PDFs with `id_` in the datetime?
+
+
+## Redirect URLs and `citation_pdf_url`
+
+We suspect that some crawlers do not like that fatcat.wiki landing pages have
+`citation_pdf_url` fields that point to a different registered domain.
+`www.semanticscholar.org -> pdfs.semanticscholar.org` is presumably ok, but
+maybe `fatcat.wiki -> web.archive.org` and `fatcat.wiki -> archive.org` are
+not.
+
+Google Scholar docs also request the PDF link be "in the same subdirectory"
+(though this obviously isn't true on, eg, semanticscholar.org):
+
+> If this page shows only the abstract of the paper and you have the full text
+> in a separate file, e.g., in the PDF format, please specify the locations of
+> all full text versions using citation_pdf_url or DC.identifier tags. The
+> content of the tag is the absolute URL of the PDF file; for security reasons,
+> it must refer to a file in the same subdirectory as the HTML abstract.
+
+Also suspect that a redirect is probably find. If a journal links from a
+landing page to a `.pdf` URL on the same domain, often there is an HTTP
+redirect to, eg, amazon AWS. These seem to get indexed fine.
+
+So, potentially have `citation_pdf_url` point to something on `fatcat.wiki`,
+which then redirects to `web.archive.org` or `archive.org`, would be
+sufficient. This would also be a reasonable URL for external services to point
+to, in that which specific access mechanism is redirected would vary as the
+catalog is improved.
+
+So, proposing two new web fatcat.wiki endpoints:
+
+    /release/<ident>/access-redirect
+    /file/<ident>/access-redirect
+
+Both of these would use an HTTP 302 "temporary" redirect to the "best" archival
+fulltext copy.
+
+If somebody wants to link to a specific file (by hash), they should use the
+file link. If they want to link to any fulltext access copy, then should use
+the release link.
+
+Open questions:
+
+- should the redirect only ever go to archive.org properties?
+- for releases, should the file type and access type be filtered? maybe with a
+  query parameter, or a `.pdf` suffix?
+
+
+## "Browsable" Site
+
+Another improvement would be to make the site more "browsable". To start, an
+index of journals (by first letter, publisher, country, or similar), then
+organize papers under the journal by volume, year, etc. This would give
+crawlers a way to spider all papers in the index.
+