status: not-implemented

Crawlability Improvements
--------------------------

NOTE: After some back and forth on this topic, we have decided for now to focus
on having scholar.archive.org indexed, not fatcat.wiki. This proposal document
document is being kept as documentation of that decision.


## Original Intro

We are interested in making the fatcat corpus more crawlable/indexable by
aggregators and academic search enginges. For example, CiteseerX, Google
Scholar, or Microsoft Academic (when themselves get used by other projects).

Some open questions:

- is the web.archive.org iframe for PDFs ok, or should we redirect to PDFs with `id_` in the datetime?



## Redirect URLs and `citation_pdf_url`

We suspect that some crawlers do not like that fatcat.wiki landing pages have
`citation_pdf_url` fields that point to a different registered domain.
`www.semanticscholar.org -> pdfs.semanticscholar.org` is presumably ok, but
maybe `fatcat.wiki -> web.archive.org` and `fatcat.wiki -> archive.org` are
not.

Google Scholar docs also request the PDF link be "in the same subdirectory"
(though this obviously isn't true on, eg, semanticscholar.org):

> If this page shows only the abstract of the paper and you have the full text
> in a separate file, e.g., in the PDF format, please specify the locations of
> all full text versions using citation_pdf_url or DC.identifier tags. The
> content of the tag is the absolute URL of the PDF file; for security reasons,
> it must refer to a file in the same subdirectory as the HTML abstract.

Also suspect that a redirect is probably find. If a journal links from a
landing page to a `.pdf` URL on the same domain, often there is an HTTP
redirect to, eg, amazon AWS. These seem to get indexed fine.

So, potentially have `citation_pdf_url` point to something on `fatcat.wiki`,
which then redirects to `web.archive.org` or `archive.org`, would be
sufficient. This would also be a reasonable URL for external services to point
to, in that which specific access mechanism is redirected would vary as the
catalog is improved.

So, proposing two new web fatcat.wiki endpoints:

    /release/<ident>/access-redirect
    /file/<ident>/access-redirect

Both of these would use an HTTP 302 "temporary" redirect to the "best" archival
fulltext copy.

If somebody wants to link to a specific file (by hash), they should use the
file link. If they want to link to any fulltext access copy, then should use
the release link.

Open questions:

- should the redirect only ever go to archive.org properties?
- for releases, should the file type and access type be filtered? maybe with a
  query parameter, or a `.pdf` suffix?


## "Browsable" Site

Another improvement would be to make the site more "browsable". To start, an
index of journals (by first letter, publisher, country, or similar), then
organize papers under the journal by volume, year, etc. This would give
crawlers a way to spider all papers in the index.