1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
|
subject: work-in-progress
Persistent Landing Pages, Access URLs, Sitemaps
===============================================
This is a proposal to start hosting persistent landing pages (for "work"
entities) and access redirect URLs on the scholar.archive.org domain.
Additionally, to create bulk sitemaps linking to these resources.
The initial motivation for these features are to enable indexing, aggregation,
and linking to preservation content by large academic search engines (like
Google Scholar, Microsoft Academic, and lens.org), as well as simple linking
from platforms like Wikipedia. All of these may, for various reasons, prefer
linking directly to PDF files, and to have have links on a *.archive.org (as
opposed to fatcat.wiki).
# Work Landing Pages
New URL endpoint:
/work/<ident>
Work landing pages will summarize bibliographic metadata about the work, list
versions ("releases") and list access options. The landing pages will include
bibliographic metadata summarized in HTML tags for the "primary" release.
In the future, these pages could be host to additional features or sub-pages
(new endpoints), such as:
- citation graph lists or visualizations
- alternative reading interface
- content previews
## Design Notes
The `citation_pdf_url` metadata tag will only link to a PDF file hosted on a
*.archive.org domain (aka, archive.org files or web.archive.org web-archived
files), via the access redirect URLs mentioned below. At least initially, only
PDF files which correspond to the "primary" version of the work will be
included. Eg, if there is a published release, a file manifestation of that
release will be linked, not earlier pre-print or accepted manuscript versions.
This behavior may change at some point to include "green" access links from the
"work" landing page.
The `citation_pdf_url` tag should contain an absolute URL, not a relative URL.
Alternatively, we could have landing pages only for "releases" (versions), like
already exist on fatcat.wiki. This would make the decision about which files to
link to simpler.
However, to date scholar.archive.org as a product/service has taken the
approach of simplifying the fatcat.wiki data model in the interest of
usability, and will probably continue with that approach here.
Landing pages will be rendered from a simple, single "GET" request to the same
elasticsearch backend index; no new backing services (eg, api.fatcat.wiki) are
introduced.
# Access Redirect URLs
Some academic search engines require a `citation_pdf_url` link from the same
domain as the landing page, with an optional HTTP redirect.
New URL endpoint:
/access-redirect/<sha1>.pdf
Requests to such URLs will redirect (HTTP 302) to an *.archive.org access
location of the exact file (by sha1), if known.
It is likely that in the future `.xml` and `.epub` access redirect links would
be added in the same format. Unclear what the scheme would be for HTML content
(SHA-1 of the "primary" HTML document? or wayback timestamp and URL?).
This URL structure was chosen to reduce confusion that the file might be served
from scholar.archive.org itself ("redirect"); to indicate the filetype
expected; and to encode information about which resource is being linked to
("content addressible").
### Design Notes
An alternative would have been to scope the URL below the work itself, eg:
/work/<ident>/access-redirect/<sha1>.pdf
Such URLs would be quite long.
These redirects are implemented by querying the same scholar elasticsearch
backend index, querying for fulltext access with the matching file SHA-1, and
using the `access_url` returned.
# Sitemap
The sitemap setup will be copied from fatcat.wiki. The new resources (URLs)
will include:
/robots.txt - updated to include sitemap references
/sitemap.xml - basic generic list of pages (homepage, about, userguide)
/sitemap-index-works.xml - XML file pointing to many sub-sitemap files; includes lastmod metadata
/sitemap-works-YYYY-MM-DD-NNNNN.txt - series of timestamped "simple" sitemaps (URL list files)
Only works for which there is an appropriate fulltext access URL
The sitemap links from robots.txt should be absolute URLs, not relative URLs.
|
