diff options
Diffstat (limited to 'fatcat-rfc.md')
| -rw-r--r-- | fatcat-rfc.md | 363 |
1 files changed, 363 insertions, 0 deletions
diff --git a/fatcat-rfc.md b/fatcat-rfc.md new file mode 100644 index 00000000..21495f6d --- /dev/null +++ b/fatcat-rfc.md @@ -0,0 +1,363 @@ +fatcat is a half-baked idea to build an open, independent, collaboratively +editable bibliographic database of most written works, with a focus on +published research outputs like journal articles, pre-prints, and conference +proceedings. + +## Technical Architecture + +The canonical backend datastore would be a very large transactional SQL server. +A relatively simple and stable back-end daemon would expose an API (could be +REST, GraphQL, gRPC, etc). As little "application logic" as possible would be +embedded in this back-end; as much as possible would be pushed to bots which +could be authored and operated by anybody. A separate web interface project +would talk to the API backend and could be developed more rapidly. + +A cronjob would make periodic database dumps, both in "full" form (all tables +and all edit history, removing only authentication credentials) and "flat" form +(with only the most recent version of each entity, using only persistent IDs +between entities). + +A goal is to be linked-data/RDF/JSON-LD/semantic-web "compatible", but not +necessarily "first". It should be possible to export the database in a +relatively clean RDF form, and to fetch data in a variety of formats, but +internally fatcat would not be backed by a triple-store, and would not be +bound to a specific third-party ontology or schema. + +Microservice daemons should be able to proxy between the primary API and +standard protocols like ResourceSync and OAI-PMH, and bots could consume +external databases in those formats. + +## Licensing + +The core fatcat database should only contain verifiable factual statements +(which isn't to say that all statements are "true"), not creative or derived +content. + +The goal is to have a very permissively licensed database: CC-0 (no rights +reserved) if possible. Under US law, it should be possible to scrape and pull +in factual data from other corpuses without adopting their licenses. The goal +here isn't to avoid all attribution (progeny information will be included, and a +large sources and acknowledgments statement should be maintained), but trying +to manage the intersection of all upstream source licenses seems untenable, and +creates burdens for downstream users. + +Special care will need to be taken around copyright and original works. I would +propose either not accepting abstracts at all, or including them in a +partitioned database to prevent copyright contamination. Likewise, even simple +user-created content like lists, reviews, ratings, comments, discussion, +documentation, etc., should go in separate services. + +## Basic Editing Workflow and Bots + +Both human editors and bots would have edits go through the same API, with +humans using either the default web interface, arbitrary integrations, or +client software. + +The usual workflow would be to create edits (or creations, merges, deletions) +to individual entities one at a time, all under a single "edit group" of +related edits (eg, correcting authorship info for multiple works related to a +single author). When ready, the editor would "submit" the edit group for +review. During the review period, humans could vote (or veto/approve if they +have higher permissions), and bots can perform automated checks. During this +period the editor can make tweaks if necessary. After some fixed time period +(72 hours?) with no changes and no blocking issues, the edit group would be +auto-accepted, if no auto-resolvable merge-conflicts have arisen. This process +balances editing labor (reviews are easy, but optional) against quality +(cool-down period makes it easier to detect and prevent spam or out-of-control +bots). Advanced permissions could allow some trusted human and bot editors to +push through edits more rapidly. + +Bots would need to be tuned to have appropriate edit group sizes (eg, daily +batches, instead of millions of works in a single edit) to make human QA and +reverts possible. + +Data progeny and citation would be left to the edit history. In the case of +importing external databases, the expectation would be that special-purpose +bot accounts would be used. Human editors would leave edit messages to clarify +their sources. + +A style guide (wiki), chat room, and discussion forum would be hosted as +separate stand-alone services for editors to propose projects and debate +process or scope changes. It would be best if these could use federated account +authorization (oauth?) to have consistent account IDs across mediums. + +## Edit Log + +As part of the process of "accepting" an edit group, a row would be written to +an immutable, append-only log table (which internally could be a SQL table) +documenting each identifier change. This log establishes a monotonically +increasing version number for the entire corpus, and should make interaction +with other systems easier (eg, search engines, replicated databases, +alternative storage backends, notification frameworks, etc.). + +## Identifiers + +A fixed number of first-class "entities" would be defined, with common +behavior and schema layouts. These would all be semantic entities like "work", +"release", "container", and "person". + +fatcat identifiers would be semantically meaningless fixed-length random numbers, +usually represented in case-insensitive base32 format. Each entity type would +have its own identifier namespace. Eg, 96-bit identifiers would have 20 +characters and look like: + + fcwork_rzga5b9cd7efgh04iljk + https://fatcat.org/work/rzga5b9cd7efgh04iljk + +128-bit (UUID size) would have 26 characters: + + fcwork_rzga5b9cd7efgh04iljk8f3jvz + https://fatcat.org/work/rzga5b9cd7efgh04iljk8f3jvz + +A 64-bit namespace is probably plenty though, and would work with most database +Integer columns: + + fcwork_rzga5b9cd7efg + https://fatcat.org/work/rzga5b9cd7efg + +The idea would be to only have fatcat identifiers be used to interlink between +databases, *not* to supplant DOIs, ISBNs, handle, ARKs, and other "registered" +persistent identifiers. + +## Entities and Internal Schema + +Internally, identifiers would be lightweight pointers to actual metadata +objects, which can be thought of as "versions". The metadata objects themselves +would be immutable once committed; the edit process is one of creating new +objects and, if the edit is approved, pointing the identifier to the new +version. Entities would reference between themselves by identifier. + +Edit objects represent a change to a single entity; edits get batched together +into edit groups (like "commits" and "pull requests" in git parlance). + +SQL tables would probably look something like the following, though be specific +to each entity type (eg, there would be an actual `work_revision` table, but +not an actual `entity_revision` table): + + entity_id + uuid + current_revision + + entity_revision + entity_id (bi-directional?) + previous: entity_revision or none + state: normal, redirect, deletion + redirect_entity_id: optional + extra: json blob + edit_id + + edit + mutable: boolean + edit_group + editor + + edit_group + +Additional type-specific columns would hold actual metadata. Additional tables +(which would reference both `entity_revision` and `entity_id` foreign keys as +appropriate) would represent things like external identifiers, ordered +author/work relationships, citations between works, etc. Every revision of an +entity would require duplicating all of these associated rows, which could end +up being a large source of inefficiency, but is necessary to represent the full +history of an object. + +## Scope + +Want the "scholarly web": the graph of works that cite other works. Certainly +every work that is cited more than once and every work that both cites and is +cited; "leaf nodes" and small islands might not be in scope. + +Focusing on written works, with some exceptions. Expect core media (for which we would pursue "completeness") to be: + + journal articles + books + conference proceedings + technical memos + dissertations + +Probably in scope: + + reports + magazine articles + published poetry + essays + government documents + conference + presentations (slides, video) + datasets + +Probably not: + + patents + court cases and legal documents + manuals + datasheets + courses + +Definitely not: + + audio recordings + tv show episodes + musical scores + advertisements + +Author, citation, and work disambiguation would be core tasks. Linking +pre-prints to final publication is in scope. + +I'm much less interested in altmetrics, funding, and grant relationships than +most existing databases in this space. + +fatcat would not include any fulltext content itself, even for cleanly licensed +(open access) works, but would have "strong" (verified) links to fulltext +content, and would include file-level metadata (like hashes and fingerprints) +to help discovery and identify content from any source. Typed file-level links +should make fatcat more useful for both humans and machines to quickly access +fulltext content of a given mimetype than existing redirect or landing page +systems. + +## Ontology + +Loosely following FRBR, but removing the "manifestation" abstraction, and +favoring files (digital artifacts) over physical items, the primary entities +are: + + work + type + <has> contributors + <about> subject/category + <has-primary> release + + release (aka "edition", "variant") + title + volume/pages/issue/chapter + open-access status + <published> date + <of a> work + <published-by> publisher + <published in> container + <has> contributors + <citation> citetext <to> release + <has> identifier + + file (aka "digital artifact") + <of a> release + <has> hashes + <found at> URLs + <held-at> institution <with> accession + + contributor + name + <has> aliases + <has> affiliation <for> date span + <has> identifier + + container + name + open-access policy + peer-review policy + <has> aliases, acronyms + <about> subject/category + <has> identifier + <published in> container + <published-by> publisher + + publisher + name + <has> aliases, acronyms + <has> identifier + +## Controlled Vocabularies + +Some special namespace tables and enums would probably be helpful; these should +live in the database (not requiring a database migration to update), but should +have more controlled editing workflow... perhaps versioned in the codebase: + +- identifier namespaces (DOI, ISBN, ISSN, ORCID, etc) +- subject categorization +- license and open access status +- work "types" (article vs. book chapter vs. proceeding, etc) +- contributor types (author, translator, illustrator, etc) +- human languages +- file mimetypes + +## Unresolved Questions + +How to handle translations of, eg, titles and author names? To be clear, not +translations of works (which are just separate releases). + +Are bi-directional links a schema anti-pattern? Eg, should "work" point to a +primary "release" (which itself points back to the work), or should "release" +have a "is-primary" flag? + +Should `identifier` and `citation` be their own entities, referencing other +entities by UUID instead of by revision? This could save a ton of database +space and chunder. + +Should contributor/author contact information be retained? It could be very +useful for disambiguation, but we don't want to build a huge database for +spammers or "innovative" start-up marketing. + +Would general-purpose SQL databases like Postgres or MySQL scale well enough +to hold several tables with billions of entries? Right from the start there +are hundreds of millions of works and releases, many of which having dozens of +citations, many authors, and many identifiers, and then we'll have potentially +dozens of edits for each of these, which multiply out to `1e8 * 2e1 * 2e1 = +4e10`, or 40 billion rows in the citation table. If each row was 32 bytes on +average (uncompressed, not including index size), that would be 1.3 TByte on +its own, larger than common SSD disk. I think a transactional SQL datastore is +the right answer. In my experience locking and index rebuild times are usually +the biggest scaling challenges; the largely-immutable architecture here should +mitigate locking. Hopefully few indexes would be needed in the primary +database, as user interfaces could rely on secondary read-only search engines +for more complex queries and views. + +I see a tension between focus and scope creep. If a central database like +fatcat doesn't support enough fields and metadata, then it will not be possible +to completely import other corpuses, and this becomes "yet another" partial +bibliographic database. On the other hand, accepting arbitrary data leads to +other problems: sparseness increases (we have more "partial" data), potential +for redundancy is high, humans will start editing content that might be +bulk-replaced, etc. + +There might be a need to support "stub" references between entities. Eg, when +adding citations from PDF extraction, the cited works are likely to be +ambiguous. Could create "stub" works to be merged/resolved later, or could +leave the citation hanging. Same with authors, containers (journals), etc. + +## References and Previous Work + +The closest overall analog of fatcat is [MusicBrainz][mb], a collaboratively +edited music database. [Open Library][ol] is a very similar existing service, +which exclusively contains book metadata. + +[Wikidata][wd] seems to be the most successful and actively edited/developed +open bibliographic database at this time (early 2018), including the +[wikicite][wikicite] conference and related Wikimedia/Wikipedia projects. +Wikidata is a general purpose semantic database of entities, facts, and +relationships; bibliographic metadata has become a large fraction of all +content in recent years. The focus there seems to be linking knowledge +(statements) to specific sources unambiguously. Potential advantages fatcat +would have would be a focus on a specific scope (not a general-purpose database +of entities) and a goal of completeness (capturing as many works and +relationships as rapidly as possible). However, it might be better to just +pitch in to the wikidata efforts. + +The technical design of fatcat is loosely inspired by the git +branch/tag/commit/tree architecture, and specifically inspired by Oliver +Charles' "New Edit System" [blog posts][nes-blog] from 2012. + +There are a whole bunch of proprietary, for-profit bibliographic databases, +including Web of Science, Google Scholar, Microsoft Academic Graph, aminer, +Scopus, and Dimensions. There are excellent field-limited databases like dblp, +MEDLINE, and Semantic Scholar. There are some large general-purpose databases +that are not directly user-editable, including the OpenCitation corpus, CORE, +BASE, and CrossRef. I don't know of any large (more than 60 million works), +open (bulk-downloadable with permissive or no license), field agnostic, +user-editable corpus of scholarly publication bibliographic metadata. + +[nes-blog]: https://ocharles.org.uk/blog/posts/2012-07-10-nes-does-it-better-1.html +[mb]: https://musicbrainz.org +[ol]: https://openlibrary.org +[wd]: https://wikidata.org +[wikicite]: https://meta.wikimedia.org/wiki/WikiCite_2017 + |