rename LICENSE and TODO to have file extensions

1 files changed, 193 insertions, 0 deletions

diff --git a/TODO.md b/TODO.md
new file mode 100644
index 00000000..a1f44f52
--- /dev/null
+++ b/TODO.md
@@ -0,0 +1,193 @@
+
+## In Progress
+
+## Prod Metadata Checks
+
+- edit and editgroup metadata
+- longtail_oa flag getting set on GROBID imports
+- crossref citation not saving 'article-title' or 'unstructured', and 'author'
+  should be 'authors' (list)
+- crossref not saving 'language' (looks like iso code already)
+- grobid reference should be under extra (not nested): issue, volume, authors
+- uniqueness of:
+    sha1 - via SQL dump
+    doi - via SQL dump
+    issnl - via JSON dump
+    orcid - via JSON dump
+
+## Next Up
+
+- bootstrap_bots script should set -ex and output admin and webface tokens
+- regression test imports for missing orcid display and journal metadata name
+- serveral tweaks/fixes to webface (eg, container metadata schema changed)
+- container count "enrich"
+- changelog elastic stuff (is there even a fatcat-export for this?)
+- QA sentry has very little host info; also not URL of request
+- start prod crossref harvesting (from ~start of 2019)
+- 158 "NULL" publishers in journal metadata
+- should elastic release_year be of date type, instead of int?
+- QA/prod needs updated credentials
+- ansible: ISSN-L download/symlink
+- searching 'N/A' is a bug
+- formalize release_status:
+    => https://wiki.surfnet.nl/display/DRIVERguidelines/DRIVER-VERSION+Mappings
+
+## Production public launch blockers
+
+- handle 'wip' status entities in web UI
+- guide updates for auth
+- webface 4xx and 5xx pages
+- privacy policy, and link from: create account, create edit
+- refactors and correctness in rust/TODO
+- update /about page
+
+## Production Tech Sanity
+
+- postgresql replication
+- pg_dump/load test
+- haproxy somewhere/how
+- logging iteration: larger journald buffers? point somewhere?
+
+## Ideas
+
+- write up notes on biblio metadata in general
+    => "extensibility" and extra keys
+    => proliferation of arrays vs. concrete values
+    => various ways to record history/progeny
+    => "subtitle", "short-title", "full-title" complexity
+    => human names
+    => translated metadata: titles/names/abstracts
+    => "typing" for metadata (eg, math in titles)
+- 'hide' flag for exporter (eg, to skip abstracts and refs in some release dumps)
+- https://tech.labs.oliverwyman.com/blog/2019/01/14/serialising-rust-tests/
+- use https://github.com/codelucas/newspaper to extract fulltext+metadata from
+  HTML crawls
+- changelog elastic index (for stats)
+- import from arabesque output (eg, specific crawls)
+- more logins: orcid, wikimedia
+- `fatcat-auth` tool should support more caveats, both when generating new or
+  mutating existing tokens
+- fast path to skip recursive redirect checks for bulk inserts
+- when getting "wip" entities, require a parameter ("allow_wip"), else get a
+  404
+- consider dropping CORE identifier
+- maybe better 'success' return message? eg, "success: true" flag
+- idea: allow users to generate their own editgroup UUIDs, to reduce a round
+  trips and "hanging" editgroups (created but never edited)
+- API: allow deletion of empty, un-accepted editgroups
+- refactor API schema for some entity-generic methos (eg, history, edit
+  operations) to take entity type as a URL path param. greatly reduce macro
+  foolery and method count/complexity, and ease creation of new entities
+    => /{entity}/edit/{edit_id}
+    => /{entity}/{ident}/redirects
+    => /{entity}/{ident}/history
+- investigate data quality by looking at, eg, most popular author strings, most
+  popular titles, duplicated containers, etc
+
+## Metadata Import
+
+- web.archive.org response not SHA1 match? => need /<dt>id_/ thing
+- XML etc in metadata
+    => (python) tests for these!
+    https://qa.fatcat.wiki/release/search?q=xmlns
+    https://qa.fatcat.wiki/release/search?q=%24gt
+- bad/weird titles
+    "[Blank page]", "blank page"
+    "Temporary Empty DOI 0"
+    "ADVERTISEMENT"
+    "Full title page with Editorial board (with Elsevier tree)"
+    "Advisory Board Editorial Board"
+- better/complete reltypes probably good (eg, list of IRs, academic domain)
+- 'expand' in lookups (derp! for single hit lookups)
+- include crossref-capitalized DOI in extra
+- some "Elsevier " stuff as publisher
+    => also title https://fatcat.wiki/release/uyjzaq3xjnd6tcrqy3vcucczsi
+- crossref import: don't store citation unstructured if len() == 0:
+    {"crossref": {"unstructured": ""}}
+- try out beautifulsoup? (https://stackoverflow.com/a/34532382/4682349)
+- manifest: multiple URLs per SHA1
+- crossref: relations ("is-preprint-of")
+- crossref: two phase: no citations, then matched citations (via DOI table)
+- special "alias" DOIs... in crossref metadata?
+
+new importers:
+- pubmed (medline) (filtered)
+    => and/or, use pubmed ID lookups on crossref import
+- arxiv.org
+- DOAJ
+- CORE (filtered)
+- semantic scholar (up to 39 million; includes author de-dupe)
+
+## Guide / Book / Style
+
+- release_type, release_status, url.rel schemas (enforced in API)
+- more+better terms+policies: https://tosdr.org/index.html
+
+## Fun Features
+
+- "save paper now"
+    => is it in GWB? if not, SPN
+    => get hash + url from GWB, verify mimetype acceptable
+    => is file in fatcat?
+    => what about HBase? GROBID?
+    => create edit, redirect user to editgroup submit page
+- python client tool and library in pypi
+    => or maybe rust?
+- bibtext (etc) export
+
+## Metadata Harvesting
+
+- datacite ingest seems to have failed... got a non-HTTP-200 status code, but also "got 50 (161950 of 21084)"
+
+## Schema / Entity Fields
+
+- elastic transform should only include authors, not editors (?)
+- `doi` field for containers (at least for "journal" type; maybe for "series"
+  as well?)
+- `retracted`, `translation`, and perhaps `corrected` as flags on releases,
+  instead of release_status?
+    => use extra flags and release_status for now
+- 'part-of' relation for releases (release to release) and possibly containers
+- `container_type` field for containers (journal, conference, book series, etc)
+
+## Other / Backburner
+
+- refactor webface views to use shared entity_view.html template
+- shadow library manifest importer
+- book identifiers: OCLC, openlibrary
+- ref from guide: https://creativecommons.org/2012/08/14/library-catalog-metadata-open-licensing-or-public-domain/
+- test redirect/delete elasticsearch change
+- fake DOI (use in examples): 10.5555/12345678
+- refactor elasticsearch inserter to be a class (eg, for command line use)
+- document: elastic query date syntax is like: date:[2018-10-01 TO 2018-12-31]
+- fileset/webcapture webface anything
+- display abstracts better. no hashes or metadata; prefer plain or HTML,
+  convert JATS if necessary
+- switch from slog to simple pretty_env_log
+- format returned datetimes with only second precision, not millisecond (RFC mode)
+    => burried in model serialization internals
+- refactor openapi schema to use shared response types
+- consider using "HTTP 202: Accepted" for entity-mutating calls
+- basic python hbase/elastic matcher
+  => takes sha1 keys
+  => checks fatcat API + hbase
+  => if not matched yet, tries elastic search
+  => simple ~exact match heuristic
+  => proof-of-concept, no tests
+- add_header Strict-Transport-Security "max-age=3600";
+    => 12 hours? 24?
+- haproxy for rate-limiting
+
+better API docs
+- readme.io has a free open source plan (or at least used to)
+- https://github.com/readmeio/api-explorer
+- https://github.com/lord/slate
+- https://sourcey.com/spectacle/
+- https://github.com/DapperDox/dapperdox
+
+CSL:
+- https://citationstyles.org/
+- https://github.com/citation-style-language/documentation/blob/master/primer.txt
+- https://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html
+- https://github.com/citation-style-language/schema/blob/master/csl-types.rnc
+- perhaps a "create from CSL" endpoint?