summaryrefslogtreecommitdiffstats
path: root/fatcat_scholar/transform.py
Commit message (Collapse)AuthorAgeFilesLines
* json: exclude None in output, and sort keysBryan Newbold2020-07-271-1/+1
| | | | | | | | | | These are both size/performance enhancements. Not including 'None' values will reduce document sizes on-disk and over network, particularly for intermediate objects. Sorting by key should improve compression ratios across multiple documents, both on-disk (gzip) and in elasticsearch itself: https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html#_put_fields_in_the_same_order_in_documents
* ensure SIM release date parses before assigningBryan Newbold2020-07-211-1/+6
|
* make fmtBryan Newbold2020-06-291-8/+13
|
* include GROBID-extracted abstracts in search documentsBryan Newbold2020-06-291-10/+15
|
* small improvements to SIM metadata mapsBryan Newbold2020-06-291-6/+11
|
* fixes for pdf_meta dictBryan Newbold2020-06-291-1/+2
|
* remove old COVID19 thumbnail hackBryan Newbold2020-06-291-1/+2
|
* fetch pdftotext and pdf_meta from blobs, postgrestBryan Newbold2020-06-291-21/+13
| | | | | This replaces the temporary COVID-19 content hack with production content (text, thumbnail URLs) stored in postgrest and seaweedfs.
* collapse pages by SIM issueBryan Newbold2020-06-041-0/+3
|
* flake8-annotation lintingBryan Newbold2020-06-031-3/+3
| | | | Added some new annotations; need to finish more.
* flake8 fixes (partial)Bryan Newbold2020-06-031-11/+2
|
* reformat python code with blackBryan Newbold2020-06-031-109/+158
|
* fixes from running pipelineBryan Newbold2020-06-031-1/+2
| | | | Not caught by mypi/lint? Hrm.
* compute and use tagsBryan Newbold2020-06-031-0/+41
|
* fixes from manual testingBryan Newbold2020-05-201-5/+4
|
* fixes to release+sim pipelineBryan Newbold2020-05-201-1/+2
|
* indexing tweaksBryan Newbold2020-05-201-3/+4
|
* first pass transform from pipelines to ES schemaBryan Newbold2020-05-201-0/+306