aboutsummaryrefslogtreecommitdiffstats
path: root/fatcat_scholar/transform.py
Commit message (Collapse)AuthorAgeFilesLines
* schema: add 'crossref' to bundle schema, and add from_json() helperBryan Newbold2021-06-021-26/+4
| | | | | from_json() refactor was an earlier TODO, to reduce duplication when updating fields on this class
* reduce max body size to 0.5M charactersBryan Newbold2021-02-241-1/+1
|
* fix body size limitBryan Newbold2021-02-241-4/+4
|
* fmt and lint fixes (including one actual bug)Bryan Newbold2021-02-151-2/+3
|
* truncate indexed fulltext body at 1 MByteBryan Newbold2021-02-151-2/+13
| | | | | | There was a large ~4 MByte document getting indexed (work_lumgqw4vqbgvha2ejbsbaepedq) with several megabytes of text, and this was causing elasticsearch indexing timeouts.
* catch TEI-XML parsing exceptionBryan Newbold2021-01-301-12/+17
|
* enable sentry exceptions for workers and pipelinesBryan Newbold2021-01-301-1/+12
| | | | It is otherwise difficult to debug multi-million record pipelines.
* bigfix: try resolving lang_code list issue againBryan Newbold2021-01-301-5/+4
|
* bugfix: lang_code sometimes a listBryan Newbold2021-01-291-2/+7
|
* make fmtBryan Newbold2021-01-251-1/+4
|
* basic support for excluding web content from indexBryan Newbold2021-01-221-6/+45
| | | | Based on particular patterns in metadata, or exclusion lists in settings
* bug fix: more html_fulltext not getting processedBryan Newbold2021-01-221-0/+2
|
* add container_sherpa_color field, and populate itBryan Newbold2021-01-221-0/+1
|
* improve 'oa' tag calculationBryan Newbold2021-01-161-4/+4
|
* small corrections to schema/transformBryan Newbold2021-01-161-2/+4
|
* add support for new identifiers and size_bytes schema fieldsBryan Newbold2021-01-141-0/+3
|
* basic HTML transform/index supportBryan Newbold2020-11-181-2/+46
|
* refs: extract fatcat crossref pages metadataBryan Newbold2020-11-131-1/+1
|
* commands: show usage on empty commandBryan Newbold2020-11-021-1/+1
|
* more SIM metadata mappingsBryan Newbold2020-10-191-3/+31
|
* SIM pipeline: more language conversionsBryan Newbold2020-10-161-2/+5
| | | | | Not sure where these language strings are coming from, but these were from existing SIM item metadata in archive.org
* transform: refactor tag generation out of transform heavy methodBryan Newbold2020-10-161-28/+37
|
* Upgrade Dynaconf to 3+Bruno Rocha2020-10-051-1/+1
| | | | | | In dynaconf 3+ it is no more recommended to use `from dynaconf import settings` now the recommendation is to create your own instance of the settings object based on Dynaconf class.
* refs and grobid2json bugfixes from testingBryan Newbold2020-09-141-3/+10
|
* bugfix: release_yearBryan Newbold2020-09-131-2/+2
|
* refs transform: both GROBID and fatcat refsBryan Newbold2020-09-131-5/+12
|
* ref transform: support more GROBID fieldsBryan Newbold2020-09-131-10/+16
|
* fixes to refs transform (for non-str author fields)Bryan Newbold2020-09-041-2/+6
|
* heavy to refs commandBryan Newbold2020-09-041-2/+142
|
* use simple names, not domain names, for some platformsBryan Newbold2020-08-121-3/+3
|
* biblio metadata hacks at transform timeBryan Newbold2020-08-121-2/+98
|
* don't index sim_page without issue_item and first_pageBryan Newbold2020-08-061-0/+3
|
* handle integer conversion and bounding for ES schemaBryan Newbold2020-08-061-10/+13
|
* json: exclude None in output, and sort keysBryan Newbold2020-07-271-1/+1
| | | | | | | | | | These are both size/performance enhancements. Not including 'None' values will reduce document sizes on-disk and over network, particularly for intermediate objects. Sorting by key should improve compression ratios across multiple documents, both on-disk (gzip) and in elasticsearch itself: https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html#_put_fields_in_the_same_order_in_documents
* ensure SIM release date parses before assigningBryan Newbold2020-07-211-1/+6
|
* make fmtBryan Newbold2020-06-291-8/+13
|
* include GROBID-extracted abstracts in search documentsBryan Newbold2020-06-291-10/+15
|
* small improvements to SIM metadata mapsBryan Newbold2020-06-291-6/+11
|
* fixes for pdf_meta dictBryan Newbold2020-06-291-1/+2
|
* remove old COVID19 thumbnail hackBryan Newbold2020-06-291-1/+2
|
* fetch pdftotext and pdf_meta from blobs, postgrestBryan Newbold2020-06-291-21/+13
| | | | | This replaces the temporary COVID-19 content hack with production content (text, thumbnail URLs) stored in postgrest and seaweedfs.
* collapse pages by SIM issueBryan Newbold2020-06-041-0/+3
|
* flake8-annotation lintingBryan Newbold2020-06-031-3/+3
| | | | Added some new annotations; need to finish more.
* flake8 fixes (partial)Bryan Newbold2020-06-031-11/+2
|
* reformat python code with blackBryan Newbold2020-06-031-109/+158
|
* fixes from running pipelineBryan Newbold2020-06-031-1/+2
| | | | Not caught by mypi/lint? Hrm.
* compute and use tagsBryan Newbold2020-06-031-0/+41
|
* fixes from manual testingBryan Newbold2020-05-201-5/+4
|
* fixes to release+sim pipelineBryan Newbold2020-05-201-1/+2
|
* indexing tweaksBryan Newbold2020-05-201-3/+4
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-27 02:53:26 +0000