summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/transforms/elasticsearch.py
Commit message (Collapse)AuthorAgeFilesLines
* ES schema: add best_url to file schemaBryan Newbold2020-06-041-0/+12
| | | | | | | | | This will increase index size (URLs are often long in our corpus, and we have many file entities), but seems worth it. Initially added `ia_url` as a second field, guaranteed to always be an *.archive.org URL, but `best_url` defaults to that anyways so didn't seem worthwhile.
* improve is_oa flag accuracyBryan Newbold2020-02-261-8/+4
| | | | | | Particularly, the ezb=green match seems mostly incorrect. Note that pmcid being assigned could still be in an embargo window?
* ES container last tweaksBryan Newbold2020-02-261-0/+3
|
* ES release: last minor tweaksBryan Newbold2020-02-261-2/+2
|
* ES files: don't remove archive.org domains/hostsBryan Newbold2020-02-071-5/+0
|
* ES releases: host/domain fixesBryan Newbold2020-01-311-2/+2
|
* fix release es transform missing 'issue'Bryan Newbold2020-01-301-0/+1
|
* add upper-case work-around from kibana map joinBryan Newbold2020-01-301-0/+1
|
* tweak file ES archive.org domain trackingBryan Newbold2020-01-301-0/+6
|
* implement host+domain parsing for file ES transformBryan Newbold2020-01-301-9/+5
|
* fix ES file schema plural field namesBryan Newbold2020-01-291-4/+3
|
* elastic schema fixesBryan Newbold2020-01-291-0/+5
|
* add country to v03b release schemaBryan Newbold2020-01-291-0/+2
|
* actually implement changelog transformBryan Newbold2020-01-291-17/+45
|
* fix some transform bugs, add some testsBryan Newbold2020-01-291-6/+8
|
* ES release schema updatesBryan Newbold2020-01-291-5/+76
|
* container ES schema changesBryan Newbold2020-01-291-16/+18
|
* first implementation of ES file schemaBryan Newbold2020-01-291-0/+45
| | | | | Includes a trivial test and transform, but not any workers or doc updates.
* refactor all python source for client lib nameBryan Newbold2019-09-051-1/+1
|
* comment clarifying container.ident in ES release transformBryan Newbold2019-09-031-0/+2
|
* fix previous fix (need tests)Bryan Newbold2019-09-031-2/+2
|
* fix typo bug in container ES transformBryan Newbold2019-09-031-2/+2
|
* use EZB and szczepanski as OA signals (ES)Bryan Newbold2019-09-031-0/+12
|
* elasticsearch transform: fix url.url bugBryan Newbold2019-05-241-11/+11
|
* add 'superceded' release extra flag to elastic schemaBryan Newbold2019-05-231-0/+1
|
* also track work_id in release elasticsearch tableBryan Newbold2019-05-221-0/+1
|
* count linked refs (not just raw refs) in elasticsearchBryan Newbold2019-05-221-0/+3
|
* include creator_ids in release elastic schemaBryan Newbold2019-05-201-0/+6
| | | | Intent is to allow fast creator search/lookup
* elastic release schema updateBryan Newbold2019-05-201-2/+5
|
* fix elastic file pdf checkBryan Newbold2019-05-161-1/+3
|
* elastic transforms: work around missing pdf mimetypesBryan Newbold2019-05-151-1/+1
|
* partial python impl of ext_id and release_stage refactorsBryan Newbold2019-05-131-10/+11
|
* handle null abstracts for releaseBryan Newbold2019-05-071-1/+1
|
* improve test coverageBryan Newbold2019-04-041-0/+1
|
* refactor transforms into sub-dirBryan Newbold2019-03-111-0/+327