aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* improve is_oa flag accuracyBryan Newbold2020-02-262-10/+6
| | | | | | Particularly, the ezb=green match seems mostly incorrect. Note that pmcid being assigned could still be in an embargo window?
* update ES transform READMEBryan Newbold2020-02-261-2/+3
| | | | | - smaller batch sizes to prevent esbulk errors - file transform/index
* fix fatcat_transform state filtersBryan Newbold2020-02-261-4/+4
|
* bulk ES transform: skip non-active entitiesBryan Newbold2020-02-261-0/+8
|
* ES container last tweaksBryan Newbold2020-02-262-3/+7
|
* ES release: last minor tweaksBryan Newbold2020-02-262-5/+7
|
* ES updates: fix tests to accept archive.org in host/domainBryan Newbold2020-02-141-2/+3
|
* release schema: do doc_value on DOIsBryan Newbold2020-02-131-1/+1
| | | | | | Because DOIs are pseudo-structured (prefix, and often structure within the publisher-controlled area), I suspect we will in fact be wanting to do analytics over these strings.
* ES files: don't remove archive.org domains/hostsBryan Newbold2020-02-071-5/+0
|
* ES release: actually do want doc_values for work_idBryan Newbold2020-02-051-1/+1
| | | | Eg, for fast "unique count"
* fix axiv/arxiv typo in release schemaBryan Newbold2020-02-041-1/+1
|
* ES release schema: fix typoBryan Newbold2020-01-311-1/+1
|
* ES releases: host/domain fixesBryan Newbold2020-01-312-2/+5
|
* pipenv: lock zipp version to work around python3.6 requirementBryan Newbold2020-01-302-7/+20
|
* fix release es transform missing 'issue'Bryan Newbold2020-01-301-0/+1
|
* fix json typos in changelog schemaBryan Newbold2020-01-301-2/+2
|
* add upper-case work-around from kibana map joinBryan Newbold2020-01-302-0/+2
|
* JSON typo in release mappingBryan Newbold2020-01-301-1/+0
|
* ES schemas: make keywords case-insensitive by defaultBryan Newbold2020-01-304-66/+115
| | | | But not applying asciifolding; don't see any need to do so?
* tweak file ES archive.org domain trackingBryan Newbold2020-01-302-0/+7
|
* implement host+domain parsing for file ES transformBryan Newbold2020-01-302-13/+8
|
* pipenv: add tldextract (url parser) and update depsBryan Newbold2020-01-302-136/+159
|
* fix ES file schema plural field namesBryan Newbold2020-01-292-5/+4
|
* new biblio-only general searchBryan Newbold2020-01-291-2/+2
| | | | The other fields are now "copy_to" the merged biblio field.
* elastic schema fixesBryan Newbold2020-01-293-7/+12
|
* add country to v03b release schemaBryan Newbold2020-01-292-0/+3
|
* update ES docs and proposalBryan Newbold2020-01-292-4/+6
|
* actually implement changelog transformBryan Newbold2020-01-293-19/+78
|
* fix some transform bugs, add some testsBryan Newbold2020-01-296-13/+48
|
* ES release schema updatesBryan Newbold2020-01-292-28/+122
|
* container ES schema changesBryan Newbold2020-01-292-29/+38
|
* first implementation of ES file schemaBryan Newbold2020-01-294-3/+115
| | | | | Includes a trivial test and transform, but not any workers or doc updates.
* fix KafkaError worker reporting for partition errorsBryan Newbold2020-01-293-3/+3
|
* additional DOI prefix filtersBryan Newbold2020-01-281-0/+8
| | | | From martin, thanks.
* increase kafka-pixy timeout to 25 secondsBryan Newbold2020-01-281-1/+1
|
* apply ingest request filtering in entity workerBryan Newbold2020-01-281-3/+34
| | | | | | | `ingest_oa_only` behavior, and other filters, now handled in the entity update worker, instead of in the transform function. Also add a DOI prefix blocklist feature.
* remove 'oa_only' feature from ingest transformBryan Newbold2020-01-282-15/+1
| | | | Refactoring to move this filter elsewhere
* more TODO/proposal cleanupBryan Newbold2020-01-224-10/+34
|
* more details on potential _edit table disk savingsBryan Newbold2020-01-221-3/+23
|
* proposal of ideas for reducing database sizeBryan Newbold2020-01-211-0/+154
|
* cleanup some of old TODO list into proposalsBryan Newbold2020-01-214-44/+269
|
* refactor fatcat_import kafka group namesBryan Newbold2020-01-211-13/+54
| | | | | | | | | | | | | My current understanding is that consumer group names should be one-to-one with topic names. I previously though offsets were stored on a {topic, group} key, but they seem to be mixed and having too many workers in the same group is bad. In particular, we don't want cross-talk or load between QA and prod. All these topics are caught up in prod, so deploying this change and restarting workers should be safe. This commit does not update the elasticsearch or entity updates workers.
* fix trivial typo in file importerBryan Newbold2020-01-201-1/+1
|
* stats: remove internal PG table sizes from old dumpsBryan Newbold2020-01-192-292/+0
| | | | For ease of reading and comparison
* update stats and table sizesBryan Newbold2020-01-194-0/+96
|
* Merge branch 'martin-openapi-typo-exmaple' into 'master'bnewbold2020-01-191-1/+1
|\ | | | | | | | | fix a typo in openapi definition See merge request webgroup/fatcat!20
| * fix a typo in openapi definitionMartin Czygan2020-01-181-1/+1
| |
* | Merge branch 'martin-guide-typos-sentance' into 'master'bnewbold2020-01-191-2/+2
|\ \ | | | | | | | | | | | | fix two typos in editing guide See merge request webgroup/fatcat!21
| * | fix two typos in editing guideMartin Czygan2020-01-181-2/+2
| |/
* | basic notes in bulk edit changelogBryan Newbold2020-01-191-0/+7
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-22 08:38:02 +0000