summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/harvest/doi_registrars.py
Commit message (Collapse)AuthorAgeFilesLines
* refactor kafka producer in crossref harvesterBryan Newbold2019-12-061-21/+26
| | | | | | | | producer creation/configuration should be happening in __init__() time, not 'daily' call. This specific refactor motivated by mocking out the producer in unit tests.
* crossref is_update isn't what I thoughtBryan Newbold2019-12-031-6/+2
| | | | | | | | I thought this would filter for metadata updates to an existing DOI, but actually "updates" are a type of DOI (eg, a retraction). TODO: handle 'updates' field. Should both do a lookup and set work_ident appropriately, and store in crossref-specific metadata.
* review/fix all confluent-kafka produce codeBryan Newbold2019-09-201-5/+8
|
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-201-1/+1
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* small kafka tweaks for robustnessBryan Newbold2019-09-201-0/+2
|
* bump max message size to ~20 MBytesBryan Newbold2019-09-201-0/+1
|
* fixes to confluent-kafka harvestersBryan Newbold2019-09-201-10/+11
|
* first draft harvesters using confluent-kafkaBryan Newbold2019-09-201-26/+45
|
* fix harvester session.get() paramsBryan Newbold2019-03-061-5/+8
|
* retry/backoff for Crossref harvesterBryan Newbold2019-03-061-2/+2
|
* bunch of lint/whitespace cleanupsBryan Newbold2019-02-221-2/+1
|
* check request status codes idiomaticallyBryan Newbold2018-12-291-2/+2
|
* clean up harvester comments/docsBryan Newbold2018-11-211-5/+1
|
* use isoformat() to format datesBryan Newbold2018-11-211-2/+2
| | | | This shouldn't change behavior; it's just more consistent.
* fix loop_sleep typoBryan Newbold2018-11-211-1/+1
|
* fix datacite DOI extractionBryan Newbold2018-11-211-1/+1
|
* initial OAI-PMH harvestersBryan Newbold2018-11-191-5/+8
|
* better DOI registrar harvestersBryan Newbold2018-11-191-48/+20
|
* bunch of pylint cleanupBryan Newbold2018-11-151-7/+12
|
* refactoring harvestersBryan Newbold2018-11-151-0/+209