summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools
Commit message (Collapse)AuthorAgeFilesLines
* factor out some basic kafka helpersBryan Newbold2019-12-102-0/+23
|
* add another ingest request source to whitelistBryan Newbold2019-12-101-2/+5
|
* refactor kafka producer in crossref harvesterBryan Newbold2019-12-061-21/+26
| | | | | | | | producer creation/configuration should be happening in __init__() time, not 'daily' call. This specific refactor motivated by mocking out the producer in unit tests.
* tweaks to file ingest importerBryan Newbold2019-12-031-3/+4
| | | | | - allow overriding source filter whitelist (common case for CLI use) - fix editgroup description env variable pass-through
* crossref is_update isn't what I thoughtBryan Newbold2019-12-031-6/+2
| | | | | | | | I thought this would filter for metadata updates to an existing DOI, but actually "updates" are a type of DOI (eg, a retraction). TODO: handle 'updates' field. Should both do a lookup and set work_ident appropriately, and store in crossref-specific metadata.
* re-order ingest want() for better statsBryan Newbold2019-11-151-7/+10
|
* project -> ingest_request_sourceBryan Newbold2019-11-153-9/+9
|
* fix release.pmcid typoBryan Newbold2019-11-151-2/+2
|
* ingest importer fixesBryan Newbold2019-11-151-3/+4
|
* more ingest importer comments and countsBryan Newbold2019-11-152-2/+29
|
* crude support for 'sandcrawler' kafka topicsBryan Newbold2019-11-151-2/+3
|
* ingest file result importerBryan Newbold2019-11-152-2/+135
|
* add ingest request feature to entity_updates workerBryan Newbold2019-11-151-4/+20
| | | | | | | | | | | | | Initially was going to create a new worker to consume from the release update channel, but couldn't get the edit context ("is this a new release, or update to an existing") from that context. Currently there is a flag in source code to control whether we only do OA releases or all releases. Starting with OA only to start slow, but should probably default to all, and make this a config flag. Should probably also have a config flag to control this entire feature. Tested locally in dev.
* add ingest request transform (and test)Bryan Newbold2019-11-152-0/+67
|
* crossref: accurate blank title countsBryan Newbold2019-11-051-0/+1
|
* crossref: component typeBryan Newbold2019-11-041-1/+3
|
* crossref: count why skip happenedBryan Newbold2019-11-041-1/+7
| | | | | | Might skip based on release type (eg container, not a paper/release), or missing title, or other reasons. Over 7 million DOIs are getting skipped, curious why.
* crossref: don't skip on short/null subtitleBryan Newbold2019-11-041-1/+1
| | | | This was a bug. Should only set subtitle black, not skip the import.
* file cleanup tweaks to actually runBryan Newbold2019-10-082-5/+4
|
* refactor duplicated b32_hex function in importersBryan Newbold2019-10-083-21/+11
|
* dict wrapper for entity_from_json()Bryan Newbold2019-10-082-3/+7
|
* new cleanup python tool/frameworkBryan Newbold2019-10-084-0/+241
|
* review/fix all confluent-kafka produce codeBryan Newbold2019-09-206-27/+75
|
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-206-24/+67
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* convert pipeline workers from pykafka to confluent-kafkaBryan Newbold2019-09-203-125/+230
|
* small kafka tweaks for robustnessBryan Newbold2019-09-202-0/+5
|
* convert importers to confluent-kafka libraryBryan Newbold2019-09-201-19/+71
|
* bump max message size to ~20 MBytesBryan Newbold2019-09-202-0/+2
|
* fixes to confluent-kafka harvestersBryan Newbold2019-09-203-20/+21
|
* first draft harvesters using confluent-kafkaBryan Newbold2019-09-203-48/+104
|
* handle more external identifiers in pythonBryan Newbold2019-09-181-14/+97
| | | | | This makes it possible to, eg, past an arxiv identifier or SHA-1 hash in the general search box and do a quick lookup.
* refactor all python source for client lib nameBryan Newbold2019-09-0521-121/+121
|
* fix Importer editgroup_extra pass-throughBryan Newbold2019-09-051-2/+1
|
* comment clarifying container.ident in ES release transformBryan Newbold2019-09-031-0/+2
|
* file rel: social -> academicsocialBryan Newbold2019-09-031-2/+2
|
* fix previous fix (need tests)Bryan Newbold2019-09-031-2/+2
|
* fix typo bug in container ES transformBryan Newbold2019-09-031-2/+2
|
* last chocula import behavior tweaksBryan Newbold2019-09-031-3/+21
|
* more careful chocula import counts; don't re-update empty URLsBryan Newbold2019-09-031-2/+6
|
* better importer 'total' countingBryan Newbold2019-09-031-4/+2
|
* chocula importer: include DOAJ updatesBryan Newbold2019-09-031-2/+2
|
* use EZB and szczepanski as OA signals (ES)Bryan Newbold2019-09-031-0/+12
|
* improvements to chocula importerBryan Newbold2019-09-031-1/+7
|
* implement ChoculaImporterBryan Newbold2019-09-032-0/+137
|
* improvements to wayback_static importerBryan Newbold2019-08-221-6/+29
|
* start new ES container worker kafka groupBryan Newbold2019-07-311-0/+2
| | | | | | | | The previous group seems to have gotten corrupted; my hypothesis is that this is due to pykafka being somewhat flakey, and am planning to move to librdkafka anyways. Re-indexing all the containers is pretty small/easy, so starting a new consumer group works find in this case; release indexer would be a bigger problem.
* crossref: allow 'name' fallback (for groups, etc)Bryan Newbold2019-06-241-1/+1
|
* add inflight edit protection to matched importerBryan Newbold2019-06-241-1/+8
|
* fix typo; do arxiv-specific match import hackBryan Newbold2019-06-241-3/+14
|
* fix syntax in existing.url cleanupBryan Newbold2019-06-241-1/+1
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-24 21:24:20 +0000