summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools
Commit message (Collapse)AuthorAgeFilesLines
* crossref: count why skip happenedBryan Newbold2019-11-041-1/+7
| | | | | | Might skip based on release type (eg container, not a paper/release), or missing title, or other reasons. Over 7 million DOIs are getting skipped, curious why.
* crossref: don't skip on short/null subtitleBryan Newbold2019-11-041-1/+1
| | | | This was a bug. Should only set subtitle black, not skip the import.
* file cleanup tweaks to actually runBryan Newbold2019-10-082-5/+4
|
* refactor duplicated b32_hex function in importersBryan Newbold2019-10-083-21/+11
|
* dict wrapper for entity_from_json()Bryan Newbold2019-10-082-3/+7
|
* new cleanup python tool/frameworkBryan Newbold2019-10-084-0/+241
|
* review/fix all confluent-kafka produce codeBryan Newbold2019-09-206-27/+75
|
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-206-24/+67
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* convert pipeline workers from pykafka to confluent-kafkaBryan Newbold2019-09-203-125/+230
|
* small kafka tweaks for robustnessBryan Newbold2019-09-202-0/+5
|
* convert importers to confluent-kafka libraryBryan Newbold2019-09-201-19/+71
|
* bump max message size to ~20 MBytesBryan Newbold2019-09-202-0/+2
|
* fixes to confluent-kafka harvestersBryan Newbold2019-09-203-20/+21
|
* first draft harvesters using confluent-kafkaBryan Newbold2019-09-203-48/+104
|
* handle more external identifiers in pythonBryan Newbold2019-09-181-14/+97
| | | | | This makes it possible to, eg, past an arxiv identifier or SHA-1 hash in the general search box and do a quick lookup.
* refactor all python source for client lib nameBryan Newbold2019-09-0521-121/+121
|
* fix Importer editgroup_extra pass-throughBryan Newbold2019-09-051-2/+1
|
* comment clarifying container.ident in ES release transformBryan Newbold2019-09-031-0/+2
|
* file rel: social -> academicsocialBryan Newbold2019-09-031-2/+2
|
* fix previous fix (need tests)Bryan Newbold2019-09-031-2/+2
|
* fix typo bug in container ES transformBryan Newbold2019-09-031-2/+2
|
* last chocula import behavior tweaksBryan Newbold2019-09-031-3/+21
|
* more careful chocula import counts; don't re-update empty URLsBryan Newbold2019-09-031-2/+6
|
* better importer 'total' countingBryan Newbold2019-09-031-4/+2
|
* chocula importer: include DOAJ updatesBryan Newbold2019-09-031-2/+2
|
* use EZB and szczepanski as OA signals (ES)Bryan Newbold2019-09-031-0/+12
|
* improvements to chocula importerBryan Newbold2019-09-031-1/+7
|
* implement ChoculaImporterBryan Newbold2019-09-032-0/+137
|
* improvements to wayback_static importerBryan Newbold2019-08-221-6/+29
|
* start new ES container worker kafka groupBryan Newbold2019-07-311-0/+2
| | | | | | | | The previous group seems to have gotten corrupted; my hypothesis is that this is due to pykafka being somewhat flakey, and am planning to move to librdkafka anyways. Re-indexing all the containers is pretty small/easy, so starting a new consumer group works find in this case; release indexer would be a bigger problem.
* crossref: allow 'name' fallback (for groups, etc)Bryan Newbold2019-06-241-1/+1
|
* add inflight edit protection to matched importerBryan Newbold2019-06-241-1/+8
|
* fix typo; do arxiv-specific match import hackBryan Newbold2019-06-241-3/+14
|
* fix syntax in existing.url cleanupBryan Newbold2019-06-241-1/+1
|
* fix existing updaterBryan Newbold2019-06-241-2/+3
|
* add minimal file URL cleanups to matched importerBryan Newbold2019-06-241-0/+8
|
* matched importer: urls, not urlBryan Newbold2019-06-241-1/+1
| | | | | | This matches the docs in the header. Previous matched imports were using 'cdx' objects with no 'dt' key, but this makes more sense. As far as I know the old 'url' code path was never actually used (or tested, derp).
* fix typo in typoBryan Newbold2019-06-241-1/+1
|
* fix typo in changelog workerBryan Newbold2019-06-241-1/+1
|
* support extids in matched importerBryan Newbold2019-06-201-3/+26
|
* more links on new homepageBryan Newbold2019-06-192-2/+2
| | | | | matching produce sizes. may want to tweak this config in the future for throughput.
* start work on 'generic' search boxBryan Newbold2019-06-131-0/+95
|
* more pubmed importer fixesBryan Newbold2019-06-031-6/+13
|
* JALC: handle empty publisher stringBryan Newbold2019-05-301-3/+4
|
* journal_metadata import: allow KBART updatesBryan Newbold2019-05-301-5/+16
|
* fix and workaround container entities in release topicBryan Newbold2019-05-302-2/+8
|
* fix syntax bugs (container elastic worker)Bryan Newbold2019-05-301-5/+5
|
* add container update elastic workerBryan Newbold2019-05-302-6/+26
|
* file and container update kafka topicsBryan Newbold2019-05-301-54/+69
|
* update elastic for releases when files addedBryan Newbold2019-05-301-1/+36
| | | | A bunch of remaining TODOs here
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-24 05:38:07 +0000