Commit message (Collapse) | Author | Age | Files | Lines | ||
---|---|---|---|---|---|---|
... | ||||||
* | crossref: don't skip on short/null subtitle | Bryan Newbold | 2019-11-04 | 1 | -1/+1 | |
| | | | | This was a bug. Should only set subtitle black, not skip the import. | |||||
* | file cleanup tweaks to actually run | Bryan Newbold | 2019-10-08 | 2 | -5/+4 | |
| | ||||||
* | refactor duplicated b32_hex function in importers | Bryan Newbold | 2019-10-08 | 3 | -21/+11 | |
| | ||||||
* | dict wrapper for entity_from_json() | Bryan Newbold | 2019-10-08 | 2 | -3/+7 | |
| | ||||||
* | new cleanup python tool/framework | Bryan Newbold | 2019-10-08 | 4 | -0/+241 | |
| | ||||||
* | review/fix all confluent-kafka produce code | Bryan Newbold | 2019-09-20 | 6 | -27/+75 | |
| | ||||||
* | small fixes to confluent-kafka importers/workers | Bryan Newbold | 2019-09-20 | 6 | -24/+67 | |
| | | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again) | |||||
* | convert pipeline workers from pykafka to confluent-kafka | Bryan Newbold | 2019-09-20 | 3 | -125/+230 | |
| | ||||||
* | small kafka tweaks for robustness | Bryan Newbold | 2019-09-20 | 2 | -0/+5 | |
| | ||||||
* | convert importers to confluent-kafka library | Bryan Newbold | 2019-09-20 | 1 | -19/+71 | |
| | ||||||
* | bump max message size to ~20 MBytes | Bryan Newbold | 2019-09-20 | 2 | -0/+2 | |
| | ||||||
* | fixes to confluent-kafka harvesters | Bryan Newbold | 2019-09-20 | 3 | -20/+21 | |
| | ||||||
* | first draft harvesters using confluent-kafka | Bryan Newbold | 2019-09-20 | 3 | -48/+104 | |
| | ||||||
* | handle more external identifiers in python | Bryan Newbold | 2019-09-18 | 1 | -14/+97 | |
| | | | | | This makes it possible to, eg, past an arxiv identifier or SHA-1 hash in the general search box and do a quick lookup. | |||||
* | refactor all python source for client lib name | Bryan Newbold | 2019-09-05 | 21 | -121/+121 | |
| | ||||||
* | fix Importer editgroup_extra pass-through | Bryan Newbold | 2019-09-05 | 1 | -2/+1 | |
| | ||||||
* | comment clarifying container.ident in ES release transform | Bryan Newbold | 2019-09-03 | 1 | -0/+2 | |
| | ||||||
* | file rel: social -> academicsocial | Bryan Newbold | 2019-09-03 | 1 | -2/+2 | |
| | ||||||
* | fix previous fix (need tests) | Bryan Newbold | 2019-09-03 | 1 | -2/+2 | |
| | ||||||
* | fix typo bug in container ES transform | Bryan Newbold | 2019-09-03 | 1 | -2/+2 | |
| | ||||||
* | last chocula import behavior tweaks | Bryan Newbold | 2019-09-03 | 1 | -3/+21 | |
| | ||||||
* | more careful chocula import counts; don't re-update empty URLs | Bryan Newbold | 2019-09-03 | 1 | -2/+6 | |
| | ||||||
* | better importer 'total' counting | Bryan Newbold | 2019-09-03 | 1 | -4/+2 | |
| | ||||||
* | chocula importer: include DOAJ updates | Bryan Newbold | 2019-09-03 | 1 | -2/+2 | |
| | ||||||
* | use EZB and szczepanski as OA signals (ES) | Bryan Newbold | 2019-09-03 | 1 | -0/+12 | |
| | ||||||
* | improvements to chocula importer | Bryan Newbold | 2019-09-03 | 1 | -1/+7 | |
| | ||||||
* | implement ChoculaImporter | Bryan Newbold | 2019-09-03 | 2 | -0/+137 | |
| | ||||||
* | improvements to wayback_static importer | Bryan Newbold | 2019-08-22 | 1 | -6/+29 | |
| | ||||||
* | start new ES container worker kafka group | Bryan Newbold | 2019-07-31 | 1 | -0/+2 | |
| | | | | | | | | The previous group seems to have gotten corrupted; my hypothesis is that this is due to pykafka being somewhat flakey, and am planning to move to librdkafka anyways. Re-indexing all the containers is pretty small/easy, so starting a new consumer group works find in this case; release indexer would be a bigger problem. | |||||
* | crossref: allow 'name' fallback (for groups, etc) | Bryan Newbold | 2019-06-24 | 1 | -1/+1 | |
| | ||||||
* | add inflight edit protection to matched importer | Bryan Newbold | 2019-06-24 | 1 | -1/+8 | |
| | ||||||
* | fix typo; do arxiv-specific match import hack | Bryan Newbold | 2019-06-24 | 1 | -3/+14 | |
| | ||||||
* | fix syntax in existing.url cleanup | Bryan Newbold | 2019-06-24 | 1 | -1/+1 | |
| | ||||||
* | fix existing updater | Bryan Newbold | 2019-06-24 | 1 | -2/+3 | |
| | ||||||
* | add minimal file URL cleanups to matched importer | Bryan Newbold | 2019-06-24 | 1 | -0/+8 | |
| | ||||||
* | matched importer: urls, not url | Bryan Newbold | 2019-06-24 | 1 | -1/+1 | |
| | | | | | | This matches the docs in the header. Previous matched imports were using 'cdx' objects with no 'dt' key, but this makes more sense. As far as I know the old 'url' code path was never actually used (or tested, derp). | |||||
* | fix typo in typo | Bryan Newbold | 2019-06-24 | 1 | -1/+1 | |
| | ||||||
* | fix typo in changelog worker | Bryan Newbold | 2019-06-24 | 1 | -1/+1 | |
| | ||||||
* | support extids in matched importer | Bryan Newbold | 2019-06-20 | 1 | -3/+26 | |
| | ||||||
* | more links on new homepage | Bryan Newbold | 2019-06-19 | 2 | -2/+2 | |
| | | | | | matching produce sizes. may want to tweak this config in the future for throughput. | |||||
* | start work on 'generic' search box | Bryan Newbold | 2019-06-13 | 1 | -0/+95 | |
| | ||||||
* | more pubmed importer fixes | Bryan Newbold | 2019-06-03 | 1 | -6/+13 | |
| | ||||||
* | JALC: handle empty publisher string | Bryan Newbold | 2019-05-30 | 1 | -3/+4 | |
| | ||||||
* | journal_metadata import: allow KBART updates | Bryan Newbold | 2019-05-30 | 1 | -5/+16 | |
| | ||||||
* | fix and workaround container entities in release topic | Bryan Newbold | 2019-05-30 | 2 | -2/+8 | |
| | ||||||
* | fix syntax bugs (container elastic worker) | Bryan Newbold | 2019-05-30 | 1 | -5/+5 | |
| | ||||||
* | add container update elastic worker | Bryan Newbold | 2019-05-30 | 2 | -6/+26 | |
| | ||||||
* | file and container update kafka topics | Bryan Newbold | 2019-05-30 | 1 | -54/+69 | |
| | ||||||
* | update elastic for releases when files added | Bryan Newbold | 2019-05-30 | 1 | -1/+36 | |
| | | | | A bunch of remaining TODOs here | |||||
* | make importer extid lookups faster by hiding | Bryan Newbold | 2019-05-29 | 1 | -2/+2 | |
| |