summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/importers/common.py
Commit message (Collapse)AuthorAgeFilesLines
* flush importer editgroups every few minutesBryan Newbold2019-12-121-5/+20
|
* EntityImporter: submit (not accept) modeBryan Newbold2019-12-121-2/+14
| | | | | For use with bots that don't have admin privileges, or where human follow-up review is desired.
* crude support for 'sandcrawler' kafka topicsBryan Newbold2019-11-151-2/+3
|
* refactor duplicated b32_hex function in importersBryan Newbold2019-10-081-0/+9
|
* review/fix all confluent-kafka produce codeBryan Newbold2019-09-201-1/+0
|
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-201-10/+24
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* small kafka tweaks for robustnessBryan Newbold2019-09-201-0/+3
|
* convert importers to confluent-kafka libraryBryan Newbold2019-09-201-19/+71
|
* refactor all python source for client lib nameBryan Newbold2019-09-051-3/+3
|
* fix Importer editgroup_extra pass-throughBryan Newbold2019-09-051-2/+1
|
* file rel: social -> academicsocialBryan Newbold2019-09-031-2/+2
|
* better importer 'total' countingBryan Newbold2019-09-031-4/+2
|
* make importer extid lookups faster by hidingBryan Newbold2019-05-291-2/+2
|
* is_cjk() handles kanji betterBryan Newbold2019-05-291-4/+6
|
* faster LargeFile XML importer for PubMedBryan Newbold2019-05-291-0/+50
|
* more MARC languages, and less verbose reportingBryan Newbold2019-05-241-3/+14
|
* missing MARC/ISO languagesBryan Newbold2019-05-221-0/+2
|
* Gaelic!Bryan Newbold2019-05-221-0/+3
|
* creative importer for bulk JSTOR importsBryan Newbold2019-05-221-0/+22
|
* bs4 XML parse cleanupBryan Newbold2019-05-221-0/+2
|
* JALC bulk file importerBryan Newbold2019-05-211-0/+20
|
* updates to pubmed importerBryan Newbold2019-05-211-1/+20
|
* tweaks to new imports/testsBryan Newbold2019-05-211-5/+78
|
* initial flesh out of JALC parserBryan Newbold2019-05-211-0/+36
|
* python implBryan Newbold2019-05-141-3/+3
|
* add limits to match importersBryan Newbold2019-04-231-0/+3
|
* archive.org isn't really a repositoryBryan Newbold2019-04-221-1/+3
|
* mechanism to not double-update entitiesBryan Newbold2019-04-181-0/+3
|
* update URL rel listBryan Newbold2019-04-181-1/+10
|
* add SqlitePusher importer optionBryan Newbold2019-04-121-0/+20
|
* bunch of lint/whitespace cleanupsBryan Newbold2019-02-221-2/+2
|
* fix bug in clean() resulting in many consistency check failsBryan Newbold2019-01-291-2/+3
|
* add stub parse_record() to make pylint happyBryan Newbold2019-01-281-0/+4
|
* don't allow empty or single-character clean stringsBryan Newbold2019-01-281-1/+1
|
* transform and import fixes/tweaksBryan Newbold2019-01-251-2/+2
|
* refactor _get_editgroup => get_editgroup_idBryan Newbold2019-01-241-4/+5
|
* refactor make_rel_urlBryan Newbold2019-01-241-0/+60
|
* clean() checks if it returns null-length stringBryan Newbold2019-01-231-1/+5
|
* ftfy all over (needs Pipfile.lock)Bryan Newbold2019-01-231-0/+31
|
* more tests; fix some importer behaviorBryan Newbold2019-01-231-10/+20
|
* improve changelog testsBryan Newbold2019-01-231-1/+0
|
* refactor remaining importersBryan Newbold2019-01-221-174/+90
|
* refactored crossref importer to new styleBryan Newbold2019-01-221-17/+107
|
* new importer API interfacesBryan Newbold2019-01-221-0/+166
|
* use full-on autoaccept modeBryan Newbold2019-01-111-2/+3
| | | | | | | | Now that editor_id is infered from token, don't *need* to create ahead of time. This backend change simplifies things greatly (either update an existing editgroup, or create new and *only* include entities in the batch transaction), at the cost of being able to configure the editgroup in any way, including setting a description.
* importers and tests all use new api-passingBryan Newbold2019-01-081-0/+1
|
* start updating importer auth with crossref importerBryan Newbold2019-01-081-9/+23
|
* don't need to supply editor_id nowBryan Newbold2018-12-311-6/+3
|
* python impl of API ident harmonizationBryan Newbold2018-12-241-6/+6
|
* start supporting kafka importersBryan Newbold2018-11-191-0/+17
| | | | A nice feature would be some/any log output as to progress.
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-27 22:33:41 +0000