summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/importers
Commit message (Collapse)AuthorAgeFilesLines
* fix spn/ingest importer duplication checkBryan Newbold2019-12-221-6/+8
| | | | | | Check was happing after the `return True` by mistake, allowing duplicates in SPN editgroups, and potentially in ingest request editgroups as well.
* write diagnostic messages to stderrMartin Czygan2019-12-161-2/+2
| | | | | During debugging, it can be helpful to keep stdout (e.g. processing results) and dignostic messages separate.
* Merge branch 'martin-importers-common-doc-fix' into 'master'Martin Czygan2019-12-141-13/+10
|\ | | | | | | | | Update EntityImporter docstring. See merge request webgroup/fatcat!9
| * complete parse_record docstringMartin Czygan2019-12-141-0/+6
| |
| * Update EntityImporter docstring.Martin Czygan2019-12-131-13/+4
| | | | | | | | I believe the required method is `parse_record`, not `parse`.
* | add ingest import file collision protectionBryan Newbold2019-12-131-0/+6
| | | | | | | | | | | | | | | | The common case is the same URL being submitted repeatedly during testing. This is only within-editgroup, and per importer (eg, won't work across spn importer "submitted" editgroups), but is better than nothing.
* | update ingest request schemaBryan Newbold2019-12-131-2/+7
| | | | | | | | | | This is mostly changing ingest_type from 'file' to 'pdf', and adding 'link_source'/'link_source_id', plus some small cleanups.
* | remove default mimetype from ingest-file importerBryan Newbold2019-12-131-2/+1
| | | | | | | | We really should just use file_meta result or nothing.
* | revert accidentally commited test timingBryan Newbold2019-12-131-2/+2
| | | | | | | | Also fix a spurious typo.
* | ensure importer description arg isn't clobberedBryan Newbold2019-12-123-5/+5
| |
* | savepapernow result importerBryan Newbold2019-12-122-4/+65
| | | | | | | | Based on ingest-file-results importer
* | flush importer editgroups every few minutesBryan Newbold2019-12-121-5/+20
| |
* | EntityImporter: submit (not accept) modeBryan Newbold2019-12-121-2/+14
|/ | | | | For use with bots that don't have admin privileges, or where human follow-up review is desired.
* add another ingest request source to whitelistBryan Newbold2019-12-101-2/+5
|
* tweaks to file ingest importerBryan Newbold2019-12-031-3/+4
| | | | | - allow overriding source filter whitelist (common case for CLI use) - fix editgroup description env variable pass-through
* re-order ingest want() for better statsBryan Newbold2019-11-151-7/+10
|
* project -> ingest_request_sourceBryan Newbold2019-11-151-6/+6
|
* ingest importer fixesBryan Newbold2019-11-151-3/+4
|
* more ingest importer comments and countsBryan Newbold2019-11-151-1/+28
|
* crude support for 'sandcrawler' kafka topicsBryan Newbold2019-11-151-2/+3
|
* ingest file result importerBryan Newbold2019-11-152-2/+135
|
* crossref: accurate blank title countsBryan Newbold2019-11-051-0/+1
|
* crossref: component typeBryan Newbold2019-11-041-1/+3
|
* crossref: count why skip happenedBryan Newbold2019-11-041-1/+7
| | | | | | Might skip based on release type (eg container, not a paper/release), or missing title, or other reasons. Over 7 million DOIs are getting skipped, curious why.
* crossref: don't skip on short/null subtitleBryan Newbold2019-11-041-1/+1
| | | | This was a bug. Should only set subtitle black, not skip the import.
* refactor duplicated b32_hex function in importersBryan Newbold2019-10-083-21/+11
|
* review/fix all confluent-kafka produce codeBryan Newbold2019-09-201-1/+0
|
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-201-10/+24
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* small kafka tweaks for robustnessBryan Newbold2019-09-201-0/+3
|
* convert importers to confluent-kafka libraryBryan Newbold2019-09-201-19/+71
|
* refactor all python source for client lib nameBryan Newbold2019-09-0514-106/+106
|
* fix Importer editgroup_extra pass-throughBryan Newbold2019-09-051-2/+1
|
* file rel: social -> academicsocialBryan Newbold2019-09-031-2/+2
|
* last chocula import behavior tweaksBryan Newbold2019-09-031-3/+21
|
* more careful chocula import counts; don't re-update empty URLsBryan Newbold2019-09-031-2/+6
|
* better importer 'total' countingBryan Newbold2019-09-031-4/+2
|
* chocula importer: include DOAJ updatesBryan Newbold2019-09-031-2/+2
|
* improvements to chocula importerBryan Newbold2019-09-031-1/+7
|
* implement ChoculaImporterBryan Newbold2019-09-032-0/+137
|
* improvements to wayback_static importerBryan Newbold2019-08-221-6/+29
|
* crossref: allow 'name' fallback (for groups, etc)Bryan Newbold2019-06-241-1/+1
|
* add inflight edit protection to matched importerBryan Newbold2019-06-241-1/+8
|
* fix typo; do arxiv-specific match import hackBryan Newbold2019-06-241-3/+14
|
* fix syntax in existing.url cleanupBryan Newbold2019-06-241-1/+1
|
* fix existing updaterBryan Newbold2019-06-241-2/+3
|
* add minimal file URL cleanups to matched importerBryan Newbold2019-06-241-0/+8
|
* matched importer: urls, not urlBryan Newbold2019-06-241-1/+1
| | | | | | This matches the docs in the header. Previous matched imports were using 'cdx' objects with no 'dt' key, but this makes more sense. As far as I know the old 'url' code path was never actually used (or tested, derp).
* support extids in matched importerBryan Newbold2019-06-201-3/+26
|
* more pubmed importer fixesBryan Newbold2019-06-031-6/+13
|
* JALC: handle empty publisher stringBryan Newbold2019-05-301-3/+4
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-16 17:41:51 +0000