summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/importers/ingest.py
Commit message (Collapse)AuthorAgeFilesLines
* ingest import: allow short version of corpus namesBryan Newbold2020-02-181-0/+3
|
* ingest importer: pass through link relBryan Newbold2020-02-181-1/+6
|
* check ingest_request_source existance for SPN as well as ingestBryan Newbold2020-02-061-0/+3
|
* additional trusted link sourcesBryan Newbold2020-02-061-0/+3
|
* add mag and s2 as trusted link sourcesBryan Newbold2020-02-061-1/+1
|
* ingest worker: handle missing ingest_request_sourceBryan Newbold2020-02-061-0/+3
| | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug.
* fix trivial typo in file importerBryan Newbold2020-01-201-1/+1
|
* ingest: improve tests, support old ingest resultsBryan Newbold2020-01-151-3/+12
|
* update ingest worker for schema tweaksBryan Newbold2020-01-151-8/+15
| | | | | | Should be backwards compatible with old ingest results. Fixed a bug with glutton ident detection.
* ingest: allow more sources to auto-importBryan Newbold2020-01-151-1/+2
|
* importers: control update behavior with more-standard flagBryan Newbold2020-01-061-1/+1
|
* allow arabesque backfill ingests for some source typesBryan Newbold2019-12-241-0/+5
|
* fix spn/ingest importer duplication checkBryan Newbold2019-12-221-6/+8
| | | | | | Check was happing after the `return True` by mistake, allowing duplicates in SPN editgroups, and potentially in ingest request editgroups as well.
* add ingest import file collision protectionBryan Newbold2019-12-131-0/+6
| | | | | | | | The common case is the same URL being submitted repeatedly during testing. This is only within-editgroup, and per importer (eg, won't work across spn importer "submitted" editgroups), but is better than nothing.
* update ingest request schemaBryan Newbold2019-12-131-2/+7
| | | | | This is mostly changing ingest_type from 'file' to 'pdf', and adding 'link_source'/'link_source_id', plus some small cleanups.
* remove default mimetype from ingest-file importerBryan Newbold2019-12-131-2/+1
| | | | We really should just use file_meta result or nothing.
* savepapernow result importerBryan Newbold2019-12-121-3/+64
| | | | Based on ingest-file-results importer
* add another ingest request source to whitelistBryan Newbold2019-12-101-2/+5
|
* tweaks to file ingest importerBryan Newbold2019-12-031-3/+4
| | | | | - allow overriding source filter whitelist (common case for CLI use) - fix editgroup description env variable pass-through
* re-order ingest want() for better statsBryan Newbold2019-11-151-7/+10
|
* project -> ingest_request_sourceBryan Newbold2019-11-151-6/+6
|
* ingest importer fixesBryan Newbold2019-11-151-3/+4
|
* more ingest importer comments and countsBryan Newbold2019-11-151-1/+28
|
* ingest file result importerBryan Newbold2019-11-151-0/+134