aboutsummaryrefslogtreecommitdiffstats
path: root/python/sandcrawler/fileset_strategies.py
Commit message (Collapse)AuthorAgeFilesLines
* filesets: fix archive.org path namingBryan Newbold2022-03-291-7/+8
|
* bugfix: sha1/md5 typoBryan Newbold2022-03-231-1/+1
| | | | Caught this prepping to ingest in to fatcat. Derp!
* small lint/typo/fmt fixesBryan Newbold2022-02-241-1/+1
|
* ingest: handle more fileset failure modesBryan Newbold2022-02-181-1/+5
|
* fileset ingest: better verification of resourcesBryan Newbold2022-01-131-7/+23
|
* null-body -> empty-blobBryan Newbold2022-01-131-0/+4
|
* more fileset ingest tweaksBryan Newbold2021-12-151-0/+5
|
* fileset ingest: more requests timeouts, sessionsBryan Newbold2021-12-151-27/+49
|
* fileset ingest: create tmp subdirectories if neededBryan Newbold2021-12-151-0/+5
|
* fileset ingest: configure IA session from envBryan Newbold2021-12-151-1/+6
| | | | | Note that this doesn't currently work for `upload()`, and as a work-around I created `~/.config/ia.ini` manually on the worker VM.
* fileset ingest: actually use spn2 CLI flagBryan Newbold2021-12-111-1/+1
|
* codespell typos in python (comments)Bryan Newbold2021-11-241-1/+1
|
* make fmt (black 21.9b0)Bryan Newbold2021-10-271-71/+100
|
* fileset: refactor out tables of helpersBryan Newbold2021-10-271-8/+0
| | | | | | | Having these objects invoked in tables resulted in a whole bunch of objects (including children) getting initialized, which seems like the wrong thing to do. Defer this until the actual ingest fileset worker is initialized.
* start handling trivial lint cleanups: unused imports, 'is None', etcBryan Newbold2021-10-261-11/+5
|
* make fmtBryan Newbold2021-10-261-27/+41
|
* python: isort all importsBryan Newbold2021-10-261-6/+7
|
* more small fileset ingest tweaksBryan Newbold2021-10-261-2/+7
|
* more fileset iterationBryan Newbold2021-10-151-6/+20
|
* move SPNv2 'simple_get' logic to SPN clientBryan Newbold2021-10-151-23/+1
|
* filesets: iteration of implementation and docsBryan Newbold2021-10-151-10/+13
|
* fileset ingest: improve error handlingBryan Newbold2021-10-151-4/+3
|
* improvements to platform helpersBryan Newbold2021-10-151-1/+4
|
* progress on web ingest strategyBryan Newbold2021-10-151-1/+102
|
* fileset ingest progress for dataverseBryan Newbold2021-10-151-10/+135
|
* progress on dataset ingestBryan Newbold2021-10-151-6/+51
|
* progress on fileset/dataset ingestBryan Newbold2021-10-151-0/+22