aboutsummaryrefslogtreecommitdiffstats
path: root/python/sandcrawler
Commit message (Expand)AuthorAgeFilesLines
* flake8 clean (with current settings)Bryan Newbold2021-10-267-24/+22
* start handling trivial lint cleanups: unused imports, 'is None', etcBryan Newbold2021-10-2615-97/+57
* make fmtBryan Newbold2021-10-2619-571/+741
* ingest_html: update trafilatura TEI-XML output kwargBryan Newbold2021-10-261-1/+1
* python: isort all importsBryan Newbold2021-10-2618-99/+108
* more small fileset ingest tweaksBryan Newbold2021-10-262-6/+21
* persist support for ingest platform table, using existing persist workerBryan Newbold2021-10-152-2/+129
* improve fileset ingest integration with file ingestBryan Newbold2021-10-153-5/+24
* more fileset iterationBryan Newbold2021-10-154-45/+80
* move SPNv2 'simple_get' logic to SPN clientBryan Newbold2021-10-153-52/+31
* filesets: iteration of implementation and docsBryan Newbold2021-10-154-82/+148
* fileset ingest: improve platform parsingBryan Newbold2021-10-151-12/+196
* fileset ingest: improve error handlingBryan Newbold2021-10-154-48/+106
* initial implementation of zenodo platform importBryan Newbold2021-10-151-0/+100
* initial figshare platform helperBryan Newbold2021-10-151-0/+95
* improvements to platform helpersBryan Newbold2021-10-153-34/+44
* component ingest support for dataverse files (individual)Bryan Newbold2021-10-152-13/+31
* progress on web ingest strategyBryan Newbold2021-10-153-12/+121
* fileset ingest progress for dataverseBryan Newbold2021-10-154-23/+291
* local-file version of gen_file_metadataBryan Newbold2021-10-152-2/+43
* progress on dataset ingestBryan Newbold2021-10-154-122/+333
* wrap up previous renaming workBryan Newbold2021-10-153-5/+3
* progress on fileset/dataset ingestBryan Newbold2021-10-154-0/+403
* refactoring; progress on filesetsBryan Newbold2021-10-152-1/+7
* rename some python files for clarityBryan Newbold2021-10-152-0/+0
* pdf ingest: journals.uchicago.edu patternBryan Newbold2021-10-111-0/+8
* spn: avoid 'None' job_idBryan Newbold2021-10-111-2/+2
* ingest: basic 'component' and 'src' supportBryan Newbold2021-10-042-20/+84
* html ingest: report dt with broken CDX recordsBryan Newbold2021-10-041-1/+1
* allow through unknown-scope HTML ingests, for possible SPN importBryan Newbold2021-10-011-11/+5
* html: fix logging of broken CDX URLBryan Newbold2021-10-011-1/+1
* ingest CDX lookup: weigh year+month of capture against in-petabox-or-notBryan Newbold2021-09-301-0/+1
* fix typo with spn_cdx_retry_sec argBryan Newbold2021-09-301-1/+1
* tune SPN CDX retry/wait depending on mode (priority vs daily)Bryan Newbold2021-09-302-3/+5
* yet another bad PDF sha1Bryan Newbold2021-09-301-0/+1
* old HTML extractors: handle null tagBryan Newbold2021-09-081-8/+9
* ingest: more block patterns, for huge databasesBryan Newbold2021-09-081-1/+4
* yet more PDF sha1 to skipBryan Newbold2021-09-031-0/+5
* yet more PDF URL patternsBryan Newbold2021-09-031-0/+48
* ingest: check URL blocklist again after redirectsBryan Newbold2021-09-031-0/+7
* refactor and expand wall/block/cookie URL patternsBryan Newbold2021-09-031-6/+25
* HTML ingest: several more PDF fulltext URL patternsBryan Newbold2021-09-031-0/+87
* HTML ingest: skip noisy print() statementBryan Newbold2021-09-031-1/+1
* HTML ingest: more meta-URI prefixesBryan Newbold2021-08-241-2/+8
* html ingest: detect some blog platforms, and allow lower wordcount thresholdBryan Newbold2021-08-161-0/+6
* html ingest: detect domain homepage (no path) as special caseBryan Newbold2021-08-161-0/+8
* html ingest: skip 'about:blank'Bryan Newbold2021-08-161-0/+3
* more bad PDF hashesBryan Newbold2021-07-261-0/+2
* ingest: fix postgrest lookup bug (double get of GROBID)Bryan Newbold2021-07-261-1/+1
* more blocked-cookie patterns; fix old typoBryan Newbold2021-07-141-2/+2
generated by cgit v1.2.3 (git 2.39.1) at 2024-05-14 07:57:24 +0000