aboutsummaryrefslogtreecommitdiffstats
path: root/python/sandcrawler/html_ingest.py
Commit message (Collapse)AuthorAgeFilesLines
* DOAJ and HTML ingest tweaks from QA runBryan Newbold2020-11-101-2/+2
|
* html: handle more traf error casesBryan Newbold2020-11-081-2/+2
|
* html: most small platform tweaksBryan Newbold2020-11-081-5/+4
|
* move fuzzy URL match method to miscBryan Newbold2020-11-081-19/+1
|
* html: more robust ingest; better platform and scope detectionBryan Newbold2020-11-081-32/+96
|
* html: small ingest improvementsBryan Newbold2020-11-081-0/+4
|
* html: start improving scope detectionBryan Newbold2020-11-081-4/+48
|
* gen_file_metadata: allow empty/null bodies (if flag set)Bryan Newbold2020-11-081-1/+1
| | | | This is for HTML sub-resources, which can validly be empty (I think)
* html: missing fetch is wayback-content-error, not wayback-errorBryan Newbold2020-11-081-2/+2
|
* html: handle no-capture for sub-resourcesBryan Newbold2020-11-081-8/+5
|
* html: refactors/tweaks from testingBryan Newbold2020-11-061-12/+18
|
* initial implementation of HTML ingest in existing workerBryan Newbold2020-11-041-7/+22
|
* html: some refactoringBryan Newbold2020-11-031-13/+16
|
* move transfer encoding helper to sandcrawler/ia.pyBryan Newbold2020-11-031-22/+16
|
* html: syntax fixes; resolve relative URLs; extract more XML fulltext URLsBryan Newbold2020-10-301-3/+3
|
* html: work around firstmonday DOCTYPE issueBryan Newbold2020-10-301-0/+3
|
* html: more ingest improvementsBryan Newbold2020-10-301-18/+118
|
* html ingest: improve data flowBryan Newbold2020-10-291-18/+41
|
* better default CLI output (show usage)Bryan Newbold2020-10-291-1/+1
|
* html: initial ingest implementationBryan Newbold2020-10-291-0/+193
generated by cgit v1.2.3 (git 2.39.1) at 2024-10-31 23:08:46 +0000