aboutsummaryrefslogtreecommitdiffstats
path: root/python/sandcrawler/html_ingest.py
Commit message (Expand)AuthorAgeFilesLines
* rename some python files for clarityBryan Newbold2021-10-151-441/+0
* html ingest: report dt with broken CDX recordsBryan Newbold2021-10-041-1/+1
* html: fix logging of broken CDX URLBryan Newbold2021-10-011-1/+1
* html ingest: detect some blog platforms, and allow lower wordcount thresholdBryan Newbold2021-08-161-0/+6
* html ingest: detect domain homepage (no path) as special caseBryan Newbold2021-08-161-0/+8
* DOAJ and HTML ingest tweaks from QA runBryan Newbold2020-11-101-2/+2
* html: handle more traf error casesBryan Newbold2020-11-081-2/+2
* html: most small platform tweaksBryan Newbold2020-11-081-5/+4
* move fuzzy URL match method to miscBryan Newbold2020-11-081-19/+1
* html: more robust ingest; better platform and scope detectionBryan Newbold2020-11-081-32/+96
* html: small ingest improvementsBryan Newbold2020-11-081-0/+4
* html: start improving scope detectionBryan Newbold2020-11-081-4/+48
* gen_file_metadata: allow empty/null bodies (if flag set)Bryan Newbold2020-11-081-1/+1
* html: missing fetch is wayback-content-error, not wayback-errorBryan Newbold2020-11-081-2/+2
* html: handle no-capture for sub-resourcesBryan Newbold2020-11-081-8/+5
* html: refactors/tweaks from testingBryan Newbold2020-11-061-12/+18
* initial implementation of HTML ingest in existing workerBryan Newbold2020-11-041-7/+22
* html: some refactoringBryan Newbold2020-11-031-13/+16
* move transfer encoding helper to sandcrawler/ia.pyBryan Newbold2020-11-031-22/+16
* html: syntax fixes; resolve relative URLs; extract more XML fulltext URLsBryan Newbold2020-10-301-3/+3
* html: work around firstmonday DOCTYPE issueBryan Newbold2020-10-301-0/+3
* html: more ingest improvementsBryan Newbold2020-10-301-18/+118
* html ingest: improve data flowBryan Newbold2020-10-291-18/+41
* better default CLI output (show usage)Bryan Newbold2020-10-291-1/+1
* html: initial ingest implementationBryan Newbold2020-10-291-0/+193
generated by cgit v1.2.3 (git 2.39.1) at 2024-04-29 19:25:48 +0000