sandcrawler/backfill, branch trawler
[no description]
https://git.bnewbold.net/sandcrawler/atom?h=trawler
2018-04-04T18:55:22+00:00
merge backfill into extraction directory
2018-04-04T18:55:22+00:00
Bryan Newbold
bnewbold@archive.org
2018-04-04T18:55:22+00:00
urn:sha1:427dd875958c8a6d2d791d55f9dda300ebdc853b
trivial whitespace
2018-04-04T18:48:48+00:00
Bryan Newbold
bnewbold@archive.org
2018-04-04T18:48:48+00:00
urn:sha1:b948cddc7fe7000fd119af4fa130b9c24da46472
more TODO
2018-04-04T18:48:00+00:00
Bryan Newbold
bnewbold@archive.org
2018-04-04T18:48:00+00:00
urn:sha1:9186783c632b63e7d7c3cd8168139718fba378e9
actually running hadoop job on cluster
2018-04-03T02:25:28+00:00
Bryan Newbold
bnewbold@archive.org
2018-04-03T02:25:28+00:00
urn:sha1:ca2d1bd7eccd619173e1d326c9a2ebb1a2d3d502
fix silly bugs in backfiller (need more tests)
2018-04-03T02:25:03+00:00
Bryan Newbold
bnewbold@archive.org
2018-04-03T02:25:03+00:00
urn:sha1:630e956c05604aaf8bf5b7154a01ad956b13e440
add setuptools (can probably remove)
2018-04-03T02:23:58+00:00
Bryan Newbold
bnewbold@archive.org
2018-04-03T02:23:58+00:00
urn:sha1:d8f3c2ffad0b685db6c3196ac3efe846c019f6d7
heritrix expects ints, not strings, for numbers
2018-04-02T21:50:05+00:00
Bryan Newbold
bnewbold@archive.org
2018-04-02T21:50:05+00:00
urn:sha1:f1c4935b2d55f3acecd824ba9b6318a33820fafe
backfill: sha1 prefix, cluster example
2018-03-31T05:53:03+00:00
Bryan Newbold
bnewbold@archive.org
2018-03-31T05:53:03+00:00
urn:sha1:31d5a1ebdfe2f4638ae1e5ec87ff228eef9114f5
clean up backfill code/tests
2018-03-31T02:12:31+00:00
Bryan Newbold
bnewbold@archive.org
2018-03-31T02:12:31+00:00
urn:sha1:bb38ea065758a719331803b4adf875f2d75a1c3d
refactor backfill for mrjob
2018-03-31T01:29:29+00:00
Bryan Newbold
bnewbold@archive.org
2018-03-31T01:29:29+00:00
urn:sha1:f6915b4b44e312cee7eda9626d0330268ab786e2