sandcrawler/pig, branch bnewbold-refactor-loggging [no description] https://git.bnewbold.net/sandcrawler/atom?h=bnewbold-refactor-loggging 2020-01-03T02:12:58+00:00 small (syntax?) changes to pig join script 2020-01-03T02:12:58+00:00 Bryan Newbold bnewbold@archive.org 2019-12-28T01:21:36+00:00 urn:sha1:028a0c27a832833e8833e3b3d0e1d6725a48e953 pig: first rev of join-cdx-sha1 script 2019-12-22T22:26:04+00:00 Bryan Newbold bnewbold@archive.org 2019-12-22T22:26:04+00:00 urn:sha1:28de71e714c1f5d70adcfd3213dc2433a701a430 pig: move count_lines helper to pighelper.py 2019-12-22T22:21:25+00:00 Bryan Newbold bnewbold@archive.org 2019-12-22T22:21:25+00:00 urn:sha1:990f28b283bce4a524b4e32178f45e40214ee0de new/additional GWB CDX filter scripts 2019-10-17T16:19:34+00:00 Bryan Newbold bnewbold@archive.org 2019-10-17T16:19:34+00:00 urn:sha1:54dabe601eaa19d0495d9a102b34e9daa056457d add ojs and dspace as in-domain patterns to look for in heuristic CDX PDF filter 2019-04-12T21:19:29+00:00 Bryan Newbold bnewbold@archive.org 2019-04-12T03:14:22+00:00 urn:sha1:8ac10ab7fe310df55ab5a66d741ea25c24389418 rework fetch_hadoop script 2018-08-24T19:05:41+00:00 Bryan Newbold bnewbold@archive.org 2018-08-24T19:05:39+00:00 urn:sha1:92584ec4201ecc27af423cbff7b4bc1573edf175 Should work on macOS now, and fetches hadoop in addition to pig. Still requires wget (not installed by default on macOS). commit old tweak to pig script (from cluster) 2018-07-06T14:48:57+00:00 Bryan Newbold bnewbold@archive.org 2018-07-06T14:48:39+00:00 urn:sha1:b3149a911df8150056c54f86cd77c3516fc9838c possibly-broken version of hbase-count-rows.pig 2018-07-06T14:48:57+00:00 Bryan Newbold bnewbold@archive.org 2018-05-01T22:33:52+00:00 urn:sha1:fde027fb269214a7e8f71f5717bcf569014b6661 This just worked a minute ago, but now throws: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/ByteStringer fix tests post-DISTINCT 2018-05-08T17:06:20+00:00 Bryan Newbold bnewbold@archive.org 2018-05-08T17:06:14+00:00 urn:sha1:18a55d37a87d4391bd8161201c523dd7d7f0f1e7 Confirms it's working! distinct on SHA1 in cdx scripts 2018-05-08T16:58:24+00:00 Bryan Newbold bnewbold@archive.org 2018-05-08T16:58:24+00:00 urn:sha1:1831a3b4495aee275e4b4b187fa545eba75eb87b