aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* update please helpers to provide hbase+zk configBryan Newbold2018-07-151-2/+13
|
* Merge branch 'bnewbold-args'Bryan Newbold2018-07-135-16/+30
|\
| * snake_case to lowerCamelCaseBryan Newbold2018-07-135-18/+18
| | | | | | | | | | | | ... following scala norms. (as per Ellen's review)
| * hbase table and zk host as argumentsBryan Newbold2018-06-175-16/+30
| |
* | Merge branch 'spertus-issue1' into 'master'bnewbold2018-07-132-2/+2
|\ \ | | | | | | | | | | | | Changed grobid0:status to grobid0:status_code, per issue 1. See merge request webgroup/sandcrawler!7
| * | Changed grobid0:status to grobid0:status_code, per issue 1.Ellen Spertus2018-06-182-2/+2
| | |
* | | commit old tweak to pig script (from cluster)Bryan Newbold2018-07-061-2/+4
| | |
* | | possibly-broken version of hbase-count-rows.pigBryan Newbold2018-07-061-0/+13
| | | | | | | | | | | | | | | | | | This just worked a minute ago, but now throws: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/ByteStringer
* | | monkey-patch SHA-1 blacklistBryan Newbold2018-07-051-0/+8
| |/ |/|
* | Merge branch 'spertus-abstract' into 'master'bnewbold2018-06-177-70/+60
|\| | | | | | | | | Refactoring to enable easy creation of groupby jobs. See merge request webgroup/sandcrawler!6
| * Refactored HBaseStatusCount{Job,Test}.scala.Ellen Spertus2018-06-092-26/+3
| |
| * Cleaned up imports in HBaseMimeCountJob.scala.Ellen Spertus2018-06-091-7/+1
| |
| * Pulled out HBaseBuilder.parseColSpec.Ellen Spertus2018-06-093-20/+25
| |
| * Tests pass.Ellen Spertus2018-06-083-18/+32
| |
* | please: status-countBryan Newbold2018-06-151-0/+21
| |
* | please: extractBryan Newbold2018-06-151-0/+31
| | | | | | | | This script needs refactoring!
* | please: split out rebuild stepsBryan Newbold2018-06-151-3/+18
| |
* | doc improvements and fixes to 'please' helperBryan Newbold2018-06-156-38/+41
| |
* | helper script for running jobsBryan Newbold2018-06-141-0/+86
| |
* | "why scalding"Bryan Newbold2018-06-141-0/+7
|/
* Merge branch 'groupby' into 'master'bnewbold2018-06-074-0/+178
|\ | | | | | | | | Added HBaseMimeCount{Job,Test}, which counts the number of rows with each mimetype, as an example of groupby See merge request webgroup/sandcrawler!4
| * Added status count.Ellen Spertus2018-06-072-0/+89
| |
| * Made test data more robust.Ellen Spertus2018-06-061-2/+2
| |
| * Removed copied comment.Ellen Spertus2018-06-061-8/+1
| |
| * Added job and test for counting mime types.Ellen Spertus2018-06-062-0/+96
| |
* | Merge branch 'spertus-packages' into 'master'bnewbold2018-06-074-16/+13
|\| | | | | | | | | Made package names match directory names. Cleaned up imports. See merge request webgroup/sandcrawler!3
| * Made package names match directory names. Cleaned up imports.Ellen Spertus2018-06-054-16/+13
|/
* Merge branch 'refactoring' into 'master'bnewbold2018-06-044-20/+101
|\ | | | | | | | | Refactoring to add, use, and test class HBaseBuilder to eliminate duplicated code and facilitate HBaseSource creation See merge request webgroup/sandcrawler!1
| * Made changes suggested in merge request review.Ellen Spertus2018-06-043-15/+10
| | | | | | | | | | - Changed inverseSchema from Map to List, eliminating incorrect comment. - Changing format of argument to HBaseBuilder.build from String to List[String].
| * Changed interface to HBaseBuilder.parseColSpec.Ellen Spertus2018-06-033-8/+12
| |
| * Added HBaseBuilder.build() and had HBaseRowCountJob call it.Ellen Spertus2018-06-032-11/+5
| |
| * Added HBaseBuilder.parseColSpecs and tests, which pass.Ellen Spertus2018-06-032-0/+92
| |
| * Factored common code out of HBaseRowCountJob and its test into a new ↵Ellen Spertus2018-06-012-16/+12
| | | | | | | | companion object.
* | Merge branch 'bnewbold-scala-build-fixes' into 'master'bnewbold2018-06-043-21/+19
|\ \ | | | | | | | | | | | | scala build fixes See merge request webgroup/sandcrawler!2
| * | try to run scala tests in gitlab CIBryan Newbold2018-06-041-2/+12
| | |
| * | fetch SpyGlass jar from archive.org (not local)Bryan Newbold2018-06-042-19/+7
| |/
* / bnewbold-dev > wbgrp-svc263Bryan Newbold2018-06-041-4/+4
|/ | | | This is a new production VM running an HBase-Thrift gateway
* Provided full path to cascading jar in command line.Ellen Spertus2018-05-311-1/+1
|
* Added tip on OutOfMemoryError.Ellen Spertus2018-05-311-1/+5
|
* Added debugging info for cascading.tuple.Fields.Ellen Spertus2018-05-311-1/+23
|
* switch HBaseRowCountJob to SCAN_ALLBryan Newbold2018-05-292-5/+11
|
* HBaseRowCountJob actually counts rowsBryan Newbold2018-05-292-13/+8
|
* update version and project nameBryan Newbold2018-05-243-4/+6
|
* cleanup scalding notes/READMEBryan Newbold2018-05-243-37/+162
|
* assemblyMergeStrategy deprecation warningBryan Newbold2018-05-241-2/+2
|
* rename jvm/scalding directoriesBryan Newbold2018-05-2414-71/+0
|
* fix up HBaseRowCountTestBryan Newbold2018-05-242-7/+15
| | | | | Again, seems like test fixture must match *exactly* or very obscure errors crop up.
* get quorum fields to match, fixing testBryan Newbold2018-05-241-1/+1
| | | | | | | | | | | Writing this commit message in anger: It seems that the HBaseSource must match exactly between the instantiated Job class and the JobTest. The error when this isn't the case is very obscure: a `None.get()` exception deep in SpyGlass internals. Blech. This may or may not explain other test failure issues.
* Added repository to find com.hadoop.gplcompression#hadoop-lzo;0.4.16.Ellen Spertus2018-05-221-0/+1
|
* more tests (failing)Bryan Newbold2018-05-222-1/+56
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-27 18:10:52 +0000