index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
path:
root
/
scalding
/
src
/
test
/
scala
Commit message (
Collapse
)
Author
Age
Files
Lines
*
Added accent removal to titleToSlug().
Ellen Spertus
2018-07-28
1
-1
/
+24
|
*
Added tests (both pass) to try to understand crash when run on real datwa.
Ellen Spertus
2018-07-28
1
-0
/
+9
|
*
Computes and outputs (score, sha1, doi, grobidTitle, crossrefTitle).
Ellen Spertus
2018-07-26
1
-9
/
+22
|
*
Made progress on crossrefPipe.
Ellen Spertus
2018-07-25
1
-6
/
+6
|
*
Fixed bug with reading from TextLine. (Thanks, Bryan\!) Still had to comment ↵
Ellen Spertus
2018-07-25
1
-5
/
+4
|
|
|
|
out some tests.
*
Restored my old tests. Commented out broken tests.
Ellen Spertus
2018-07-25
1
-22
/
+24
|
*
Checked all fields of first entry in grobid pipe.
Ellen Spertus
2018-07-25
1
-2
/
+4
|
*
Grobid entries without legal slugs are removed from the pipe.
Ellen Spertus
2018-07-25
1
-6
/
+12
|
*
Figured out string conversion. Tests pass. Still WIP.
Ellen Spertus
2018-07-25
1
-3
/
+7
|
*
WIP. I'm having problems converting between ImmutableBytesWritable and String.
Ellen Spertus
2018-07-24
1
-5
/
+44
|
*
Clean-up
Ellen Spertus
2018-07-24
1
-2
/
+3
|
*
Added grobidToSlug().
Ellen Spertus
2018-07-24
1
-5
/
+75
|
*
Changed return type of grobidToSlug() to Option[String].
Ellen Spertus
2018-07-24
1
-3
/
+9
|
*
grobidToSlug() seems to work, including parsing of valid JSON strings.
Ellen Spertus
2018-07-24
1
-0
/
+73
|
*
Undid changes that did not belong in this branch.
Ellen Spertus
2018-07-19
1
-117
/
+0
|
*
more complete CdxBackfillJob (untested)
Bryan Newbold
2018-07-19
1
-2
/
+68
|
*
fix confusing whitespace in HBaseStatusCountTest
Bryan Newbold
2018-07-19
1
-11
/
+11
|
*
early work on scalding CDX backfill
Bryan Newbold
2018-07-19
1
-0
/
+51
|
*
refactor HBaseStatusCountJob to convert Long column
Bryan Newbold
2018-07-17
1
-17
/
+19
|
*
Merge branch 'bnewbold-args'
Bryan Newbold
2018-07-13
3
-3
/
+12
|
\
|
*
snake_case to lowerCamelCase
Bryan Newbold
2018-07-13
3
-12
/
+12
|
|
|
|
|
|
|
|
|
|
|
|
... following scala norms. (as per Ellen's review)
|
*
hbase table and zk host as arguments
Bryan Newbold
2018-06-17
3
-3
/
+12
|
|
*
|
Changed grobid0:status to grobid0:status_code, per issue 1.
Ellen Spertus
2018-06-18
1
-1
/
+1
|
/
*
Refactored HBaseStatusCount{Job,Test}.scala.
Ellen Spertus
2018-06-09
1
-1
/
+1
|
*
Pulled out HBaseBuilder.parseColSpec.
Ellen Spertus
2018-06-09
1
-6
/
+6
|
*
Tests pass.
Ellen Spertus
2018-06-08
1
-1
/
+1
|
*
Added status count.
Ellen Spertus
2018-06-07
1
-0
/
+61
|
*
Made test data more robust.
Ellen Spertus
2018-06-06
1
-2
/
+2
|
*
Removed copied comment.
Ellen Spertus
2018-06-06
1
-8
/
+1
|
*
Added job and test for counting mime types.
Ellen Spertus
2018-06-06
1
-0
/
+68
|
*
Made package names match directory names. Cleaned up imports.
Ellen Spertus
2018-06-05
3
-11
/
+9
|
*
Made changes suggested in merge request review.
Ellen Spertus
2018-06-04
1
-5
/
+5
|
|
|
|
|
- Changed inverseSchema from Map to List, eliminating incorrect comment. - Changing format of argument to HBaseBuilder.build from String to List[String].
*
Changed interface to HBaseBuilder.parseColSpec.
Ellen Spertus
2018-06-03
1
-5
/
+5
|
*
Added HBaseBuilder.parseColSpecs and tests, which pass.
Ellen Spertus
2018-06-03
1
-0
/
+43
|
*
Factored common code out of HBaseRowCountJob and its test into a new ↵
Ellen Spertus
2018-06-01
1
-11
/
+3
|
|
|
|
companion object.
*
switch HBaseRowCountJob to SCAN_ALL
Bryan Newbold
2018-05-29
1
-4
/
+10
|
*
HBaseRowCountJob actually counts rows
Bryan Newbold
2018-05-29
1
-2
/
+6
|
*
rename jvm/scalding directories
Bryan Newbold
2018-05-24
3
-0
/
+153