diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2018-08-24 12:34:58 -0700 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2018-08-24 12:34:58 -0700 |
| commit | 1dd9e8da5912ef0f190aacf20d27586559a277f5 (patch) | |
| tree | fd1dddb085c557200d1fc075bcef47a885a44a37 /scalding | |
| parent | 3782311e29b7e477e1936c89f55ff6483fd02e65 (diff) | |
| download | sandcrawler-1dd9e8da5912ef0f190aacf20d27586559a277f5.tar.gz sandcrawler-1dd9e8da5912ef0f190aacf20d27586559a277f5.zip | |
add un-commited scalding debug notes
Diffstat (limited to 'scalding')
| -rw-r--r-- | scalding/scalding-debugging.md | 26 |
1 files changed, 26 insertions, 0 deletions
diff --git a/scalding/scalding-debugging.md b/scalding/scalding-debugging.md index 9143698..5a54742 100644 --- a/scalding/scalding-debugging.md +++ b/scalding/scalding-debugging.md @@ -45,6 +45,22 @@ resolved by ensuring that the `HBaseSource` constructors had exactly identical names and arguments (eg, table names and zookeeper quorums have to be exact matches). +If you get: + + value toTypedPipe is not a member of cascading.pipe.Pipe + +You probably need to [import some types][tdsl] from: + + import com.twitter.scalding.typed.TDsl._ + +[tdsl]: https://github.com/twitter/scalding/wiki/Type-safe-api-reference#interoperating-between-fields-api-and-type-safe-api + +## Running Individual Tests + +You can run a single test matching a string glob pattern like: + + sbt:sandcrawler> testOnly *CdxBackfill* + ## Fields Values of type `List[Fields]` are not printed in the expected way: @@ -67,3 +83,13 @@ Values of type `List[Fields]` are not printed in the expected way: scala> allFields.length res0: Int = 2 + +## SpyGlass Column Selection + +Two equivalent ways to specify `columns`/`column_families`: + + List("f", "file"), + List(new Fields("c"), new Fields("size", "mimetype")), + + List("f", "file", "file") + List(new Fields("c"), new Fields("size"), new Fields("mimetype")), |