aboutsummaryrefslogtreecommitdiffstats
path: root/scalding
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@archive.org>2018-08-24 12:34:58 -0700
committerBryan Newbold <bnewbold@archive.org>2018-08-24 12:34:58 -0700
commit1dd9e8da5912ef0f190aacf20d27586559a277f5 (patch)
treefd1dddb085c557200d1fc075bcef47a885a44a37 /scalding
parent3782311e29b7e477e1936c89f55ff6483fd02e65 (diff)
downloadsandcrawler-1dd9e8da5912ef0f190aacf20d27586559a277f5.tar.gz
sandcrawler-1dd9e8da5912ef0f190aacf20d27586559a277f5.zip
add un-commited scalding debug notes
Diffstat (limited to 'scalding')
-rw-r--r--scalding/scalding-debugging.md26
1 files changed, 26 insertions, 0 deletions
diff --git a/scalding/scalding-debugging.md b/scalding/scalding-debugging.md
index 9143698..5a54742 100644
--- a/scalding/scalding-debugging.md
+++ b/scalding/scalding-debugging.md
@@ -45,6 +45,22 @@ resolved by ensuring that the `HBaseSource` constructors had exactly identical
names and arguments (eg, table names and zookeeper quorums have to be exact
matches).
+If you get:
+
+ value toTypedPipe is not a member of cascading.pipe.Pipe
+
+You probably need to [import some types][tdsl] from:
+
+ import com.twitter.scalding.typed.TDsl._
+
+[tdsl]: https://github.com/twitter/scalding/wiki/Type-safe-api-reference#interoperating-between-fields-api-and-type-safe-api
+
+## Running Individual Tests
+
+You can run a single test matching a string glob pattern like:
+
+ sbt:sandcrawler> testOnly *CdxBackfill*
+
## Fields
Values of type `List[Fields]` are not printed in the expected way:
@@ -67,3 +83,13 @@ Values of type `List[Fields]` are not printed in the expected way:
scala> allFields.length
res0: Int = 2
+
+## SpyGlass Column Selection
+
+Two equivalent ways to specify `columns`/`column_families`:
+
+ List("f", "file"),
+ List(new Fields("c"), new Fields("size", "mimetype")),
+
+ List("f", "file", "file")
+ List(new Fields("c"), new Fields("size"), new Fields("mimetype")),