aboutsummaryrefslogtreecommitdiffstats
path: root/jvm-mapreduce/learning.txt
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@archive.org>2018-05-21 11:59:14 -0700
committerBryan Newbold <bnewbold@archive.org>2018-05-21 11:59:14 -0700
commitca8c5dedea7907cf26ec63309f4e99030e644351 (patch)
tree08c944c1cd0ed90d10a5171bbb56d89e82be53d9 /jvm-mapreduce/learning.txt
parent53c2b11d4d234d933742651051ef57ca3c145382 (diff)
downloadsandcrawler-ca8c5dedea7907cf26ec63309f4e99030e644351.tar.gz
sandcrawler-ca8c5dedea7907cf26ec63309f4e99030e644351.zip
success running with com.twitter.scalding.Tool
Diffstat (limited to 'jvm-mapreduce/learning.txt')
-rw-r--r--jvm-mapreduce/learning.txt9
1 files changed, 9 insertions, 0 deletions
diff --git a/jvm-mapreduce/learning.txt b/jvm-mapreduce/learning.txt
index 1fe64bd..6fe1442 100644
--- a/jvm-mapreduce/learning.txt
+++ b/jvm-mapreduce/learning.txt
@@ -12,6 +12,15 @@ Or, with actual files on hadoop:
Horray! One issue with this was that building scalding took *forever* (meaning
30+ minutes).
+potentially instead:
+
+ hadoop jar scald-mvp-assembly-0.1.0-SNAPSHOT.jar com.twitter.scalding.Tool main.scala.example.WordCountJob --hdfs --input hdfs:///user/bnewbold/dummy.txt --output hdfs:///user/bnewbold/test_scalding_out2
+
+Hypothesis: class name should be same as file name. Don't need `main` function
+if using Scalding Tool wrapper jar. Don't need scald.rb.
+
+ hadoop jar scald-mvp-assembly-0.1.0-SNAPSHOT.jar com.twitter.scalding.Tool example.WordCount --hdfs --input hdfs:///user/bnewbold/dummy.txt --output hdfs:///user/bnewbold/test_scalding_out2
+
## sbt
Uncommenting this line in scalding:build.sbt sped things way up (don't need to