From ca8c5dedea7907cf26ec63309f4e99030e644351 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Mon, 21 May 2018 11:59:14 -0700 Subject: success running with com.twitter.scalding.Tool --- jvm-mapreduce/learning.txt | 9 +++++++++ scald-mvp/README.md | 6 ++---- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/jvm-mapreduce/learning.txt b/jvm-mapreduce/learning.txt index 1fe64bd..6fe1442 100644 --- a/jvm-mapreduce/learning.txt +++ b/jvm-mapreduce/learning.txt @@ -12,6 +12,15 @@ Or, with actual files on hadoop: Horray! One issue with this was that building scalding took *forever* (meaning 30+ minutes). +potentially instead: + + hadoop jar scald-mvp-assembly-0.1.0-SNAPSHOT.jar com.twitter.scalding.Tool main.scala.example.WordCountJob --hdfs --input hdfs:///user/bnewbold/dummy.txt --output hdfs:///user/bnewbold/test_scalding_out2 + +Hypothesis: class name should be same as file name. Don't need `main` function +if using Scalding Tool wrapper jar. Don't need scald.rb. + + hadoop jar scald-mvp-assembly-0.1.0-SNAPSHOT.jar com.twitter.scalding.Tool example.WordCount --hdfs --input hdfs:///user/bnewbold/dummy.txt --output hdfs:///user/bnewbold/test_scalding_out2 + ## sbt Uncommenting this line in scalding:build.sbt sped things way up (don't need to diff --git a/scald-mvp/README.md b/scald-mvp/README.md index 10cac0f..dfa2bca 100644 --- a/scald-mvp/README.md +++ b/scald-mvp/README.md @@ -23,8 +23,6 @@ running on my laptop: # on cluster: yarn jar scald-mvp-assembly-0.1.0-SNAPSHOT.jar WordCount --hdfs --input hdfs:///user/bnewbold/dummy.txt -## ATTIC +later, using hadop command instead: -wrote build.sbt from scratch - -`sbt` command from `twitter/scalding` upstream repo + hadoop jar scald-mvp-assembly-0.1.0-SNAPSHOT.jar com.twitter.scalding.Tool example.WordCount --hdfs --input hdfs:///user/bnewbold/dummy.txt --output hdfs:///user/bnewbold/test_scalding_out2 -- cgit v1.2.3