diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2018-05-21 10:55:22 -0700 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2018-05-21 10:55:22 -0700 |
| commit | 5d5b828730fdf34dcd2a6aeba64c7df2c1be23c5 (patch) | |
| tree | 5f7322b2dc25317ab847d98e182ce801a1a5eae6 /scald-mvp/README.md | |
| parent | b18c68c81b4c426b5d83f2e6c31026b9febcb6e0 (diff) | |
| download | sandcrawler-5d5b828730fdf34dcd2a6aeba64c7df2c1be23c5.tar.gz sandcrawler-5d5b828730fdf34dcd2a6aeba64c7df2c1be23c5.zip | |
copy in scalding learning example
Diffstat (limited to 'scald-mvp/README.md')
| -rw-r--r-- | scald-mvp/README.md | 30 |
1 files changed, 30 insertions, 0 deletions
diff --git a/scald-mvp/README.md b/scald-mvp/README.md new file mode 100644 index 0000000..10cac0f --- /dev/null +++ b/scald-mvp/README.md @@ -0,0 +1,30 @@ + +following https://medium.com/@gayani.nan/how-to-run-a-scalding-job-567160fa193 + + +running on my laptop: + + openjdk version "1.8.0_171" + OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~deb9u1-b11) + OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode) + + Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL + + sbt: 1.1.5 + + sbt new scala/scala-seed.g8 + + # inserted additional deps, tweaked versions + # hadoop 2.5.0 seems to conflict with cascading; sticking with 2.6.0 + + sbt assembly + scp target/scala-2.11/scald-mvp-assembly-0.1.0-SNAPSHOT.jar devbox: + + # on cluster: + yarn jar scald-mvp-assembly-0.1.0-SNAPSHOT.jar WordCount --hdfs --input hdfs:///user/bnewbold/dummy.txt + +## ATTIC + +wrote build.sbt from scratch + +`sbt` command from `twitter/scalding` upstream repo |