aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@archive.org>2018-06-04 13:38:39 -0700
committerBryan Newbold <bnewbold@archive.org>2018-06-04 13:38:39 -0700
commitaef52ff041258c1eaa5bcbb623538346d6a8668d (patch)
treea7eba626d501c4b918424e413c237b4868d2264c
parent8e3a5ba6f3bdfad32e9bfefeebcb85bbe254cbc9 (diff)
downloadsandcrawler-aef52ff041258c1eaa5bcbb623538346d6a8668d.tar.gz
sandcrawler-aef52ff041258c1eaa5bcbb623538346d6a8668d.zip
fetch SpyGlass jar from archive.org (not local)
-rw-r--r--scalding/README.md25
-rw-r--r--scalding/build.sbt1
2 files changed, 7 insertions, 19 deletions
diff --git a/scalding/README.md b/scalding/README.md
index 13c2e1a..c40da5c 100644
--- a/scalding/README.md
+++ b/scalding/README.md
@@ -30,26 +30,13 @@ Run on cluster:
If your `sbt` task fails with this error:
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Metaspace
+
try restarting `sbt` with more memory (e.g., `sbt -mem 2048`).
-## Building SpyGlass Jar
+## SpyGlass Jar
SpyGlass is a "scalding-to-HBase" connector. It isn't maintained, so we needed
-to rebuild to support our versions of HBase/scalding/etc. From SpyGlass fork
-(<https://github.com/bnewbold/SpyGlass>,
-`bnewbold-scala2.11` branch):
-
- cd ~/src/SpyGlass
- git checkout bnewbold-scala2.11
-
- # This builds the new .jar and installs it in the (laptop local) ~/.m2
- # repository
- mvn clean install -U
-
- # Copy that .jar (and associated pom.xml) over to where sbt can find it
- mkdir -p ~/.sbt/preloaded/parallelai/
- cp -r ~/.m2/repository/parallelai/parallelai.spyglass ~/.sbt/preloaded/parallelai/
-
-The medium-term plan here is to push the custom SpyGlass jar as a static maven
-repo to an archive.org item, and point build.sbt to that folder.
-
+to rebuild to support our versions of HBase/scalding/etc. Our fork (including
+build instructions) is at <https://github.com/bnewbold/SpyGlass>
+(`bnewbold-scala2.11` branch); compiled .jar files are available from
+<https://archive.org/download/ia_sandcrawler_maven2>.
diff --git a/scalding/build.sbt b/scalding/build.sbt
index 1a90168..f333111 100644
--- a/scalding/build.sbt
+++ b/scalding/build.sbt
@@ -19,6 +19,7 @@ lazy val root = (project in file(".")).
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases",
resolvers += "Cloudera Maven Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos",
resolvers += "Twitter Maven Repository" at "https://maven.twttr.com",
+ resolvers += "IA Sandcrawler Rebuilt Jars" at "https://archive.org/download/ia_sandcrawler_maven2/repository",
libraryDependencies += scalaTest % Test,
libraryDependencies += "org.scala-lang" % "scala-library" % "2.11.8",