diff options
author | bnewbold <bnewbold@archive.org> | 2018-06-04 22:04:44 +0000 |
---|---|---|
committer | bnewbold <bnewbold@archive.org> | 2018-06-04 22:04:44 +0000 |
commit | 70069a78c8798352b5bef815a3fd4aa9e9b52394 (patch) | |
tree | 6e67fe63f6e2051ce7a50184b90637065da8e1a8 /scalding/README.md | |
parent | d2dd016aa8da93ad14654237dbb7cfac214f9da8 (diff) | |
parent | d434f5676a5d37692fe00fb14d72120d0c9933f8 (diff) | |
download | sandcrawler-70069a78c8798352b5bef815a3fd4aa9e9b52394.tar.gz sandcrawler-70069a78c8798352b5bef815a3fd4aa9e9b52394.zip |
Merge branch 'bnewbold-scala-build-fixes' into 'master'
scala build fixes
See merge request webgroup/sandcrawler!2
Diffstat (limited to 'scalding/README.md')
-rw-r--r-- | scalding/README.md | 25 |
1 files changed, 6 insertions, 19 deletions
diff --git a/scalding/README.md b/scalding/README.md index 13c2e1a..c40da5c 100644 --- a/scalding/README.md +++ b/scalding/README.md @@ -30,26 +30,13 @@ Run on cluster: If your `sbt` task fails with this error: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Metaspace + try restarting `sbt` with more memory (e.g., `sbt -mem 2048`). -## Building SpyGlass Jar +## SpyGlass Jar SpyGlass is a "scalding-to-HBase" connector. It isn't maintained, so we needed -to rebuild to support our versions of HBase/scalding/etc. From SpyGlass fork -(<https://github.com/bnewbold/SpyGlass>, -`bnewbold-scala2.11` branch): - - cd ~/src/SpyGlass - git checkout bnewbold-scala2.11 - - # This builds the new .jar and installs it in the (laptop local) ~/.m2 - # repository - mvn clean install -U - - # Copy that .jar (and associated pom.xml) over to where sbt can find it - mkdir -p ~/.sbt/preloaded/parallelai/ - cp -r ~/.m2/repository/parallelai/parallelai.spyglass ~/.sbt/preloaded/parallelai/ - -The medium-term plan here is to push the custom SpyGlass jar as a static maven -repo to an archive.org item, and point build.sbt to that folder. - +to rebuild to support our versions of HBase/scalding/etc. Our fork (including +build instructions) is at <https://github.com/bnewbold/SpyGlass> +(`bnewbold-scala2.11` branch); compiled .jar files are available from +<https://archive.org/download/ia_sandcrawler_maven2>. |