aboutsummaryrefslogtreecommitdiffstats
path: root/scalding/README.md
blob: b09e0e8241e2b72833f7fbdd5a4c126d337c0867 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
This directory contains Hadoop map/reduce jobs written in Scala (compiled to
the JVM) using the Scalding framework. Scalding builds on the Java Cascading
library, which itself builds on the Java Hadoop libraries.

See the other markdown files in this directory for more background and tips.

## Dependencies

To develop locally, you need to have the JVM (eg, OpenJDK 1.8), `sbt` build
tool, and might need (exactly) Scala version 2.11.8.

On a debian/ubuntu machine:

    echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
    sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
    sudo apt-get update
    sudo apt install scala sbt

It's also helpful to have a local copy of the `hadoop` binary for running
benchmarks. The `fetch_hadoop.sh` script in the top level directory will fetch
an appropriate version.

## Building and Running

You can run `sbt` commands individually:

    # run all test
    sbt test

    # build a jar (also runs tests)
    sbt assembly

Or you can start a session and run commands within that, which is *much*
faster:

    sbt -mem 2048

    sbt> test
    sbt> assembly
    sbt> testOnly sandcrawler.SomeTestClassName

On the cluster, you usually use the `please` script to kick off jobs. Be sure
to build the jars first, or pass `--rebuild` to do it automatically. You need
`hadoop` on your path for this.

## Troubleshooting

If your `sbt` task fails with this error:

     java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Metaspace

try restarting `sbt` with more memory (e.g., `sbt -mem 2048`).

See `scalding-debugging.md` or maybe `../notes/` for more.