diff options
author | Bryan Newbold <bnewbold@archive.org> | 2018-08-24 12:19:09 -0700 |
---|---|---|
committer | Bryan Newbold <bnewbold@archive.org> | 2018-08-24 12:19:09 -0700 |
commit | f50d4e081f7994a167c4974ee9d3f6e1f8eae478 (patch) | |
tree | 00cf69ffe345f7766e7477cb9b1f5f7448b4e4fb /pig | |
parent | 344531eb6a5cdd4ea15e4d82050368c5af0eafee (diff) | |
parent | 5340caad7b39ad29bba77d2a3e486db7a6b1977b (diff) | |
download | sandcrawler-f50d4e081f7994a167c4974ee9d3f6e1f8eae478.tar.gz sandcrawler-f50d4e081f7994a167c4974ee9d3f6e1f8eae478.zip |
Merge branch 'bnewbold-match-quality'
Manually resolved merge conflict in:
please
Diffstat (limited to 'pig')
-rw-r--r-- | pig/README.md | 9 | ||||
-rwxr-xr-x | pig/fetch_deps.sh | 20 |
2 files changed, 5 insertions, 24 deletions
diff --git a/pig/README.md b/pig/README.md index d14d2ae..df8ce68 100644 --- a/pig/README.md +++ b/pig/README.md @@ -12,12 +12,13 @@ by `fetch_deps.sh`) due to [dependency/jar issues][pig-bug] in local mode of To run tests, you need Java installed and `JAVA_HOME` configured. -Fetch dependencies (pig): +Fetch dependencies (including pig) from top-level directory: - ./fetch_deps.sh + ./fetch_hadoop.sh -Write .pig scripts here, and add a pytho wrapper test to `./tests/` when done. -Test vector files (input/output) can go in `./tests/files/`. +Write `.pig` scripts in this directory, and add a python wrapper test to +`./tests/` when done. Test vector files (input/output) can go in +`./tests/files/`. Run the tests with: diff --git a/pig/fetch_deps.sh b/pig/fetch_deps.sh deleted file mode 100755 index 4cefa5e..0000000 --- a/pig/fetch_deps.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail - -#PIG_VERSION="0.12.0-cdh5.2.0" -# Using more recent version to work around snappy classpath problem -PIG_VERSION="0.17.0" -JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::") - -mkdir -p deps/ -cd deps/ - -# Fetch Pig -#wget -c https://archive.cloudera.com/cdh5/cdh/5/pig-${PIG_VERSION}.tar.gz -#wget -c http://mirror.metrocast.net/apache/pig/pig-${PIG_VERSION}/pig-${PIG_VERSION}.tar.gz -wget -c https://archive.org/serve/hadoop_pig_mirror/pig-${PIG_VERSION}.tar.gz -tar xvf pig-${PIG_VERSION}.tar.gz -ln -fs pig-${PIG_VERSION} pig -./pig/bin/pig -x local -version - |