diff options
author | Bryan Newbold <bnewbold@archive.org> | 2018-03-30 12:49:49 -0700 |
---|---|---|
committer | Bryan Newbold <bnewbold@archive.org> | 2018-03-30 12:49:49 -0700 |
commit | 719538b753423ac301e686886282695150192e47 (patch) | |
tree | 9a3e7447bcc34ee92cecde1430ad2648db73483c /pig/README.md | |
parent | 1ee37422c649aeb11a3f0bbe873b9c479992a5aa (diff) | |
download | sandcrawler-719538b753423ac301e686886282695150192e47.tar.gz sandcrawler-719538b753423ac301e686886282695150192e47.zip |
clean up pig test stuff
Diffstat (limited to 'pig/README.md')
-rw-r--r-- | pig/README.md | 35 |
1 files changed, 20 insertions, 15 deletions
diff --git a/pig/README.md b/pig/README.md index c518591..86d487a 100644 --- a/pig/README.md +++ b/pig/README.md @@ -2,27 +2,32 @@ As of March 2018, the archive runs Pig version 0.12.0, via CDH5.0.1 (Cloudera Distribution). +"Local mode" unit tests in this folder run with Pig version 0.17.0 (controlled +by `fetch_deps.sh`) due to [dependency/jar issues][pig-bug] in local mode of +0.12.0. + +[pig-bug]: https://issues.apache.org/jira/browse/PIG-3530 + ## Development and Testing -To run pig in development on your laptop, you can either use docker or +Fetch dependencies (pig): + + ./fetch_deps.sh + +Write .pig scripts here, and add a pytho wrapper test to `./tests/` when done. +Test vector files (input/output) can go in `./tests/files/`. -https://hub.docker.com/r/chalimartines/local-pig +Install pipenv system-wide if you don't have it: - wget https://archive.cloudera.com/cdh5/cdh/5/pig-0.12.0-cdh5.0.1.tar.gz - tar xvf pig-*.tar.gz - ln -s pig-0.12.0-cdh5.0.1/pig-0.12.0-cdh5.0.1.jar pig-0.12.0-cdh5.0.1/pig.jar - ./pig-*/bin/pig -x local -version + # or use apt, homebrew, etc + sudo pip3 install pipenv - #XXX: don't need Hadoop? - #wget https://archive.cloudera.com/cdh5/cdh/5/hadoop-2.3.0-cdh5.0.1.tar.gz - #tar xvf hadoop-*.tar.gz - #export HADOOP_HOME=hadoop-2.3* +Run the tests with: -Tests require python3, nosetests3, and pigpy. You can install these with: + pipenv run pytest - pip install pipenv - pipenv install --three +Could also, in theory, use a docker image ([local-pig][]), but it's pretty easy +to just download. -Then: +[local-pig]: https://hub.docker.com/r/chalimartines/local-pig - pipenv run nosetests3 |