aboutsummaryrefslogtreecommitdiffstats
path: root/pig/README.md
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@archive.org>2018-03-30 12:49:49 -0700
committerBryan Newbold <bnewbold@archive.org>2018-03-30 12:49:49 -0700
commit719538b753423ac301e686886282695150192e47 (patch)
tree9a3e7447bcc34ee92cecde1430ad2648db73483c /pig/README.md
parent1ee37422c649aeb11a3f0bbe873b9c479992a5aa (diff)
downloadsandcrawler-719538b753423ac301e686886282695150192e47.tar.gz
sandcrawler-719538b753423ac301e686886282695150192e47.zip
clean up pig test stuff
Diffstat (limited to 'pig/README.md')
-rw-r--r--pig/README.md35
1 files changed, 20 insertions, 15 deletions
diff --git a/pig/README.md b/pig/README.md
index c518591..86d487a 100644
--- a/pig/README.md
+++ b/pig/README.md
@@ -2,27 +2,32 @@
As of March 2018, the archive runs Pig version 0.12.0, via CDH5.0.1 (Cloudera
Distribution).
+"Local mode" unit tests in this folder run with Pig version 0.17.0 (controlled
+by `fetch_deps.sh`) due to [dependency/jar issues][pig-bug] in local mode of
+0.12.0.
+
+[pig-bug]: https://issues.apache.org/jira/browse/PIG-3530
+
## Development and Testing
-To run pig in development on your laptop, you can either use docker or
+Fetch dependencies (pig):
+
+ ./fetch_deps.sh
+
+Write .pig scripts here, and add a pytho wrapper test to `./tests/` when done.
+Test vector files (input/output) can go in `./tests/files/`.
-https://hub.docker.com/r/chalimartines/local-pig
+Install pipenv system-wide if you don't have it:
- wget https://archive.cloudera.com/cdh5/cdh/5/pig-0.12.0-cdh5.0.1.tar.gz
- tar xvf pig-*.tar.gz
- ln -s pig-0.12.0-cdh5.0.1/pig-0.12.0-cdh5.0.1.jar pig-0.12.0-cdh5.0.1/pig.jar
- ./pig-*/bin/pig -x local -version
+ # or use apt, homebrew, etc
+ sudo pip3 install pipenv
- #XXX: don't need Hadoop?
- #wget https://archive.cloudera.com/cdh5/cdh/5/hadoop-2.3.0-cdh5.0.1.tar.gz
- #tar xvf hadoop-*.tar.gz
- #export HADOOP_HOME=hadoop-2.3*
+Run the tests with:
-Tests require python3, nosetests3, and pigpy. You can install these with:
+ pipenv run pytest
- pip install pipenv
- pipenv install --three
+Could also, in theory, use a docker image ([local-pig][]), but it's pretty easy
+to just download.
-Then:
+[local-pig]: https://hub.docker.com/r/chalimartines/local-pig
- pipenv run nosetests3