diff options
author | Bryan Newbold <bnewbold@archive.org> | 2018-04-03 02:25:57 +0000 |
---|---|---|
committer | Bryan Newbold <bnewbold@archive.org> | 2018-04-03 02:25:57 +0000 |
commit | 9d4520e8e18d7bf9b36d98d330417360194e80a3 (patch) | |
tree | bedee72dd46c0fc444476b7250cd59a51bbb4fb4 | |
parent | ca2d1bd7eccd619173e1d326c9a2ebb1a2d3d502 (diff) | |
download | sandcrawler-9d4520e8e18d7bf9b36d98d330417360194e80a3.tar.gz sandcrawler-9d4520e8e18d7bf9b36d98d330417360194e80a3.zip |
shift docs around a bit
-rw-r--r-- | README.md | 16 | ||||
-rw-r--r-- | pig/README.md | 5 |
2 files changed, 12 insertions, 9 deletions
@@ -1,9 +1,9 @@ _ _ - _________ ___ __ _ _ __ __| | ___ _ __ __ ___ _| | ___ _ __ - \ | / __|/ _` | '_ \ / _` |/ __| '__/ _` \ \ /\ / / |/ _ \ '__| - \ | \__ \ (_| | | | | (_| | (__| | | (_| |\ V V /| | __/ | - \@@@@@@| |___/\__,_|_| |_|\__,_|\___|_| \__,_| \_/\_/ |_|\___|_| + __________ ___ __ _ _ __ __| | ___ _ __ __ ___ _| | ___ _ __ + \ | / __|/ _` | '_ \ / _` |/ __| '__/ _` \ \ /\ / / |/ _ \ '__| + \ | \__ \ (_| | | | | (_| | (__| | | (_| |\ V V /| | __/ | + \ooooooo| |___/\__,_|_| |_|\__,_|\___|_| \__,_| \_/\_/ |_|\___|_| This repo contains hadoop tasks (mapreduce and pig), luigi jobs, and other @@ -14,3 +14,11 @@ This repository is potentially public. Archive-specific deployment/production guides and ansible scripts at: [journal-infra](https://git.archive.org/bnewbold/journal-infra) +## Python Setup + +Pretty much everything here uses python/pipenv. To setup your environment for +this, and python in general: + + # libjpeg-dev is for some wayback/pillow stuff + sudo apt install python3-dev python3-pip python3-wheel libjpeg-dev + pip3 install --user pipenv diff --git a/pig/README.md b/pig/README.md index 86d487a..9e45333 100644 --- a/pig/README.md +++ b/pig/README.md @@ -17,11 +17,6 @@ Fetch dependencies (pig): Write .pig scripts here, and add a pytho wrapper test to `./tests/` when done. Test vector files (input/output) can go in `./tests/files/`. -Install pipenv system-wide if you don't have it: - - # or use apt, homebrew, etc - sudo pip3 install pipenv - Run the tests with: pipenv run pytest |