aboutsummaryrefslogtreecommitdiffstats
path: root/mapreduce
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@archive.org>2018-04-06 17:18:49 -0700
committerBryan Newbold <bnewbold@archive.org>2018-04-06 17:18:49 -0700
commit574d84bfbb84d2c46661d5400b1ff50b013195c0 (patch)
treee4d40bdd94012ee60a90f6e2cf3b6d470334b0af /mapreduce
parentc9af02681cb8ddf9b01225980792023bc46cb2f8 (diff)
downloadsandcrawler-574d84bfbb84d2c46661d5400b1ff50b013195c0.tar.gz
sandcrawler-574d84bfbb84d2c46661d5400b1ff50b013195c0.zip
updates to running
Diffstat (limited to 'mapreduce')
-rw-r--r--mapreduce/README.md19
1 files changed, 14 insertions, 5 deletions
diff --git a/mapreduce/README.md b/mapreduce/README.md
index 47747c2..99dd4f9 100644
--- a/mapreduce/README.md
+++ b/mapreduce/README.md
@@ -20,28 +20,37 @@ TODO: GROBID and HBase during development?
## Extraction Task
-TODO:
+An example actually connecting to HBase from a local machine, with thrift
+running on a devbox and GROBID running on a dedicated machine:
+
+ ./extraction_cdx_grobid.py \
+ --hbase-table wbgrp-journal-extract-0-qa \
+ --hbase-host bnewbold-dev.us.archive.org \
+ --grobid-uri http://wbgrp-svc096.us.archive.org:8070
+ tests/files/example.cdx
## Backfill Task
An example actually connecting to HBase from a local machine, with thrift
running on a devbox:
- ./backfill_hbase_from_cdx.py tests/files/example.cdx \
+ ./backfill_hbase_from_cdx.py \
--hbase-table wbgrp-journal-extract-0-qa \
--hbase-host bnewbold-dev.us.archive.org
+ tests/files/example.cdx
Actual invocation to run on Hadoop cluster (running on an IA devbox, where
hadoop environment is configured):
# Create tarball of virtualenv
pipenv shell
- tar -czf backfill-4OmRI0zZ.tar.gz -C /home/bnewbold/.local/share/virtualenvs/backfill-4OmRI0zZ .
+ export VENVSHORT=`basename $VIRTUAL_ENV`
+ tar -czf $VENVSHORT.tar.gz -C /home/bnewbold/.local/share/virtualenvs/$VENVSHORT .
./backfill_hbase_from_cdx.py \
- -r hadoop \
--hbase-host bnewbold-dev.us.archive.org \
--hbase-table wbgrp-journal-extract-0-qa \
+ -r hadoop \
-c mrjob.conf \
- --archive backfill-4OmRI0zZ.tar.gz#venv \
+ --archive $VENVSHORT#venv \
hdfs:///user/bnewbold/journal_crawl_cdx/citeseerx_crawl_2017.cdx