diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2018-03-29 20:16:05 -0700 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2018-03-29 20:16:33 -0700 |
| commit | 7c81b7bea3d670876faff1eb290c40656697dddb (patch) | |
| tree | 4d3413d98089d56fa50de75f0f9c7ea310f02ce4 /mapreduce/cdx-record-pipeline/README.md | |
| parent | d2203182c9ed6e1ff13fa70fb25f049ef87c75a0 (diff) | |
| download | sandcrawler-7c81b7bea3d670876faff1eb290c40656697dddb.tar.gz sandcrawler-7c81b7bea3d670876faff1eb290c40656697dddb.zip | |
move to top level
Diffstat (limited to 'mapreduce/cdx-record-pipeline/README.md')
| -rw-r--r-- | mapreduce/cdx-record-pipeline/README.md | 33 |
1 files changed, 0 insertions, 33 deletions
diff --git a/mapreduce/cdx-record-pipeline/README.md b/mapreduce/cdx-record-pipeline/README.md deleted file mode 100644 index 797b8eb..0000000 --- a/mapreduce/cdx-record-pipeline/README.md +++ /dev/null @@ -1,33 +0,0 @@ -CDX Record Pipeline (GrobId Edition) -===================================== - -Hadoop based pipeline to process PDFs from a specified IA CDX dataset - -## Local mode example ## - -``` -cat -n /home/bnewbold/100k_random_gwb_pdf.cdx | ./cdx-record-pipeline.py - -``` - -## Cluster mode example ## - -``` -input=100k_random_gwb_pdf.cdx -output=100k_random_gwb_pdf.out -lines_per_map=1000 - -hadoop jar /home/webcrawl/hadoop-2/hadoop-mapreduce/hadoop-streaming.jar - -archives "hdfs://ia802400.us.archive.org:6000/lib/cdx-record-pipeline-venv.zip#cdx-record-pipeline-venv" - -D mapred.reduce.tasks=0 - -D mapred.job.name=Cdx-Record-Pipeline - -D mapreduce.job.queuename=extraction - -D mapred.line.input.format.linespermap=${lines_per_map} - -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat - -input ${input} - -output ${output} - -mapper cdx-record-pipeline.py - -file cdx-record-pipeline.py - -``` - |