aboutsummaryrefslogtreecommitdiffstats
path: root/mapreduce/cdx-record-pipeline/README.md
blob: 797b8eb9e2ab5dae7831db9d512cfbea3dda6f2a (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
CDX Record Pipeline (GrobId Edition)
=====================================

Hadoop based pipeline to process PDFs from a specified IA CDX dataset

## Local mode example ##

```
cat -n /home/bnewbold/100k_random_gwb_pdf.cdx | ./cdx-record-pipeline.py
 
```

## Cluster mode example ##

```
input=100k_random_gwb_pdf.cdx
output=100k_random_gwb_pdf.out
lines_per_map=1000

hadoop jar /home/webcrawl/hadoop-2/hadoop-mapreduce/hadoop-streaming.jar
	-archives "hdfs://ia802400.us.archive.org:6000/lib/cdx-record-pipeline-venv.zip#cdx-record-pipeline-venv"
	-D mapred.reduce.tasks=0
	-D mapred.job.name=Cdx-Record-Pipeline
	-D mapreduce.job.queuename=extraction
	-D mapred.line.input.format.linespermap=${lines_per_map} 
	-inputformat org.apache.hadoop.mapred.lib.NLineInputFormat 
	-input ${input}
	-output ${output}
	-mapper cdx-record-pipeline.py
	-file cdx-record-pipeline.py

```