diff options
| author | Martin Czygan <martin.czygan@gmail.com> | 2021-05-18 22:55:03 +0200 |
|---|---|---|
| committer | Martin Czygan <martin.czygan@gmail.com> | 2021-05-18 22:55:03 +0200 |
| commit | 0e8b18a2cc200a75fa69c808404e298d6ea0d154 (patch) | |
| tree | 72c15be56a8be863f19344ec286f0eb2eb712bee /pig/README.md | |
| parent | bb0e9b312f8248e882f8650897966ff57117aa17 (diff) | |
| download | refcat-0e8b18a2cc200a75fa69c808404e298d6ea0d154.tar.gz refcat-0e8b18a2cc200a75fa69c808404e298d6ea0d154.zip | |
pig test run resulted in 2k matches (pdf only)
Diffstat (limited to 'pig/README.md')
| -rw-r--r-- | pig/README.md | 25 |
1 files changed, 25 insertions, 0 deletions
diff --git a/pig/README.md b/pig/README.md index 5281169..c186e8e 100644 --- a/pig/README.md +++ b/pig/README.md @@ -30,6 +30,31 @@ $ pig -p INPUT_CDX=/user/wmdata2/cdx-all-index/20210422171221/part-a-00031.gz -p * http://ia802401.us.archive.org:6988/cluster/app/application_1611217683160_298042 * http://ia802401.us.archive.org:6988/proxy/application_1611217683160_298042/ +Running against 1/300 block of global CDX took about 15h. + +``` +2021-05-18 09:15:24,941 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server +2021-05-18 09:15:30,959 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success! + +real 882m52.192s +user 44m25.048s +sys 5m51.749s +``` + +How many links? On live web? + +``` +$ gohdfs cat /user/martin/fatcat-refs-lookup-0/part-r-00000 | awk '{ print $3 }' > refs_links_testrun.tsv +$ time cat refs_links_testrun.tsv | clinker -w 128 -verbose > refs_links_liveweb.json +$ wc -l refs_links_liveweb.json +2623 refs_links_liveweb.json +$ jq -rc .status refs_links_liveweb.json | sort | uniq -c | sort -nr 2> /dev/null + 2252 200 + 266 403 + 154 404 + 10 null +``` + ---- |