From d2203182c9ed6e1ff13fa70fb25f049ef87c75a0 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Thu, 29 Mar 2018 16:04:44 -0700 Subject: sandcrawler --- README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6ea387f..1a251eb 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,15 @@ + _ _ + _________ ___ __ _ _ __ __| | ___ _ __ __ ___ _| | ___ _ __ + \ | / __|/ _` | '_ \ / _` |/ __| '__/ _` \ \ /\ / / |/ _ \ '__| + \ | \__ \ (_| | | | | (_| | (__| | | (_| |\ V V /| | __/ | + \@@@@@@| |___/\__,_|_| |_|\__,_|\___|_| \__,_| \_/\_/ |_|\___|_| + + This repo contains hadoop tasks (mapreduce and pig), luigi jobs, and other -scripts and code for the journal ingest pipeline. +scripts and code for the internet archive (web group) journal ingest pipeline. -This repository is potentially public. Maybe we'll rename it "sandcrawler"? +This repository is potentially public. Archive-specific deployment/production guides and ansible scripts at: [journal-infra](https://git.archive.org/bnewbold/journal-infra) -- cgit v1.2.3