diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2020-03-03 10:24:43 -0800 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2020-03-03 10:24:43 -0800 |
| commit | 720a45a1d9eea673e0f10d3a7dac0ca85fb913d3 (patch) | |
| tree | 8b974774d7d8efeb85446911db73099fecbb667d /notes/tasks/2020-02-18_ingest_backfills.md | |
| parent | 46cd3516637fccd388bac6e0357d9ce7e3c7d8f1 (diff) | |
| download | sandcrawler-720a45a1d9eea673e0f10d3a7dac0ca85fb913d3.tar.gz sandcrawler-720a45a1d9eea673e0f10d3a7dac0ca85fb913d3.zip | |
update (and move) ingest notes
Diffstat (limited to 'notes/tasks/2020-02-18_ingest_backfills.md')
| -rw-r--r-- | notes/tasks/2020-02-18_ingest_backfills.md | 42 |
1 files changed, 0 insertions, 42 deletions
diff --git a/notes/tasks/2020-02-18_ingest_backfills.md b/notes/tasks/2020-02-18_ingest_backfills.md deleted file mode 100644 index 1ab18f4..0000000 --- a/notes/tasks/2020-02-18_ingest_backfills.md +++ /dev/null @@ -1,42 +0,0 @@ - -Select: - - COPY ( - SELECT row_to_json(ingest_request.*) FROM ingest_request - LEFT JOIN ingest_file_result ON ingest_file_result.base_url = ingest_request.base_url - WHERE ingest_request.ingest_type = 'pdf' - AND ingest_file_result.ingest_type = 'pdf' - AND ingest_file_result.updated < NOW() - '2 day'::INTERVAL - AND ingest_file_result.hit = false - AND ingest_file_result.status like 'spn2-error%' - ) TO '/grande/snapshots/reingest_spn2err_20200218.rows.json'; - => COPY 6537 - - COPY ( - SELECT row_to_json(ingest_request.*) FROM ingest_request - LEFT JOIN ingest_file_result ON ingest_file_result.base_url = ingest_request.base_url - WHERE ingest_request.ingest_type = 'pdf' - AND ingest_file_result.ingest_type = 'pdf' - AND ingest_file_result.hit = false - AND ingest_file_result.status like 'wayback-error' - ) TO '/grande/snapshots/reingest_waybackerr_20200218.rows.json'; - => COPY 33022 - -Transform: - - ./scripts/ingestrequest_row2json.py reingest_spn2err_20200218.rows.json > reingest_spn2err_20200218.json - ./scripts/ingestrequest_row2json.py reingest_waybackerr_20200218.rows.json > reingest_waybackerr_20200218.json - -Push to kafka: - - cat reingest_spn2err_20200218.json | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests -p -1 - cat reingest_waybackerr_20200218.json | jq . -c | kafkacat -P -b wbgrp-svc263.us.archive.org -t sandcrawler-prod.ingest-file-requests -p -1 - -Many had null `ingest_request_source`, so won't actually import into fatcat: - - bnewbold@ia601101$ cat reingest_waybackerr_20200218.json | jq .ingest_request_source | sort | uniq -c | sort -n - 1 "savepapernow-web" - 112 "fatcat-ingest-container" - 11750 "fatcat-changelog" - 21159 null - |