diff options
| author | Bryan Newbold <bnewbold@archive.org> | 2021-10-15 18:17:32 -0700 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@archive.org> | 2021-10-15 18:17:32 -0700 |
| commit | 12d041b781912dc376444198c920ade2d6cee7c8 (patch) | |
| tree | b840986d37cfa37e2eb761bdfe426294ccc59da1 | |
| parent | 8310ac89e08afc322122ba9c9365d32950b062d7 (diff) | |
| download | sandcrawler-12d041b781912dc376444198c920ade2d6cee7c8.tar.gz sandcrawler-12d041b781912dc376444198c920ade2d6cee7c8.zip | |
commit SPN account changes
| -rw-r--r-- | proposals/2021-09-21_spn_accounts.md | 14 |
1 files changed, 14 insertions, 0 deletions
diff --git a/proposals/2021-09-21_spn_accounts.md b/proposals/2021-09-21_spn_accounts.md new file mode 100644 index 0000000..e41c162 --- /dev/null +++ b/proposals/2021-09-21_spn_accounts.md @@ -0,0 +1,14 @@ + +Formalization of SPNv2 API requests from fatcat/sandcrawler + +Create two new system accounts, one for regular/daily ingest requests, one for +priority requests (save-paper-now or as a flag with things like fatcat-ingest; +"interactive"). These accounts should have @archive.org emails. Request the +daily one to have the current rate limit as bnewbold@archive.org account; the +priority queue can have less. + +Create new ingest kafka queues from scratch, one for priority and one for +regular. Chose sizes carefully, probably keep 24x for the regular and do 6x or +so (small) for priority queue. + +Deploy new priority workers; reconfigure/deploy broadly. |