diff options
| author | Bryan Newbold <bnewbold@robocracy.org> | 2019-12-10 10:29:39 -0800 |
|---|---|---|
| committer | Bryan Newbold <bnewbold@robocracy.org> | 2019-12-10 10:29:41 -0800 |
| commit | 7838a3c15a82281eec435ef16aad63e97015bdfc (patch) | |
| tree | c65df48c37b448421da4ef766108d581ab1b3428 /extra/sql_dumps/stats_table_size.sql | |
| parent | a7736d91665f6a98090cd448d02f1542aec6c180 (diff) | |
| download | fatcat-7838a3c15a82281eec435ef16aad63e97015bdfc.tar.gz fatcat-7838a3c15a82281eec435ef16aad63e97015bdfc.zip | |
add ingest-container command (new CLI tool)
The intent of this tool is to make it easy to enque ingest requests into
kafka, to be processed by a worker pool and eventually end up inserted
into fatcat (for ingest hits that pass various checks).
As a specific example use-case, we have pretty good coverage of eLife (a
prominent OA publisher), but have missed some publications in the past,
and have a large gap for the year 2019:
https://fatcat.wiki/container/en4qj5ijrbf5djxx7p5zzpjyoq/coverage
This tool would make it trivial to enqueue all the missing releases to
be crawled.
Future variants on this tool could query for, eg, long-tail OA works.
Diffstat (limited to 'extra/sql_dumps/stats_table_size.sql')
0 files changed, 0 insertions, 0 deletions