Grouped Release Exports ======================= Want to have rich "work" entity dumps, without duplicating with enriched release dumps. The motivation for this is to do work-level processing of releases. For example, transform into scholar.archive.org (indexed at work level), bulk downloading fulltext (only need "best copy for entire work"), citation analysis, etc. One elegant way to do this would be having release exports be sorted by `work_id`, with all release entities with the same `work_id` contiguous (but not all on the same line). Plan is to update fatcat-export tool (rust) to read in an ident list with `work_id` included (and sorted by the `work_id`). ## Rust Implementation Remembering that `fatcat-export` operates on lines, and uses channels and worker threads for processing and serialization, we can't just assume that the output lines will be in the same order as the input lines (eg, worker threads can shift order). To preserve order, we will add a new dump mode which operates on batches (`Vec`) of rows instead, and write the batches contiguously. The batches themselves may end up out of order, but that is fine. ## SQL Ident Dump Database timing (running 2020-08-04 on fatcat-qa): COPY (SELECT id, rev_id, redirect_id FROM release_ident WHERE is_live=true) TO '/tmp/fatcat_ident_releases.tsv' WITH NULL ''; => COPY 142635153 => Time: 143264.483 ms (02:23.264) COPY ( SELECT release_ident.id, release_ident.rev_id, release_ident.redirect_id, release_rev.work_ident_id FROM release_ident LEFT JOIN release_rev ON release_ident.rev_id = release_rev.id WHERE release_ident.is_live=true AND release_ident.redirect_id IS NULL AND release_ident.rev_id IS NOT NULL ORDER BY release_rev.work_ident_id ASC NULLS LAST ) TO '/tmp/fatcat_ident_releases_by_work.tsv' WITH NULL ''; => COPY 142635147 => Time: 610540.112 ms (10:10.540) Much slower, but not a blocking issue. Apparently postgresql will do a full rowscan instead of using the existing `work_id` sorted index on `release_rev` when going of a large fraction of the table (here almost the entire table) because it thinks the read will be so much faster. I don't think this is true with SSDs? Haven't actually confirmed this is the behavior, just assuming based on postgresql docs.