diff options
| -rw-r--r-- | notes/match_filter_enrich.txt | 12 |
1 files changed, 12 insertions, 0 deletions
diff --git a/notes/match_filter_enrich.txt b/notes/match_filter_enrich.txt index 58d496b..0c9a2c3 100644 --- a/notes/match_filter_enrich.txt +++ b/notes/match_filter_enrich.txt @@ -17,3 +17,15 @@ Join/merge the output: zcat 2018-09-14-0559.05-dumpfilemeta.tsv.gz | LC_ALL=C join -t$'\t' 2018-08-27-2352.17-matchcrossref.filtered.tsv - | pv -l | ./enrich_scored_matches.py | gzip > 2018-08-27-2352.17-matchcrossref.insertable.json.gz # 5.79M 0:09:09 [10.5k/s] +## Fatcat Insertable + +I can't remember now what the plan was for the 'insertable' output mode, which +bundles {key, cdx, mime, and size} info along with the {slug, score, json1, +json2} columns from the regular match script. The filter_scored_matches.py +doesn't know what to do with those columns at the moment, and the output isn't +sorted by slug... need to tweak scripts to fix this. + +In the meanwhile, as a work around just take the columns we want and resort: + + export LC_ALL=C + zcat 2018-12-18-2237.09-matchcrossref.insertable.tsv.gz | cut -f2-5 | sort -u | gzip > 2018-12-18-2237.09-matchcrossref.tsv.gz |