From a1c908b83d6e07c4065be52e405977114a9f37c4 Mon Sep 17 00:00:00 2001 From: Martin Czygan Date: Tue, 14 Sep 2021 00:44:44 +0200 Subject: add todo --- TODO.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 TODO.md diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..5666bc0 --- /dev/null +++ b/TODO.md @@ -0,0 +1,28 @@ +# TODO + +* [ ] clustering should be broken up, e.g. into "map" and "sort" + +In +[refcat/skate](https://gitlab.com/internetarchive/refcat/-/tree/master/skate) +we have one simple operation: extract a list of fields from blob of bytes. We +use [16 +mappers](https://gitlab.com/internetarchive/refcat/-/blob/f33e586d11f5f575f71ad209608ac9ba74fad2e5/skate/cmd/skate-map/main.go#L70-86) +currently, they are easy to write. + +In refcat, we use GNU sort, and just when we need it, e.g. +[skate-map](https://gitlab.com/internetarchive/refcat/-/blob/f33e586d11f5f575f71ad209608ac9ba74fad2e5/python/refcat/tasks.py#L531-534). + +The `Cluster` class bundles, iteration, key extraction, sorting and group by +operation into a single entity. + +Also in refcat, we do not work on a single file with clusters any more, but +mostly with two sorted streams, which are iterated over "comm" style. This +spares us an extra step of generating the cluster documents, but requires an +extra component, that allows to plug in various "reduce" functions. In refcat, +this component is called "zipkey", which is support batching, too. + +* [ ] match release fuzzy should work not just with title +* [ ] match container name functions (maybe also with abbreviations, etc) +* [ ] better documentation, more examples +* [ ] shiv based packaging + -- cgit v1.2.3