diff options
| -rw-r--r-- | notes/Clustering.md | 18 |
1 files changed, 18 insertions, 0 deletions
diff --git a/notes/Clustering.md b/notes/Clustering.md index d390035..d794bdc 100644 --- a/notes/Clustering.md +++ b/notes/Clustering.md @@ -36,3 +36,21 @@ Numbers of clusters: * [ ] do a SS like clustering, using title and author ngrams * [ ] cluster by doi without "vX" suffix + +# Verification + +* we only need to look at identified duplicates, which will be a few millions +* we want fast access to all release JSON blob via ident, maybe do a + "fuzzycat-cache" that copies relevant files into the fs, e.g. +"~/.cache/fuzzycat/releases/d9/e4d4be49faafc750563351a126e7bafe29.json or via microblob (but http we do not need), or sqlite3 (https://www.sqlite.org/fasterthanfs.html) + +For verification we need to have the cached json blobs in some fast, +thread-safe store. Estimated: 1K/s accesses, we still would need a few hours +for a run. + +* [ ] find all ids we need, generate cache, maybe reduce number of fields +* [ ] run verification on each cluster; generate a file of same format of + "verified" clusters; take note the clustering and verification method + +Overall, we can combine various clustering and verification methods. We can +also put together a list of maybe 100-200 test cases and evaluate methods. |