refcat/skate/zipkey, branch masterUnnamed repository; edit this file 'description' to name the repository.
https://git.bnewbold.net/refcat/atom?h=master2021-12-07T20:17:58Zbatcher: a single call suffices2021-12-07T20:17:58ZMartin Czyganmartin.czygan@gmail.com2021-12-07T20:17:58Zurn:sha1:5df39489600b8910d356cde9aece09381edf547frename module to gitlab.com/internetarchive/refcat2021-10-20T19:28:06ZMartin Czyganmartin.czygan@gmail.com2021-10-20T19:28:06Zurn:sha1:5591df04795ec1af7fbd5eae8f5994e15e5f4b57
This changes all the import paths to the current canonical location on
http://gitlab.com/internetarchive/refcat.
cleanup (old) clustering related code2021-07-21T22:23:00ZMartin Czyganmartin.czygan@gmail.com2021-07-21T22:23:00Zurn:sha1:912dd621a6ac66f86b2ad32df8db6ebc1570c0f3minor doc fixes2021-07-21T21:54:03ZMartin Czyganmartin.czygan@gmail.com2021-07-21T21:54:03Zurn:sha1:2f69336ef77d6396d6db4d6e8536e94dc7aa2b08update docs2021-07-14T20:59:28ZMartin Czyganmartin.czygan@gmail.com2021-07-14T20:59:28Zurn:sha1:902ae78beda424f203db8b8f59cd5e54516475afreduce: small tweaks2021-07-13T16:01:26ZMartin Czyganmartin.czygan@gmail.com2021-07-13T16:01:26Zurn:sha1:9137aa22a5240bc1d9d0b5583be384bc716060aabatch: drop logging2021-07-10T11:15:17ZMartin Czyganmartin.czygan@gmail.com2021-07-10T11:15:17Zurn:sha1:11eb477e73ccda3e84e46c53324c46ded0b4a840batch: log batch size2021-07-10T10:49:44ZMartin Czyganmartin.czygan@gmail.com2021-07-10T10:49:44Zurn:sha1:7db9f87f5ab9539cae6ee5c6fe3d18d94c0ae5d8reduce: short circuit large groups2021-07-10T10:41:00ZMartin Czyganmartin.czygan@gmail.com2021-07-10T10:41:00Zurn:sha1:ad8a18332a0bd32a3bf7dd1e045c31dcd99ba237
we saw a jump in memory usage, and it may be related to groups with
thousands of elements; e.g. maybe some weird string, that appears too
many times as key, e.g. 123/test; as a first measure, we sort circuit
further batching; other mitigiation may to be limit groups size
completely
reduce: move batch size2021-07-09T16:08:04ZMartin Czyganmartin.czygan@gmail.com2021-07-09T16:08:04Zurn:sha1:4df87f9381ce73074abdb1b00635a342f2abadfd