notes from prod run of file de-dupe

2 files changed, 36 insertions, 0 deletions

diff --git a/notes/bulk_edits/2021-11-24_file_sha1_dedupe.md b/notes/bulk_edits/2021-11-24_file_sha1_dedupe.md
new file mode 100644
index 00000000..012bcf62
--- /dev/null
+++ b/notes/bulk_edits/2021-11-24_file_sha1_dedupe.md
@@ -0,0 +1,35 @@
+
+See notes and scripts about `file_sha1_dedupe` cleanup for prep details.
+
+## Prod Run
+
+Run as `cleanup-bot`:
+
+    export FATCAT_AUTH_API_TOKEN=[...]
+
+    git log | head -n1
+    # commit 5bc5eeed5e3ba54c2129c4233b881291c5fa7449
+
+First do a sample in dry-run mode:
+
+    head -n25 /srv/fatcat/datasets/file_sha1_dupes.json \
+        | python -m fatcat_tools.mergers.files --editgroup-description-override "Automated merging of file entities with duplicate SHA-1 hashes" --dry-run merge-files -
+    # Counter({'updated-entities': 59, 'lines': 25, 'merged': 25, 'skip': 0, 'updated-total': 0})
+
+Gah, the dry-run mode still creates (empty) editgroups:
+
+    https://fatcat.wiki/editgroup/iqzjg3vxu5elvotknmmjln3gv4
+    https://fatcat.wiki/editgroup/2mxsl7lxo5dezem42whnr7zxxe
+
+Actually run (merge) the sample:
+
+    head -n25 /srv/fatcat/datasets/file_sha1_dupes.json \
+        | python -m fatcat_tools.mergers.files --editgroup-description-override "Automated merging of file entities with duplicate SHA-1 hashes" merge-files -
+    # Counter({'updated-entities': 59, 'lines': 25, 'merged': 25, 'skip': 0, 'updated-total': 0})
+
+
+Run the full batch:
+
+    cat /srv/fatcat/datasets/file_sha1_dupes.json \
+        | python -m fatcat_tools.mergers.files --editgroup-description-override "Automated merging of file entities with duplicate SHA-1 hashes" merge-files -
+    # Counter({'updated-entities': 6197, 'lines': 2039, 'merged': 2014, 'skip': 25, 'skip-not-active-entity': 25, 'updated-total': 0})
diff --git a/notes/bulk_edits/CHANGELOG.md b/notes/bulk_edits/CHANGELOG.md
index d82e126e..cf5d9829 100644
--- a/notes/bulk_edits/CHANGELOG.md
+++ b/notes/bulk_edits/CHANGELOG.md
@@ -25,6 +25,7 @@ and specific final commands in this directory. Quick summary:
   import-time code bug, were fixed. a couple hundred questionable cases remain,
   but are all mismatched due to DOI slash/double-slash issues and will not be
   fixed in an automated way.
+- de-uplicated a few thousand file entities, on the basis of SHA-1 hash
 
 
 ## 2021-06