status: implemented

Entity Mergers
===============

One category of type of catalog metadata cleanup is merging multiple duplicate
entries into a single record. The fatcat catalog allows this via during the
duplicate entities into "redirect reccords" which point at the single merged
record.

This proposal briefly describes the process for doing bulk merges.


## External Identifier Duplicates

The easiest category of entity duplicates to discover is cases where multiple
entities have the same external (persistent) identifier. For example, releases
with the exact same DOI, containers with the same ISSN-L, or creators with the
same ORCiD. Files with the same SHA-1 hash is a similar issue. The catalog does
not block the creation of such entities, though it is assumed that editors and
bots will do their best to prevent creating duplicates, and that this is
checked and monitored via review bots (auto-annotation) and bulk quality
checks.

In these cases, it is simple enough to use the external identifier dumps (part
of the fatcat bulk exports), find duplicates by identifier, and create merge
requests.


## Merge Requests JSON Schema

Proposed JSON schema for bulk entity merging:

    entity_type: str, required. eg: "file"
    primary_id: str, optional, entity ident
    duplicate_ids: [str], required, entity idents
    evidence: dict, optional, merger/entity specific
        # evidence fields for external identifier dupes
        extid: str, the identifier value
        extid_type: str, eg "doi" or "sha1"

The merge request generation process might indicate which of the entities
should end up as the "primary", or it might leave that determination to the
merger itself. `primary_id` should not be set arbitrarily or randomly if there
is not a good reason for a specific entity to be the "primary" which others
redirect to.

The `primary_id` should not be included in `duplicate_ids`, but the merger code
will remove it if included accidentally.

The `evidence` fields are flexible. By default they will all be included as
top-level "edit extra" metadata on each individual entity redirected, but not
on the primary entity (if it gets updated).


## Merge Process and Semantics

The assumption is that all the entities indicated in `duplicate_ids` will be
redirected to the `primary_id`. Any metadata included in the duplicates which
is not included in the primary will be copied in to the primary, but existing
primary metadata fields will not be "clobbered" (overwritten) by duplicate
metadata. This includes top-level fields of the `extra` metadata dict, if
appropriate. If there is no unique metadata in the redirected entities, the
primary does not need to be updated and will not be.


## Work/Release Grouping and Merging

Work and Release entities are something of a special case.

Merging two release entities will result in all artifact entities (files,
filesets, webcaptures) being updated which previously pointed at the duplicate
entity to point to the primary entity. If the work entities associated with the
duplicate releases have no other releases associated with them, they also will
be merged (redirected) to the primary release's work entity.

"Grouping" releases is the same as merging their works. In this situation, the
number of distinct release entities stays the same, but the duplicates are
updated to be under the same work as the primary. This is initially implemented
by merging the work entites, and then updating *all* the releases under each
merged work towards the primary work identifier. No artifact entities need to
be updated in this scenario.

A currently planned option would be to pull a single release out of a group of
releases under a work, and point it to a new work. This would be a form of
"regrouping". For now this can only be achieved by updating the release
entities individually, not in a bulk/automated manner.


## Container Merging

Because many releases point to containers, it is not practical to update all
the releases at the same time as merging the containers. In the long run it is
good for the health of the catalog to have all the releases updated to point at
the the primary container, but these updates can be delayed.

To keep statistics and functionality working before release updates happen,
downstream users of release entities should "expand" container sub-entities and
use the "redirect" ident of the container entity instead of "ident", if the
"redirect" is set. For example, when linking in web interfaces, or when doing a
schema transform in the fatcat and scholar.archive.org search index.


## Background Reading

"The Lens MetaRecord and LensID: An open identifier system for aggregated
metadata and versioning of knowledge artefacts"
https://osf.io/preprints/lissa/t56yh/