path: root/proposals/2021-09-09_fileset_ingest.md

status: implemented

Fileset Ingest Pipeline (for Datasets)
======================================

Sandcrawler currently has ingest support for individual files saved as `file`
entities in fatcat (xml and pdf ingest types) and HTML files with
sub-components saved as `webcapture` entities in fatcat (html ingest type).

This document describes extensions to this ingest system to flexibly support
groups of files, which may be represented in fatcat as `fileset` entities. The
main new ingest type is `dataset`.

Compared to the existing ingest process, there are two major complications with
datasets:

- the ingest process often requires more than parsing HTML files, and will be
  specific to individual platforms and host software packages
- the storage backend and fatcat entity type is flexible: a dataset might be
  represented by a single file, multiple files combined in to a single .zip
  file, or multiple separate files; the data may get archived in wayback or in
  an archive.org item

The new concepts of "strategy" and "platform" are introduced to accommodate
these complications.


## Ingest Strategies

The ingest strategy describes the fatcat entity type that will be output; the
storage backend used; and whether an enclosing file format is used. The
strategy to use can not be determined until the number and size of files is
known. It is a function of file count, total file size, and publication
platform.

Strategy names are compact strings with the format
`{storage_backend}-{fatcat_entity}`. A `-bundled` suffix after a `fileset`
entity type indicates that metadata about multiple files is retained, but that
in the storage backend only a single enclosing file (eg, `.zip`) will be
stored.

The supported strategies are:

- `web-file`: single file of any type, stored in wayback, represented as fatcat `file`
- `web-fileset`: multiple files of any type, stored in wayback, represented as fatcat `fileset`
- `web-fileset-bundled`: single bundle file, stored in wayback, represented as fatcat `fileset`
- `archiveorg-file`: single file of any type, stored in archive.org item, represented as fatcat `file`
- `archiveorg-fileset`: multiple files of any type, stored in archive.org item, represented as fatcat `fileset`
- `archiveorg-fileset-bundled`: single bundle file, stored in archive.org item, represented as fatcat `fileset`

"Bundle" or "enclosing" files are things like .zip or .tar.gz. Not all .zip
files are handled as bundles! Only when the transfer from the hosting platform
is via a "download all as .zip" (or similar) do we consider a zipfile a
"bundle" and index the interior files as a fileset.

The term "bundle file" is used over "archive file" or "container file" to
prevent confusion with the other use of those terms in the context of fatcat
(container entities; archive; Internet Archive as an organization).

The motivation for supporting both `web` and `archiveorg` is that `web` is
somewhat simpler for small files, but `archiveorg` is better for larger groups
of files (say more than 20) and larger total size (say more than 1 GByte total,
or 128 MByte for any one file).

The motivation for supporting "bundled" filesets is that there is only a single
file to archive.


## Ingest Pseudocode

1. Determine `platform`, which may involve resolving redirects and crawling a landing page.

  a. currently we always crawl the ingest `base_url`, capturing a platform landing page
  b. we don't currently handle the case of `base_url` leading to a non-HTML
     terminal resource. the `component` ingest type does handle this

2. Use platform-specific methods to fetch manifest metadata and decide on an `ingest_strategy`.

  a. depending on platform, may include access URLs for multiple strategies
     (eg, URL for each file and a bundle URL), metadata about the item for, eg,
     archive.org item upload, etc

3. Use strategy-specific methods to archive all files in platform manifest, and verify manifest metadata.

4. Summarize status and return structured result metadata.

  a. if the strategy was `web-file` or `archiveorg-file`, potentially submit an
  `ingest_file_result` object down the file ingest pipeline (Kafka topic and
  later persist and fatcat import workers), with `dataset-file` ingest
  type (or `{ingest_type}-file` more generally).

New python types:

    FilesetManifestFile
        path: str
        size: Optional[int]
        md5: Optional[str]
        sha1: Optional[str]
        sha256: Optional[str]
        mimetype: Optional[str]
        extra: Optional[Dict[str, Any]]

        status: Optional[str]
        platform_url: Optional[str]
        terminal_url: Optional[str]
        terminal_dt: Optional[str]

    FilesetPlatformItem
        platform_name: str
        platform_status: str
        platform_domain: Optional[str]
        platform_id: Optional[str]
        manifest: Optional[List[FilesetManifestFile]]
        archiveorg_item_name: Optional[str]
        archiveorg_item_meta
        web_base_url
        web_bundle_url

    ArchiveStrategyResult
        ingest_strategy: str
        status: str
        manifest: List[FilesetManifestFile]
        file_file_meta: Optional[dict]
        file_terminal: Optional[dict]
        file_cdx: Optional[dict]
        bundle_file_meta: Optional[dict]
        bundle_terminal: Optional[dict]
        bundle_cdx: Optional[dict]
        bundle_archiveorg_path: Optional[dict]

New python APIs/classes:

    FilesetPlatformHelper
        match_request(request, resource, html_biblio) -> bool
            does the request and landing page metadata indicate a match for this platform?
        process_request(request, resource, html_biblio) -> FilesetPlatformItem
            do API requests, parsing, etc to fetch metadata and access URLs for this fileset/dataset. platform-specific
        chose_strategy(item: FilesetPlatformItem) -> IngestStrategy
            select an archive strategy for the given fileset/dataset

    FilesetIngestStrategy
        check_existing(item: FilesetPlatformItem) -> Optional[ArchiveStrategyResult]
            check the given backend for an existing capture/archive; if found, return result
        process(item: FilesetPlatformItem) -> ArchiveStrategyResult
            perform an actual archival capture

## Limits and Failure Modes

- `too-large-size`: total size of the fileset is too large for archiving.
  initial limit is 64 GBytes, controlled by `max_total_size` parameter.
- `too-many-files`: number of files (and thus file-level metadata) is too
  large. initial limit is 200, controlled by `max_file_count` parameter.
- `platform-scope / FilesetPlatformScopeError`: for when `base_url` leads to a
  valid platform, which could be found via API or parsing, but has the wrong
  scope. Eg, tried to fetch a dataset, but got a DOI which represents all
  versions of the dataset, not a specific version.
- `platform-restricted`/`PlatformRestrictedError`: for, eg, embargoes 
- `platform-404`: got to a landing page, and seemed like in-scope, but no
  platform record found anyways


## New Sandcrawler Code and Worker

    sandcrawler-ingest-fileset-worker@{1..6}  (or up to 1..12 later)

Worker consumes from ingest request topic, produces to fileset ingest results,
and optionally produces to file ingest results.

    sandcrawler-persist-ingest-fileset-worker@1

Simply writes fileset ingest rows to SQL.


## New Fatcat Worker and Code Changes

    fatcat-import-ingest-fileset-worker

This importer is modeled on file and web worker. Filters for `success` with
strategy of `*-fileset*`.

Existing `fatcat-import-ingest-file-worker` should be updated to allow
`dataset` single-file imports, with largely same behavior and semantics as
current importer (`component` mode).

Existing fatcat transforms, and possibly even elasticsearch schemas, should be
updated to include fileset status and `in_ia` flag for dataset type releases.

Existing entity updates worker submits `dataset` type ingests to ingest request
topic.


## Ingest Result Schema

Common with file results, and mostly relating to landing page HTML:

    hit: bool
    status: str
        success
        success-existing
        success-file (for `web-file` or `archiveorg-file` only)
    request: object
    terminal: object
    file_meta: object
    cdx: object
    revisit_cdx: object
    html_biblio: object

Additional fileset-specific fields:

    manifest: list of objects
    platform_name: str
    platform_domain: str
    platform_id: str
    platform_base_url: str
    ingest_strategy: str
    archiveorg_item_name: str (optional, only for `archiveorg-*` strategies)
    file_count: int
    total_size: int
    fileset_bundle (optional, only for `*-fileset-bundle` strategy)
        file_meta
        cdx
        revisit_cdx
        terminal
        archiveorg_bundle_path
    fileset_file (optional, only for `*-file` strategy)
        file_meta
        terminal
        cdx
        revisit_cdx

If the strategy was `web-file` or `archiveorg-file` and the status is
`success-file`, then an ingest file result will also be published to
`sandcrawler-ENV.ingest-file-results`, using the same ingest type and fields as
regular ingest.


All fileset ingest results get published to ingest-fileset-result.

Existing sandcrawler persist workers also subscribe to this topic and persist
status and landing page terminal info to tables just like with file ingest.
GROBID, HTML, and other metadata is not persisted in this path.

If the ingest strategy was a single file (`*-file`), then an ingest file is
also published to the ingest-file-result topic, with the `fileset_file`
metadata, and ingest type `dataset-file`. This should only happen on success
condition.


## New SQL Tables

Note that this table *complements* `ingest_file_result`, doesn't replace it.
`ingest_file_result` could more accurately be called `ingest_result`.

    CREATE TABLE IF NOT EXISTS ingest_fileset_platform (
        ingest_type             TEXT NOT NULL CHECK (octet_length(ingest_type) >= 1),
        base_url                TEXT NOT NULL CHECK (octet_length(base_url) >= 1),
        updated                 TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL,
        hit                     BOOLEAN NOT NULL,
        status                  TEXT CHECK (octet_length(status) >= 1),

        platform_name           TEXT NOT NULL CHECK (octet_length(platform_name) >= 1),
        platform_domain         TEXT NOT NULL CHECK (octet_length(platform_domain) >= 1),
        platform_id             TEXT NOT NULL CHECK (octet_length(platform_id) >= 1),
        ingest_strategy         TEXT CHECK (octet_length(ingest_strategy) >= 1),
        total_size              BIGINT,
        file_count              BIGINT,
        archiveorg_item_name    TEXT CHECK (octet_length(archiveorg_item_name) >= 1),

        archiveorg_item_bundle_path TEXT CHECK (octet_length(archiveorg_item_bundle_path) >= 1),
        web_bundle_url          TEXT CHECK (octet_length(web_bundle_url) >= 1),
        web_bundle_dt           TEXT CHECK (octet_length(web_bundle_dt) = 14),

        manifest                JSONB,
        -- list, similar to fatcat fileset manifest, plus extra:
        --   status (str)
        --   path (str)
        --   size (int)
        --   md5 (str)
        --   sha1 (str)
        --   sha256 (str)
        --   mimetype (str)
        --   extra (dict)
        --   platform_url (str)
        --   terminal_url (str)
        --   terminal_dt (str)

        PRIMARY KEY (ingest_type, base_url)
    );
    CREATE INDEX ingest_fileset_platform_name_domain_id_idx ON ingest_fileset_platform(platform_name, platform_domain, platform_id);

Persist worker should only insert in to this table if `platform_name` is
identified.

## New Kafka Topic

    sandcrawler-ENV.ingest-fileset-results 6x, no retention limit


## Implementation Plan

First implement ingest worker, including platform and strategy helpers, and
test those as simple stdin/stdout CLI tools in sandcrawler repo to validate
this proposal.

Second implement fatcat importer and test locally and/or in QA.

Lastly implement infrastructure, automation, and other "glue":

- SQL schema
- persist worker


## Design Note: Single-File Datasets

Should datasets and other groups of files which only contain a single file get
imported as a fatcat `file` or `fileset`? This can be broken down further as
documents (single PDF) vs other individual files.

Advantages of `file`:

- handles case of article PDFs being marked as dataset accidentally
- `file` entities get de-duplicated with simple lookup (eg, on `sha1`)
- conceptually simpler if individual files are `file` entity
- easier to download individual files

Advantages of `fileset`:

- conceptually simpler if all `dataset` entities have `fileset` form factor
- code path is simpler: one fewer strategy, and less complexity of sending
  files down separate import path
- metadata about platform is retained
- would require no modification of existing fatcat file importer
- fatcat import of archive.org of `file` is not actually implemented yet?

Decision is to do individual files. Fatcat fileset import worker should reject
single-file (and empty) manifest filesets. Fatcat file import worker should
accept all mimetypes for `dataset-file` (similar to `component`).


## Example Entities

See `notes/dataset_examples.txt`