aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
-rw-r--r--proposals/2021-09-09_dataset_ingest.md239
-rw-r--r--proposals/2021-09-09_fileset_ingest.md337
2 files changed, 337 insertions, 239 deletions
diff --git a/proposals/2021-09-09_dataset_ingest.md b/proposals/2021-09-09_dataset_ingest.md
deleted file mode 100644
index cbfeb68..0000000
--- a/proposals/2021-09-09_dataset_ingest.md
+++ /dev/null
@@ -1,239 +0,0 @@
-
-Dataset Ingest Pipeline
-=======================
-
-Sandcrawler currently has ingest support for individual files saved as `file`
-entities in fatcat (xml and pdf ingest types) and HTML files with
-sub-components saved as `webcapture` entities in fatcat (html ingest type).
-
-This document describes extensions to this ingest system to flexibly support
-groups of files, which may be represented in fatcat as `fileset` entities. The
-new ingest type is `dataset`.
-
-Compared to the existing ingest process, there are two major complications with
-datasets:
-
-- the ingest process often requires more than parsing HTML files, and will be
- specific to individual platforms and host software packages
-- the storage backend and fatcat entity type is flexible: a dataset might be
- represented by a single file, multiple files combined in to a single .zip
- file, or mulitple separate files; the data may get archived in wayback or in
- an archive.org item
-
-The new concepts of "strategy" and "platform" are introduced to accomodate
-these complications.
-
-
-## Ingest Strategies
-
-The ingest strategy describes the fatcat entity type that will be output; the
-storage backend used; and whether an enclosing file format is used. The
-strategy to use can not be determined until the number and size of files is
-known. It is a function of file count, total file size, and platform.
-
-Strategy names are compact strings with the format
-`{storage_backend}-{fatcat_entity}`. A `-bundled` suffix after a `fileset`
-entity type indicates that metadata about multiple files is retained, but that
-in the storage backend only a single enclosing file (eg, `.zip`) will be
-stored.
-
-The supported strategies are:
-
-- `web-file`: single file of any type, stored in wayback, represented as fatcat `file`
-- `web-fileset`: multiple files of any type, stored in wayback, represented as fatcat `fileset`
-- `web-fileset-bundled`: single bundle file, stored in wayback, represented as fatcat `fileset`
-- `archiveorg-file`: single file of any type, stored in archive.org item, represented as fatcat `file`
-- `archiveorg-fileset`: multiple files of any type, stored in archive.org item, represented as fatcat `fileset`
-- `archiveorg-fileset-bundled`: single bundle file, stored in archive.org item, represented as fatcat `fileset`
-
-"Bundle" files are things like .zip or .tar.gz. Not all .zip files are handled
-as bundles! Only when the transfer from the hosting platform is via a "download
-all as .zip" (or similar) do we consider a zipfile a "bundle" and index the
-interior files as a fileset.
-
-The term "bundle file" is used over "archive file" or "container file" to
-prevent confusion with the other use of those terms in the context of fatcat
-(container entities; archive; Internet Archive as an organiztion).
-
-The motivation for supporting both `web` and `archiveorg` is that `web` is
-somewhat simpler for small files, but `archiveorg` is better for larger groups
-of files (say more than 20) and larger total size (say more than 1 GByte total,
-or 128 MByte for any one file).
-
-The motivation for supporting "bundled" filesets is that there is only a single
-file to archive.
-
-
-## Ingest Pseudocode
-
-1. Determine `platform`, which may involve resolving redirects and crawling a landing page.
-
- a. TODO: do we always try crawling `base_url`? would simplify code flow, but results in extra SPN calls (slow). start with yes, always
- b. TODO: what if we trivially crawl directly to a non-HTML file? Bypass most of the below? `direct-file` strategy?
- c. `infer_platform(request, terminal_url, html_biblio)`
-
-2. Use platform-specific methods to fetch manifest metadata and decide on an `ingest_strategy`.
-
-3. Use strategy-specific methods to archive all files in platform manifest, and verify manifest metadata.
-
-4. Summarize status and return structured result metadata.
-
-Python APIs, as abstract classes (TODO):
-
- PlatformDatasetContext
- platform_name
- platform_domain
- platform_id
- manifest
- archiveorg_metadata
- web_base_url
- DatasetPlatformHelper
- match_request(request: Request, resource: Resource, html_biblio: Optional[BiblioMetadata]) -> bool
- process_request(?) -> ?
- StrategyArchiver
- process(manifest, archiveorg_metadata, web_metadata) -> ?
- check_existing(?) -> ?
-
-
-## New Sandcrawler Code and Worker
-
- sandcrawler-ingest-fileset-worker@{1..12}
-
-Worker consumes from ingest request topic, produces to fileset ingest results,
-and optionally produces to file ingest results.
-
- sandcrawler-persist-ingest-fileset-worker@1
-
-Simply writes fileset ingest rows in to SQL.
-
-## New Fatcat Worker and Code Changes
-
- fatcat-import-ingest-fileset-worker
-
-This importer should be modeled on file and web worker. Filters for `success`
-with strategy of `*-fileset*`.
-
-Existing `fatcat-import-ingest-file-worker` should be updated to allow
-`dataset` single-file imports, with largely same behavior and semantics as
-current importer.
-
-TODO: Existing fatcat transforms, and possibly even elasticsearch schemas,
-should be updated to include fileset status and `in_ia` flag for dataset type
-releases.
-
-TODO: Existing entity updates worker submits `dataset` type ingests to ingest
-request topic.
-
-
-## New SQL Tables
-
- CREATE TABLE IF NOT EXISTS ingest_fileset_result (
- ingest_type TEXT NOT NULL CHECK (octet_length(ingest_type) >= 1),
- base_url TEXT NOT NULL CHECK (octet_length(base_url) >= 1),
- updated TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL,
- hit BOOLEAN NOT NULL,
- status TEXT CHECK (octet_length(status) >= 1),
-
- terminal_url TEXT CHECK (octet_length(terminal_url) >= 1),
- terminal_dt TEXT CHECK (octet_length(terminal_dt) = 14),
- terminal_status_code INT,
- terminal_sha1hex TEXT CHECK (octet_length(terminal_sha1hex) = 40),
-
- platform TEXT CHECK (octet_length(platform) >= 1),
- platform_domain TEXT CHECK (octet_length(platform_domain) >= 1),
- platform_id TEXT CHECK (octet_length(platform_id) >= 1),
- ingest_strategy TEXT CHECK (octet_length(ingest_strategy) >= 1),
- total_size BIGINT,
- file_count INT,
- item_name TEXT CHECK (octet_length(item_name) >= 1),
- item_bundle_path TEXT CHECK (octet_length(item_path_bundle) >= 1),
-
- manifest JSONB,
- -- list, similar to fatcat fileset manifest, plus extra:
- -- status (str)
- -- path (str)
- -- size (int)
- -- md5 (str)
- -- sha1 (str)
- -- sha256 (str)
- -- mimetype (str)
- -- platform_url (str)
- -- terminal_url (str)
- -- terminal_dt (str)
- -- extra (dict) (?)
-
- PRIMARY KEY (ingest_type, base_url)
- );
- CREATE INDEX ingest_fileset_result_terminal_url_idx ON ingest_fileset_result(terminal_url);
-
-
-## New Kafka Topic and JSON Schema
-
-
- sandcrawler-ENV.ingest-fileset-results 6x, no retention limit
-
-
-## Implementation Plan
-
-First implement ingest worker, including platform and strategy helpers, and
-test those as simple stdin/stdout CLI tools in sandcrawler repo to validate
-this proposal.
-
-Second implement fatcat importer and test locally and/or in QA.
-
-Lastly implement infrastructure, automation, and other "glue".
-
-
-## Example Entities
-
-### ArchiveOrg: CAT dataset
-
-<https://archive.org/details/CAT_DATASET>
-
-`release_36vy7s5gtba67fmyxlmijpsaui`
-
-###
-
-<https://archive.org/details/academictorrents_70e0794e2292fc051a13f05ea6f5b6c16f3d3635>
-
-doi:10.1371/journal.pone.0120448
-
-Single .rar file
-
-### Dataverse
-
-<https://dataverse.rsu.lv/dataset.xhtml?persistentId=doi:10.48510/FK2/IJO02B>
-
-Single excel file
-
-### Dataverse
-
-<https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/CLSFKX&version=1.1>
-
-doi:10.7910/DVN/CLSFKX
-
-Mulitple files; multiple versions?
-
-API fetch: <https://dataverse.harvard.edu/api/datasets/:persistentId/?persistentId=doi:10.7910/DVN/CLSFKX&version=1.1>
-
- .data.id
- .data.latestVersion.datasetPersistentId
- .data.latestVersion.versionNumber, .versionMinorNumber
- .data.latestVersion.files[]
- .dataFile
- .contentType (mimetype)
- .filename
- .filesize (int, bytes)
- .md5
- .persistendId
- .description
- .label (filename?)
- .version
-
-Single file inside: <https://dataverse.harvard.edu/file.xhtml?persistentId=doi:10.7910/DVN/CLSFKX/XWEHBB>
-
-Download single file: <https://dataverse.harvard.edu/api/access/datafile/:persistentId/?persistentId=doi:10.7910/DVN/CLSFKX/XWEHBB> (redirects to AWS S3)
-
-Dataverse refs:
-- 'doi' and 'hdl' are the two persistentId styles
-- file-level persistentIds are optional, on a per-instance basis: https://guides.dataverse.org/en/latest/installation/config.html#filepidsenabled
diff --git a/proposals/2021-09-09_fileset_ingest.md b/proposals/2021-09-09_fileset_ingest.md
new file mode 100644
index 0000000..bb9d358
--- /dev/null
+++ b/proposals/2021-09-09_fileset_ingest.md
@@ -0,0 +1,337 @@
+
+status: implemented
+
+Fileset Ingest Pipeline (for Datasets)
+======================================
+
+Sandcrawler currently has ingest support for individual files saved as `file`
+entities in fatcat (xml and pdf ingest types) and HTML files with
+sub-components saved as `webcapture` entities in fatcat (html ingest type).
+
+This document describes extensions to this ingest system to flexibly support
+groups of files, which may be represented in fatcat as `fileset` entities. The
+main new ingest type is `dataset`.
+
+Compared to the existing ingest process, there are two major complications with
+datasets:
+
+- the ingest process often requires more than parsing HTML files, and will be
+ specific to individual platforms and host software packages
+- the storage backend and fatcat entity type is flexible: a dataset might be
+ represented by a single file, multiple files combined in to a single .zip
+ file, or mulitple separate files; the data may get archived in wayback or in
+ an archive.org item
+
+The new concepts of "strategy" and "platform" are introduced to accomodate
+these complications.
+
+
+## Ingest Strategies
+
+The ingest strategy describes the fatcat entity type that will be output; the
+storage backend used; and whether an enclosing file format is used. The
+strategy to use can not be determined until the number and size of files is
+known. It is a function of file count, total file size, and publication
+platform.
+
+Strategy names are compact strings with the format
+`{storage_backend}-{fatcat_entity}`. A `-bundled` suffix after a `fileset`
+entity type indicates that metadata about multiple files is retained, but that
+in the storage backend only a single enclosing file (eg, `.zip`) will be
+stored.
+
+The supported strategies are:
+
+- `web-file`: single file of any type, stored in wayback, represented as fatcat `file`
+- `web-fileset`: multiple files of any type, stored in wayback, represented as fatcat `fileset`
+- `web-fileset-bundled`: single bundle file, stored in wayback, represented as fatcat `fileset`
+- `archiveorg-file`: single file of any type, stored in archive.org item, represented as fatcat `file`
+- `archiveorg-fileset`: multiple files of any type, stored in archive.org item, represented as fatcat `fileset`
+- `archiveorg-fileset-bundled`: single bundle file, stored in archive.org item, represented as fatcat `fileset`
+
+"Bundle" or "enclosing" files are things like .zip or .tar.gz. Not all .zip
+files are handled as bundles! Only when the transfer from the hosting platform
+is via a "download all as .zip" (or similar) do we consider a zipfile a
+"bundle" and index the interior files as a fileset.
+
+The term "bundle file" is used over "archive file" or "container file" to
+prevent confusion with the other use of those terms in the context of fatcat
+(container entities; archive; Internet Archive as an organiztion).
+
+The motivation for supporting both `web` and `archiveorg` is that `web` is
+somewhat simpler for small files, but `archiveorg` is better for larger groups
+of files (say more than 20) and larger total size (say more than 1 GByte total,
+or 128 MByte for any one file).
+
+The motivation for supporting "bundled" filesets is that there is only a single
+file to archive.
+
+
+## Ingest Pseudocode
+
+1. Determine `platform`, which may involve resolving redirects and crawling a landing page.
+
+ a. currently we always crawl the ingest `base_url`, capturing a platform landing page
+ b. we don't currently handle the case of `base_url` leading to a non-HTML
+ terminal resource. the `component` ingest type does handle this
+
+2. Use platform-specific methods to fetch manifest metadata and decide on an `ingest_strategy`.
+
+ a. depending on platform, may include access URLs for multiple strategies
+ (eg, URL for each file and a bundle URL), metadata about the item for, eg,
+ archive.org item upload, etc
+
+3. Use strategy-specific methods to archive all files in platform manifest, and verify manifest metadata.
+
+4. Summarize status and return structured result metadata.
+
+ a. if the strategy was `web-file` or `archiveorg-file`, potentially submit an
+ `ingest_file_result` object down the file ingest pipeline (Kafka topic and
+ later persist and fatcat import workers), with `dataset-file` ingest
+ type (or `{ingest_type}-file` more generally).
+
+New python types:
+
+ FilesetManifestFile
+ path: str
+ size: Optional[int]
+ md5: Optional[str]
+ sha1: Optional[str]
+ sha256: Optional[str]
+ mimetype: Optional[str]
+ extra: Optional[Dict[str, Any]]
+
+ status: Optional[str]
+ platform_url: Optional[str]
+ terminal_url: Optional[str]
+ terminal_dt: Optional[str]
+
+ FilesetPlatformItem
+ platform_name: str
+ platform_status: str
+ platform_domain: Optional[str]
+ platform_id: Optional[str]
+ manifest: Optional[List[FilesetManifestFile]]
+ archiveorg_item_name: Optional[str]
+ archiveorg_item_meta
+ web_base_url
+ web_bundle_url
+
+ ArchiveStrategyResult
+ ingest_strategy: str
+ status: str
+ manifest: List[FilesetManifestFile]
+
+ FilesetIngestResult
+ ingest_strategy: str
+ status: str
+ manifest: List[FilesetManifestFile]
+ single_file_meta: Optional[dict]
+ single_terminal: Optional[dict]
+ single_cdx: Optional[dict]
+ bundle_file_meta: Optional[dict]
+ bundle_terminal: Optional[dict]
+ bundle_cdx: Optional[dict]
+ bundle_archiveorg_path: Optional[dict]
+
+New python APIs/classes:
+
+ FilesetPlatformHelper
+ match_request(request, resource, html_biblio) -> bool
+ does the request and landing page metadata indicate a match for this platform?
+ process_request(request, resource, html_biblio) -> FilesetPlatformItem
+ do API requests, parsing, etc to fetch metadata and access URLs for this fileset/dataset. platform-specific
+ chose_strategy(item: FilesetPlatformItem) -> IngestStrategy
+ select an archive strategy for the given fileset/dataset
+
+ FilesetIngestStrategy
+ check_existing(item: FilesetPlatformItem) -> Optional[ArchiveStrategyResult]
+ check the given backend for an existing capture/archive; if found, return result
+ process(item: FilesetPlatformItem) -> ArchiveStrategyResult
+ perform an actual archival capture
+
+## Limits and Failure Modes
+
+- `too-large-size`: total size of the fileset is too large for archiving.
+ initial limit is 64 GBytes, controlled by `max_total_size` parameter.
+- `too-many-files`: number of files (and thus file-level metadata) is too
+ large. initial limit is 200, controlled by `max_file_count` parameter.
+- `platform-scope / FilesetPlatformScopeError`: for when `base_url` leads to a
+ valid platform, which could be found via API or parsing, but has the wrong
+ scope. Eg, tried to fetch a dataset, but got a DOI which represents all
+ versions of the dataset, not a specific version.
+
+
+## New Sandcrawler Code and Worker
+
+ sandcrawler-ingest-fileset-worker@{1..6} (or up to 1..12 later)
+
+Worker consumes from ingest request topic, produces to fileset ingest results,
+and optionally produces to file ingest results.
+
+ sandcrawler-persist-ingest-fileset-worker@1
+
+Simply writes fileset ingest rows to SQL.
+
+
+## New Fatcat Worker and Code Changes
+
+ fatcat-import-ingest-fileset-worker
+
+This importer is modeled on file and web worker. Filters for `success` with
+strategy of `*-fileset*`.
+
+Existing `fatcat-import-ingest-file-worker` should be updated to allow
+`dataset` single-file imports, with largely same behavior and semantics as
+current importer (`component` mode).
+
+Existing fatcat transforms, and possibly even elasticsearch schemas, should be
+updated to include fileset status and `in_ia` flag for dataset type releases.
+
+Existing entity updates worker submits `dataset` type ingests to ingest request
+topic.
+
+
+## Ingest Result Schema
+
+Common with file results, and mostly relating to landing page HTML:
+
+ hit: bool
+ status: str
+ success
+ success-existing
+ success-file (for `web-file` or `archiveorg-file` only)
+ request: object
+ terminal: object
+ file_meta: object
+ cdx: object
+ revisit_cdx: object
+ html_biblio: object
+
+Additional fileset-specific fields:
+
+ manifest: list of objects
+ platform_name: str
+ platform_domain: str
+ platform_id: str
+ ingest_strategy: str
+ archiveorg_item_name: str (optional, only for `archiveorg-*` strategies)
+ fileset_bundle (optional, only for `*-fileset-bundle` strategy)
+ archiveorg_bundle_path
+ file_meta
+ cdx
+ terminal
+ fileset_file (optional, only for `*-file` strategy)
+ file_meta
+ terminal
+ cdx
+ revisit_cdx
+
+If the strategy was `web-file` or `archiveorg-file` and the status is
+`success-file`, then an ingest file result will also be published to
+`sandcrawler-ENV.ingest-file-results`, using the same ingest type and fields as
+regular ingest.
+
+
+All fileset ingest results get published to ingest-fileset-result.
+
+Existing sandcrawler persist workers also subscribe to this topic and persist
+status and landing page terminal info to tables just like with file ingest.
+GROBID, HTML, and other metadata is not persisted in this path.
+
+If the ingest strategy was a single file (`*-file`), then an ingest file is
+also published to the ingest-file-result topic, with the `fileset_file`
+metadata, and ingest type `dataset-file`. This should only happen on success
+condition.
+
+
+## New SQL Tables
+
+ CREATE TABLE IF NOT EXISTS ingest_fileset_platform (
+ ingest_type TEXT NOT NULL CHECK (octet_length(ingest_type) >= 1),
+ base_url TEXT NOT NULL CHECK (octet_length(base_url) >= 1),
+ updated TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL,
+ hit BOOLEAN NOT NULL,
+ status TEXT CHECK (octet_length(status) >= 1),
+
+ platform_name TEXT CHECK (octet_length(platform) >= 1),
+ platform_domain TEXT CHECK (octet_length(platform_domain) >= 1),
+ platform_id TEXT CHECK (octet_length(platform_id) >= 1),
+ ingest_strategy TEXT CHECK (octet_length(ingest_strategy) >= 1),
+ total_size BIGINT,
+ file_count INT,
+ archiveorg_item_name TEXT CHECK (octet_length(item_name) >= 1),
+
+ archiveorg_item_bundle_path TEXT CHECK (octet_length(item_path_bundle) >= 1),
+ web_bundle_url TEXT CHECK (octet_length(terminal_url) >= 1),
+ web_bundle_dt TEXT CHECK (octet_length(terminal_dt) = 14),
+
+ manifest JSONB,
+ -- list, similar to fatcat fileset manifest, plus extra:
+ -- status (str)
+ -- path (str)
+ -- size (int)
+ -- md5 (str)
+ -- sha1 (str)
+ -- sha256 (str)
+ -- mimetype (str)
+ -- extra (dict)
+ -- platform_url (str)
+ -- terminal_url (str)
+ -- terminal_dt (str)
+
+ PRIMARY KEY (ingest_type, base_url)
+ );
+ CREATE INDEX ingest_fileset_result_terminal_url_idx ON ingest_fileset_result(terminal_url);
+ # TODO: index on (platform_name,platform_domain,platform_id) ?
+
+
+## New Kafka Topic
+
+ sandcrawler-ENV.ingest-fileset-results 6x, no retention limit
+
+
+## Implementation Plan
+
+First implement ingest worker, including platform and strategy helpers, and
+test those as simple stdin/stdout CLI tools in sandcrawler repo to validate
+this proposal.
+
+Second implement fatcat importer and test locally and/or in QA.
+
+Lastly implement infrastructure, automation, and other "glue":
+
+- SQL schema
+- persist worker
+
+
+## Design Note: Single-File Datasets
+
+Should datasets and other groups of files which only contain a single file get
+imported as a fatcat `file` or `fileset`? This can be broken down further as
+documents (single PDF) vs other individual files.
+
+Advantages of `file`:
+
+- handles case of article PDFs being marked as dataset accidentally
+- `file` entities get de-duplicated with simple lookup (eg, on `sha1`)
+- conceptually simpler if individual files are `file` entity
+- easier to download individual files
+
+Advantages of `fileset`:
+
+- conceptually simpler if all `dataset` entities have `fileset` form factor
+- code path is simpler: one fewer strategy, and less complexity of sending
+ files down separate import path
+- metadata about platform is retained
+- would require no modification of existing fatcat file importer
+- fatcat import of archive.org of `file` is not actually implemented yet?
+
+Decision is to do individual files. Fatcat fileset import worker should reject
+single-file (and empty) manifest filesets. Fatcat file import worker should
+accept all mimetypes for `dataset-file` (similar to `component`).
+
+
+## Example Entities
+
+See `notes/dataset_examples.txt`