diff options
| -rw-r--r-- | proposals/2021-09-09_dataset_ingest.md | 239 | ||||
| -rw-r--r-- | proposals/2021-09-09_fileset_ingest.md | 337 |
2 files changed, 337 insertions, 239 deletions
diff --git a/proposals/2021-09-09_dataset_ingest.md b/proposals/2021-09-09_dataset_ingest.md deleted file mode 100644 index cbfeb68..0000000 --- a/proposals/2021-09-09_dataset_ingest.md +++ /dev/null @@ -1,239 +0,0 @@ - -Dataset Ingest Pipeline -======================= - -Sandcrawler currently has ingest support for individual files saved as `file` -entities in fatcat (xml and pdf ingest types) and HTML files with -sub-components saved as `webcapture` entities in fatcat (html ingest type). - -This document describes extensions to this ingest system to flexibly support -groups of files, which may be represented in fatcat as `fileset` entities. The -new ingest type is `dataset`. - -Compared to the existing ingest process, there are two major complications with -datasets: - -- the ingest process often requires more than parsing HTML files, and will be - specific to individual platforms and host software packages -- the storage backend and fatcat entity type is flexible: a dataset might be - represented by a single file, multiple files combined in to a single .zip - file, or mulitple separate files; the data may get archived in wayback or in - an archive.org item - -The new concepts of "strategy" and "platform" are introduced to accomodate -these complications. - - -## Ingest Strategies - -The ingest strategy describes the fatcat entity type that will be output; the -storage backend used; and whether an enclosing file format is used. The -strategy to use can not be determined until the number and size of files is -known. It is a function of file count, total file size, and platform. - -Strategy names are compact strings with the format -`{storage_backend}-{fatcat_entity}`. A `-bundled` suffix after a `fileset` -entity type indicates that metadata about multiple files is retained, but that -in the storage backend only a single enclosing file (eg, `.zip`) will be -stored. - -The supported strategies are: - -- `web-file`: single file of any type, stored in wayback, represented as fatcat `file` -- `web-fileset`: multiple files of any type, stored in wayback, represented as fatcat `fileset` -- `web-fileset-bundled`: single bundle file, stored in wayback, represented as fatcat `fileset` -- `archiveorg-file`: single file of any type, stored in archive.org item, represented as fatcat `file` -- `archiveorg-fileset`: multiple files of any type, stored in archive.org item, represented as fatcat `fileset` -- `archiveorg-fileset-bundled`: single bundle file, stored in archive.org item, represented as fatcat `fileset` - -"Bundle" files are things like .zip or .tar.gz. Not all .zip files are handled -as bundles! Only when the transfer from the hosting platform is via a "download -all as .zip" (or similar) do we consider a zipfile a "bundle" and index the -interior files as a fileset. - -The term "bundle file" is used over "archive file" or "container file" to -prevent confusion with the other use of those terms in the context of fatcat -(container entities; archive; Internet Archive as an organiztion). - -The motivation for supporting both `web` and `archiveorg` is that `web` is -somewhat simpler for small files, but `archiveorg` is better for larger groups -of files (say more than 20) and larger total size (say more than 1 GByte total, -or 128 MByte for any one file). - -The motivation for supporting "bundled" filesets is that there is only a single -file to archive. - - -## Ingest Pseudocode - -1. Determine `platform`, which may involve resolving redirects and crawling a landing page. - - a. TODO: do we always try crawling `base_url`? would simplify code flow, but results in extra SPN calls (slow). start with yes, always - b. TODO: what if we trivially crawl directly to a non-HTML file? Bypass most of the below? `direct-file` strategy? - c. `infer_platform(request, terminal_url, html_biblio)` - -2. Use platform-specific methods to fetch manifest metadata and decide on an `ingest_strategy`. - -3. Use strategy-specific methods to archive all files in platform manifest, and verify manifest metadata. - -4. Summarize status and return structured result metadata. - -Python APIs, as abstract classes (TODO): - - PlatformDatasetContext - platform_name - platform_domain - platform_id - manifest - archiveorg_metadata - web_base_url - DatasetPlatformHelper - match_request(request: Request, resource: Resource, html_biblio: Optional[BiblioMetadata]) -> bool - process_request(?) -> ? - StrategyArchiver - process(manifest, archiveorg_metadata, web_metadata) -> ? - check_existing(?) -> ? - - -## New Sandcrawler Code and Worker - - sandcrawler-ingest-fileset-worker@{1..12} - -Worker consumes from ingest request topic, produces to fileset ingest results, -and optionally produces to file ingest results. - - sandcrawler-persist-ingest-fileset-worker@1 - -Simply writes fileset ingest rows in to SQL. - -## New Fatcat Worker and Code Changes - - fatcat-import-ingest-fileset-worker - -This importer should be modeled on file and web worker. Filters for `success` -with strategy of `*-fileset*`. - -Existing `fatcat-import-ingest-file-worker` should be updated to allow -`dataset` single-file imports, with largely same behavior and semantics as -current importer. - -TODO: Existing fatcat transforms, and possibly even elasticsearch schemas, -should be updated to include fileset status and `in_ia` flag for dataset type -releases. - -TODO: Existing entity updates worker submits `dataset` type ingests to ingest -request topic. - - -## New SQL Tables - - CREATE TABLE IF NOT EXISTS ingest_fileset_result ( - ingest_type TEXT NOT NULL CHECK (octet_length(ingest_type) >= 1), - base_url TEXT NOT NULL CHECK (octet_length(base_url) >= 1), - updated TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL, - hit BOOLEAN NOT NULL, - status TEXT CHECK (octet_length(status) >= 1), - - terminal_url TEXT CHECK (octet_length(terminal_url) >= 1), - terminal_dt TEXT CHECK (octet_length(terminal_dt) = 14), - terminal_status_code INT, - terminal_sha1hex TEXT CHECK (octet_length(terminal_sha1hex) = 40), - - platform TEXT CHECK (octet_length(platform) >= 1), - platform_domain TEXT CHECK (octet_length(platform_domain) >= 1), - platform_id TEXT CHECK (octet_length(platform_id) >= 1), - ingest_strategy TEXT CHECK (octet_length(ingest_strategy) >= 1), - total_size BIGINT, - file_count INT, - item_name TEXT CHECK (octet_length(item_name) >= 1), - item_bundle_path TEXT CHECK (octet_length(item_path_bundle) >= 1), - - manifest JSONB, - -- list, similar to fatcat fileset manifest, plus extra: - -- status (str) - -- path (str) - -- size (int) - -- md5 (str) - -- sha1 (str) - -- sha256 (str) - -- mimetype (str) - -- platform_url (str) - -- terminal_url (str) - -- terminal_dt (str) - -- extra (dict) (?) - - PRIMARY KEY (ingest_type, base_url) - ); - CREATE INDEX ingest_fileset_result_terminal_url_idx ON ingest_fileset_result(terminal_url); - - -## New Kafka Topic and JSON Schema - - - sandcrawler-ENV.ingest-fileset-results 6x, no retention limit - - -## Implementation Plan - -First implement ingest worker, including platform and strategy helpers, and -test those as simple stdin/stdout CLI tools in sandcrawler repo to validate -this proposal. - -Second implement fatcat importer and test locally and/or in QA. - -Lastly implement infrastructure, automation, and other "glue". - - -## Example Entities - -### ArchiveOrg: CAT dataset - -<https://archive.org/details/CAT_DATASET> - -`release_36vy7s5gtba67fmyxlmijpsaui` - -### - -<https://archive.org/details/academictorrents_70e0794e2292fc051a13f05ea6f5b6c16f3d3635> - -doi:10.1371/journal.pone.0120448 - -Single .rar file - -### Dataverse - -<https://dataverse.rsu.lv/dataset.xhtml?persistentId=doi:10.48510/FK2/IJO02B> - -Single excel file - -### Dataverse - -<https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/CLSFKX&version=1.1> - -doi:10.7910/DVN/CLSFKX - -Mulitple files; multiple versions? - -API fetch: <https://dataverse.harvard.edu/api/datasets/:persistentId/?persistentId=doi:10.7910/DVN/CLSFKX&version=1.1> - - .data.id - .data.latestVersion.datasetPersistentId - .data.latestVersion.versionNumber, .versionMinorNumber - .data.latestVersion.files[] - .dataFile - .contentType (mimetype) - .filename - .filesize (int, bytes) - .md5 - .persistendId - .description - .label (filename?) - .version - -Single file inside: <https://dataverse.harvard.edu/file.xhtml?persistentId=doi:10.7910/DVN/CLSFKX/XWEHBB> - -Download single file: <https://dataverse.harvard.edu/api/access/datafile/:persistentId/?persistentId=doi:10.7910/DVN/CLSFKX/XWEHBB> (redirects to AWS S3) - -Dataverse refs: -- 'doi' and 'hdl' are the two persistentId styles -- file-level persistentIds are optional, on a per-instance basis: https://guides.dataverse.org/en/latest/installation/config.html#filepidsenabled diff --git a/proposals/2021-09-09_fileset_ingest.md b/proposals/2021-09-09_fileset_ingest.md new file mode 100644 index 0000000..bb9d358 --- /dev/null +++ b/proposals/2021-09-09_fileset_ingest.md @@ -0,0 +1,337 @@ + +status: implemented + +Fileset Ingest Pipeline (for Datasets) +====================================== + +Sandcrawler currently has ingest support for individual files saved as `file` +entities in fatcat (xml and pdf ingest types) and HTML files with +sub-components saved as `webcapture` entities in fatcat (html ingest type). + +This document describes extensions to this ingest system to flexibly support +groups of files, which may be represented in fatcat as `fileset` entities. The +main new ingest type is `dataset`. + +Compared to the existing ingest process, there are two major complications with +datasets: + +- the ingest process often requires more than parsing HTML files, and will be + specific to individual platforms and host software packages +- the storage backend and fatcat entity type is flexible: a dataset might be + represented by a single file, multiple files combined in to a single .zip + file, or mulitple separate files; the data may get archived in wayback or in + an archive.org item + +The new concepts of "strategy" and "platform" are introduced to accomodate +these complications. + + +## Ingest Strategies + +The ingest strategy describes the fatcat entity type that will be output; the +storage backend used; and whether an enclosing file format is used. The +strategy to use can not be determined until the number and size of files is +known. It is a function of file count, total file size, and publication +platform. + +Strategy names are compact strings with the format +`{storage_backend}-{fatcat_entity}`. A `-bundled` suffix after a `fileset` +entity type indicates that metadata about multiple files is retained, but that +in the storage backend only a single enclosing file (eg, `.zip`) will be +stored. + +The supported strategies are: + +- `web-file`: single file of any type, stored in wayback, represented as fatcat `file` +- `web-fileset`: multiple files of any type, stored in wayback, represented as fatcat `fileset` +- `web-fileset-bundled`: single bundle file, stored in wayback, represented as fatcat `fileset` +- `archiveorg-file`: single file of any type, stored in archive.org item, represented as fatcat `file` +- `archiveorg-fileset`: multiple files of any type, stored in archive.org item, represented as fatcat `fileset` +- `archiveorg-fileset-bundled`: single bundle file, stored in archive.org item, represented as fatcat `fileset` + +"Bundle" or "enclosing" files are things like .zip or .tar.gz. Not all .zip +files are handled as bundles! Only when the transfer from the hosting platform +is via a "download all as .zip" (or similar) do we consider a zipfile a +"bundle" and index the interior files as a fileset. + +The term "bundle file" is used over "archive file" or "container file" to +prevent confusion with the other use of those terms in the context of fatcat +(container entities; archive; Internet Archive as an organiztion). + +The motivation for supporting both `web` and `archiveorg` is that `web` is +somewhat simpler for small files, but `archiveorg` is better for larger groups +of files (say more than 20) and larger total size (say more than 1 GByte total, +or 128 MByte for any one file). + +The motivation for supporting "bundled" filesets is that there is only a single +file to archive. + + +## Ingest Pseudocode + +1. Determine `platform`, which may involve resolving redirects and crawling a landing page. + + a. currently we always crawl the ingest `base_url`, capturing a platform landing page + b. we don't currently handle the case of `base_url` leading to a non-HTML + terminal resource. the `component` ingest type does handle this + +2. Use platform-specific methods to fetch manifest metadata and decide on an `ingest_strategy`. + + a. depending on platform, may include access URLs for multiple strategies + (eg, URL for each file and a bundle URL), metadata about the item for, eg, + archive.org item upload, etc + +3. Use strategy-specific methods to archive all files in platform manifest, and verify manifest metadata. + +4. Summarize status and return structured result metadata. + + a. if the strategy was `web-file` or `archiveorg-file`, potentially submit an + `ingest_file_result` object down the file ingest pipeline (Kafka topic and + later persist and fatcat import workers), with `dataset-file` ingest + type (or `{ingest_type}-file` more generally). + +New python types: + + FilesetManifestFile + path: str + size: Optional[int] + md5: Optional[str] + sha1: Optional[str] + sha256: Optional[str] + mimetype: Optional[str] + extra: Optional[Dict[str, Any]] + + status: Optional[str] + platform_url: Optional[str] + terminal_url: Optional[str] + terminal_dt: Optional[str] + + FilesetPlatformItem + platform_name: str + platform_status: str + platform_domain: Optional[str] + platform_id: Optional[str] + manifest: Optional[List[FilesetManifestFile]] + archiveorg_item_name: Optional[str] + archiveorg_item_meta + web_base_url + web_bundle_url + + ArchiveStrategyResult + ingest_strategy: str + status: str + manifest: List[FilesetManifestFile] + + FilesetIngestResult + ingest_strategy: str + status: str + manifest: List[FilesetManifestFile] + single_file_meta: Optional[dict] + single_terminal: Optional[dict] + single_cdx: Optional[dict] + bundle_file_meta: Optional[dict] + bundle_terminal: Optional[dict] + bundle_cdx: Optional[dict] + bundle_archiveorg_path: Optional[dict] + +New python APIs/classes: + + FilesetPlatformHelper + match_request(request, resource, html_biblio) -> bool + does the request and landing page metadata indicate a match for this platform? + process_request(request, resource, html_biblio) -> FilesetPlatformItem + do API requests, parsing, etc to fetch metadata and access URLs for this fileset/dataset. platform-specific + chose_strategy(item: FilesetPlatformItem) -> IngestStrategy + select an archive strategy for the given fileset/dataset + + FilesetIngestStrategy + check_existing(item: FilesetPlatformItem) -> Optional[ArchiveStrategyResult] + check the given backend for an existing capture/archive; if found, return result + process(item: FilesetPlatformItem) -> ArchiveStrategyResult + perform an actual archival capture + +## Limits and Failure Modes + +- `too-large-size`: total size of the fileset is too large for archiving. + initial limit is 64 GBytes, controlled by `max_total_size` parameter. +- `too-many-files`: number of files (and thus file-level metadata) is too + large. initial limit is 200, controlled by `max_file_count` parameter. +- `platform-scope / FilesetPlatformScopeError`: for when `base_url` leads to a + valid platform, which could be found via API or parsing, but has the wrong + scope. Eg, tried to fetch a dataset, but got a DOI which represents all + versions of the dataset, not a specific version. + + +## New Sandcrawler Code and Worker + + sandcrawler-ingest-fileset-worker@{1..6} (or up to 1..12 later) + +Worker consumes from ingest request topic, produces to fileset ingest results, +and optionally produces to file ingest results. + + sandcrawler-persist-ingest-fileset-worker@1 + +Simply writes fileset ingest rows to SQL. + + +## New Fatcat Worker and Code Changes + + fatcat-import-ingest-fileset-worker + +This importer is modeled on file and web worker. Filters for `success` with +strategy of `*-fileset*`. + +Existing `fatcat-import-ingest-file-worker` should be updated to allow +`dataset` single-file imports, with largely same behavior and semantics as +current importer (`component` mode). + +Existing fatcat transforms, and possibly even elasticsearch schemas, should be +updated to include fileset status and `in_ia` flag for dataset type releases. + +Existing entity updates worker submits `dataset` type ingests to ingest request +topic. + + +## Ingest Result Schema + +Common with file results, and mostly relating to landing page HTML: + + hit: bool + status: str + success + success-existing + success-file (for `web-file` or `archiveorg-file` only) + request: object + terminal: object + file_meta: object + cdx: object + revisit_cdx: object + html_biblio: object + +Additional fileset-specific fields: + + manifest: list of objects + platform_name: str + platform_domain: str + platform_id: str + ingest_strategy: str + archiveorg_item_name: str (optional, only for `archiveorg-*` strategies) + fileset_bundle (optional, only for `*-fileset-bundle` strategy) + archiveorg_bundle_path + file_meta + cdx + terminal + fileset_file (optional, only for `*-file` strategy) + file_meta + terminal + cdx + revisit_cdx + +If the strategy was `web-file` or `archiveorg-file` and the status is +`success-file`, then an ingest file result will also be published to +`sandcrawler-ENV.ingest-file-results`, using the same ingest type and fields as +regular ingest. + + +All fileset ingest results get published to ingest-fileset-result. + +Existing sandcrawler persist workers also subscribe to this topic and persist +status and landing page terminal info to tables just like with file ingest. +GROBID, HTML, and other metadata is not persisted in this path. + +If the ingest strategy was a single file (`*-file`), then an ingest file is +also published to the ingest-file-result topic, with the `fileset_file` +metadata, and ingest type `dataset-file`. This should only happen on success +condition. + + +## New SQL Tables + + CREATE TABLE IF NOT EXISTS ingest_fileset_platform ( + ingest_type TEXT NOT NULL CHECK (octet_length(ingest_type) >= 1), + base_url TEXT NOT NULL CHECK (octet_length(base_url) >= 1), + updated TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL, + hit BOOLEAN NOT NULL, + status TEXT CHECK (octet_length(status) >= 1), + + platform_name TEXT CHECK (octet_length(platform) >= 1), + platform_domain TEXT CHECK (octet_length(platform_domain) >= 1), + platform_id TEXT CHECK (octet_length(platform_id) >= 1), + ingest_strategy TEXT CHECK (octet_length(ingest_strategy) >= 1), + total_size BIGINT, + file_count INT, + archiveorg_item_name TEXT CHECK (octet_length(item_name) >= 1), + + archiveorg_item_bundle_path TEXT CHECK (octet_length(item_path_bundle) >= 1), + web_bundle_url TEXT CHECK (octet_length(terminal_url) >= 1), + web_bundle_dt TEXT CHECK (octet_length(terminal_dt) = 14), + + manifest JSONB, + -- list, similar to fatcat fileset manifest, plus extra: + -- status (str) + -- path (str) + -- size (int) + -- md5 (str) + -- sha1 (str) + -- sha256 (str) + -- mimetype (str) + -- extra (dict) + -- platform_url (str) + -- terminal_url (str) + -- terminal_dt (str) + + PRIMARY KEY (ingest_type, base_url) + ); + CREATE INDEX ingest_fileset_result_terminal_url_idx ON ingest_fileset_result(terminal_url); + # TODO: index on (platform_name,platform_domain,platform_id) ? + + +## New Kafka Topic + + sandcrawler-ENV.ingest-fileset-results 6x, no retention limit + + +## Implementation Plan + +First implement ingest worker, including platform and strategy helpers, and +test those as simple stdin/stdout CLI tools in sandcrawler repo to validate +this proposal. + +Second implement fatcat importer and test locally and/or in QA. + +Lastly implement infrastructure, automation, and other "glue": + +- SQL schema +- persist worker + + +## Design Note: Single-File Datasets + +Should datasets and other groups of files which only contain a single file get +imported as a fatcat `file` or `fileset`? This can be broken down further as +documents (single PDF) vs other individual files. + +Advantages of `file`: + +- handles case of article PDFs being marked as dataset accidentally +- `file` entities get de-duplicated with simple lookup (eg, on `sha1`) +- conceptually simpler if individual files are `file` entity +- easier to download individual files + +Advantages of `fileset`: + +- conceptually simpler if all `dataset` entities have `fileset` form factor +- code path is simpler: one fewer strategy, and less complexity of sending + files down separate import path +- metadata about platform is retained +- would require no modification of existing fatcat file importer +- fatcat import of archive.org of `file` is not actually implemented yet? + +Decision is to do individual files. Fatcat fileset import worker should reject +single-file (and empty) manifest filesets. Fatcat file import worker should +accept all mimetypes for `dataset-file` (similar to `component`). + + +## Example Entities + +See `notes/dataset_examples.txt` |