status: deployed

NOTE: while this has been used in production, as of December 2022 the results
are not used much in practice, and we don't score every PDF that comes along

PDF Trio (ML Classification)
==============================

This document describes how we intent to integrate the first generation of PDF
classification work into the sandcrawler processing system.

- abstractions (APIs)
- schemas
- how models and dependencies are deployed
- what code is release where under what license


## Code Structure

Major components:

**Training code, documentation, datasets:** Not used at run-time (does not need
to be deployed). Should be public. The datasets (PDFs) are copyrighted, so we
should only release URL lists that point to wayback.

**Models:** all are static, uploaded to archive.org items, simple download to
deploy. Should be versioned, and have unique versioned file names or directory
paths (aka, deploy in parallel).

**Image classifier backend:** vanilla tensorflow serving docker image, with a
bunch of invocation configs, plus static models.

**BERT backend:** vanilla tensorflow serving docker image, plus config, plus
models. Basically same as image classifier.

**API service:** currently Flask. Depends on tools like imagemagik, fasttext,
pdftotext. Seems like apt+pipenv should work?


## API Refactors

Changes:

- probably re-write README?
- refactor python code into directories
- add python tests
- tweak schema
- proper parallelization: uwsgi? async?

New features:

- option to send images, raw text in batches in addition to PDFs.

## Client Code

Basically just like GROBID client for now. Requests, JSON.

## JSON Schema

Output that goes in Kafka topic:

    key (sha1hex)
    pdf_trio
        status
        status_code
        ensemble_score
        bert_score
        image_score
        linear_score
        versions
            pdftrio_version (string)
            models_date (string, ISO date)
            git_rev (string)
            bert_model (string)
            image_model (string)
            linear_model (string)
        timing (optional/future: as reported by API)
            ...
    file_meta
        sha1hex
        ...
    timing
        ...


## SQL Schema

Ensemble model versions are summarized as a date.

    CREATE TABLE IF NOT EXISTS pdftrio (
        sha1hex             TEXT PRIMARY KEY CHECK (octet_length(sha1hex) = 40),
        updated             TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL,
        status_code         INT NOT NULL,
        status              TEXT CHECK (octet_length(status) >= 1) NOT NULL,
        pdftrio_version     TEXT CHECK (octet_length(pdftrio_version) >= 1),
        models_date         DATE,
        ensemble_score      REAL,
        bert_score          REAL,
        linear_score        REAL,
        image_score         REAL
    );

## Kafka Topic

sandcrawler-qa.pdftrio-output