| Commit message (Collapse) | Author | Age | Files | Lines | ||
|---|---|---|---|---|---|---|
| ... | ||||||
| * | 'trawling' proposal (in progress) | Bryan Newbold | 2022-01-27 | 1 | -0/+177 | |
| | | ||||||
| * | ingest notes: various in-progress projects | Bryan Newbold | 2022-01-27 | 4 | -3/+800 | |
| | | ||||||
| * | sandcrawler: additional extracts, mostly OJS | Bryan Newbold | 2022-01-13 | 1 | -1/+23 | |
| | | ||||||
| * | filesets: more figshare URL patterns | Bryan Newbold | 2022-01-13 | 1 | -0/+13 | |
| | | ||||||
| * | fileset ingest: better verification of resources | Bryan Newbold | 2022-01-13 | 1 | -7/+23 | |
| | | ||||||
| * | ingest: PDF pattern for integrityresjournals.org | Bryan Newbold | 2022-01-13 | 1 | -0/+8 | |
| | | ||||||
| * | null-body -> empty-blob | Bryan Newbold | 2022-01-13 | 3 | -4/+8 | |
| | | ||||||
| * | spn: handle blocked-url (etc) better | Bryan Newbold | 2022-01-11 | 1 | -0/+10 | |
| | | ||||||
| * | enqueue PLATFORM PDFs for crawl | Bryan Newbold | 2022-01-07 | 1 | -0/+23 | |
| | | ||||||
| * | document progress on re-GROBID-ing | Bryan Newbold | 2022-01-05 | 1 | -0/+89 | |
| | | ||||||
| * | filesets: handle weird figshare link-only case better | Bryan Newbold | 2021-12-16 | 1 | -1/+4 | |
| | | ||||||
| * | lint ('not in') | Bryan Newbold | 2021-12-15 | 1 | -2/+2 | |
| | | ||||||
| * | lint: ignore unused 'sentry_client' | Bryan Newbold | 2021-12-15 | 1 | -1/+1 | |
| | | ||||||
| * | fix type with --enable-sentry | Bryan Newbold | 2021-12-15 | 1 | -1/+1 | |
| | | ||||||
| * | ingest tool: allow enabling sentry (for exception debugging) | Bryan Newbold | 2021-12-15 | 1 | -0/+13 | |
| | | ||||||
| * | more fileset ingest tweaks | Bryan Newbold | 2021-12-15 | 2 | -0/+7 | |
| | | ||||||
| * | fileset ingest: more requests timeouts, sessions | Bryan Newbold | 2021-12-15 | 3 | -37/+68 | |
| | | ||||||
| * | fileset ingest: create tmp subdirectories if needed | Bryan Newbold | 2021-12-15 | 1 | -0/+5 | |
| | | ||||||
| * | fileset ingest: configure IA session from env | Bryan Newbold | 2021-12-15 | 1 | -1/+6 | |
| | | | | | | Note that this doesn't currently work for `upload()`, and as a work-around I created `~/.config/ia.ini` manually on the worker VM. | |||||
| * | pipenv: add pymupdf; update trafilatura | Bryan Newbold | 2021-12-15 | 2 | -420/+644 | |
| | | ||||||
| * | fileset ingest: actually use spn2 CLI flag | Bryan Newbold | 2021-12-11 | 2 | -3/+4 | |
| | | ||||||
| * | notes on re-GROBID-ing (and re-extracting) some filestrawler | Bryan Newbold | 2021-12-09 | 1 | -0/+289 | |
| | | ||||||
| * | grobid: set a maximum file size (256 MByte) | Bryan Newbold | 2021-12-07 | 1 | -0/+8 | |
| | | ||||||
| * | worker: add kafka_group_suffix option | Bryan Newbold | 2021-12-07 | 1 | -3/+19 | |
| | | ||||||
| * | ingest tool: allow configuration of GROBID endpoint | Bryan Newbold | 2021-12-07 | 1 | -0/+7 | |
| | | ||||||
| * | 2021-12-02 database table size stats | Bryan Newbold | 2021-12-07 | 1 | -0/+22 | |
| | | ||||||
| * | sandcrawler SQL dump and upload updates | Bryan Newbold | 2021-12-07 | 1 | -4/+12 | |
| | | ||||||
| * | update fatcat_file SQL table schema, and add backfill notes | Bryan Newbold | 2021-12-07 | 1 | -1/+3 | |
| | | ||||||
| * | update fatcat_file SQL table schema, and add backfill notes | Bryan Newbold | 2021-12-01 | 1 | -0/+13 | |
| | | ||||||
| * | commit old patch crawl notes | Bryan Newbold | 2021-12-01 | 1 | -0/+488 | |
| | | ||||||
| * | Revert "pipenv: update deps" | Bryan Newbold | 2021-12-01 | 2 | -574/+382 | |
| | | | | | | | This reverts commit 7a5b203dbb37958a452eb1be3bd1bf8ed94cbbce. There is a problem with `internetarchive` 2.2.0, so reverting for now. | |||||
| * | pipenv: update deps | Bryan Newbold | 2021-12-01 | 2 | -382/+574 | |
| | | ||||||
| * | add CDX sha1hex lookup/fetch helper script | Bryan Newbold | 2021-11-30 | 1 | -0/+170 | |
| | | ||||||
| * | sandcrawler SQL stats | Bryan Newbold | 2021-11-27 | 2 | -12/+425 | |
| | | ||||||
| * | codespell typos in README and original RFC | Bryan Newbold | 2021-11-24 | 2 | -2/+2 | |
| | | ||||||
| * | codespell typos in python (comments) | Bryan Newbold | 2021-11-24 | 5 | -5/+5 | |
| | | ||||||
| * | html_meta: actual typo in code (CSS selector) caught by codespell | Bryan Newbold | 2021-11-24 | 1 | -1/+1 | |
| | | ||||||
| * | codespell fixes in proposals | Bryan Newbold | 2021-11-24 | 8 | -16/+16 | |
| | | ||||||
| * | ingest tool: new backfill mode | Bryan Newbold | 2021-11-16 | 1 | -0/+76 | |
| | | ||||||
| * | make fmt | Bryan Newbold | 2021-11-16 | 1 | -1/+1 | |
| | | ||||||
| * | SPNv2: make 'resources' optional | Bryan Newbold | 2021-11-16 | 1 | -1/+1 | |
| | | | | | | | | | This was always present previously. A change was made to SPNv2 API recently that borked it a bit, though in theory should be present on new captures. I'm not seeing it for some captures, so pushing this work around. It seems like we don't actually use this field anyways, at least for ingest pipeline. | |||||
| * | grobid: handle XML parsing errors, and have them recorded in sandcrawler-db | Bryan Newbold | 2021-11-12 | 1 | -1/+5 | |
| | | ||||||
| * | ingest_file: more efficient GROBID metadata copy | Bryan Newbold | 2021-11-12 | 1 | -3/+3 | |
| | | ||||||
| * | wrap up crossref refs backfill notes | Bryan Newbold | 2021-11-10 | 1 | -0/+47 | |
| | | ||||||
| * | grobid_tool: helper to process a single file | Bryan Newbold | 2021-11-10 | 1 | -0/+15 | |
| | | ||||||
| * | ingest: start re-processing GROBID with newer version | Bryan Newbold | 2021-11-10 | 1 | -2/+6 | |
| | | ||||||
| * | simple persist worker/tool to backfill grobid_refs | Bryan Newbold | 2021-11-10 | 2 | -0/+62 | |
| | | ||||||
| * | grobid: extract more metadata in document TEI-XML | Bryan Newbold | 2021-11-10 | 1 | -0/+5 | |
| | | ||||||
| * | grobid: update 'TODO' comment based on review | Bryan Newbold | 2021-11-04 | 1 | -3/+0 | |
| | | ||||||
| * | update crossref/grobid refs generation notes | Bryan Newbold | 2021-11-04 | 1 | -4/+96 | |
| | | ||||||
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |