sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
*	gitlab CI: switch to ubuntu:focal	Bryan Newbold	2021-04-07	1	-14/+10
\|
*	2021-04-07 sandcrawler DB stats	Bryan Newbold	2021-04-07	1	-0/+428
\|
*	grobid: disable biblio-glutton consolidation	Bryan Newbold	2021-04-07	1	-3/+3
\|
*	ingest: handle current degruyter PDF link pattern	Bryan Newbold	2021-03-26	1	-0/+8
\|
*	add missing dotfiles (due to gitignore oops)	Bryan Newbold	2021-01-18	2	-0/+12
\|
*	pipenv: lock minio S3 library to <7.0.0	Bryan Newbold	2021-01-14	2	-242/+196
\| \| \| \| \| \| \| \| \| \| \|	In this upstream commit: https://github.com/minio/minio-py/commit/b81883a98e6f8a09e2903609caabbf0956dd0ec9 The API for errors changes, which makes it harder for use to catch specific exceptions (such as "NoSuchKey" as a Not Found / 404 error). Instead of refactoring, just going to pin the library. We should probably remove this library for a non-implementation-specific S3 client at some point; minio seems simpler than, eg, boto3, but there is probably something ever simpler out there.
*	more expansive python/.gitignore rules (all .gz)	Bryan Newbold	2021-01-05	1	-1/+1
\|
*	doaj ingest request updates (from prod)	Bryan Newbold	2021-01-05	1	-1/+5
\|
*	gitlab CI: update kitware (updated cmake on xenial) signing key	Bryan Newbold	2021-01-05	1	-2/+2
\|
*	gitlab CI: add libmagic1 dep (was a recommends)	Bryan Newbold	2021-01-05	1	-1/+1
\|
*	gitlab CI: add build deps	Bryan Newbold	2021-01-05	1	-1/+1
\|
*	gitlab CI: install fewer packages (no recommends)	Bryan Newbold	2021-01-05	1	-2/+2
\|
*	python makefile: don't duplicate 'lint' commands during 'test'	Bryan Newbold	2021-01-05	1	-2/+0
\|
*	update to python3.8	Bryan Newbold	2021-01-05	4	-403/+417
\|
*	pdf: yet more bad SHA1 (commiting lines from prod)	Bryan Newbold	2021-01-05	1	-0/+20
\|
*	ia CDX: handle bad CDX rows	Bryan Newbold	2021-01-05	1	-2/+4
\|
*	late-2020 OA DOI crawl ingest notes	Bryan Newbold	2021-01-04	1	-3/+46
\|
*	DOAJ crawl ingest stats	Bryan Newbold	2020-12-31	1	-0/+295
\|
*	progress notes on OA DOI ingest (still running)	Bryan Newbold	2020-12-28	1	-11/+102
\|
*	update HTML ingest proposal	Bryan Newbold	2020-12-23	1	-1/+3
\|
*	spn: more status codes	Bryan Newbold	2020-12-21	1	-1/+2
\|
*	HTML ingest deployment notes	Bryan Newbold	2020-12-16	1	-1/+71
\|
*	persist: html_meta is ON CONFLICT DO UPDATE	Bryan Newbold	2020-12-15	1	-1/+1
\|
*	persist: don't expect HTML TEI-XML in result object	Bryan Newbold	2020-12-15	1	-1/+1
\|
*	unpaywall crawl/ingest update (from Oct 2020)	Bryan Newbold	2020-12-08	1	-0/+134
\|
*	commit sept 2020 scielo ingest notes	Bryan Newbold	2020-12-08	1	-0/+21
\|
*	handle more wayback error conditions	Bryan Newbold	2020-11-20	1	-0/+6
\|
*	kafka docs for rolling back a consumer group	Bryan Newbold	2020-11-20	1	-0/+9
\|
*	html: more conservative parsing of element attr	Bryan Newbold	2020-11-20	1	-2/+4
\|
*	xml: catch parse error	Bryan Newbold	2020-11-19	1	-3/+8
\|
*	SQL: more ingest monitoring	Bryan Newbold	2020-11-16	3	-1/+660
\|
*	spn 'forbidden' status code	Bryan Newbold	2020-11-12	1	-1/+1
\|
*	html biblio: handle 'content not in attrs' case	Bryan Newbold	2020-11-12	1	-2/+2
\|
*	add implementation notes about HTML ingest	Bryan Newbold	2020-11-10	1	-0/+248
\|
*	fuzzy matching notes	Bryan Newbold	2020-11-10	1	-0/+148
\|
*	blacklist -> denylist	Bryan Newbold	2020-11-10	2	-9/+9
\|
*	pipenv: updates (mostly for trafilatura 0.6.0)	Bryan Newbold	2020-11-10	1	-25/+32
\|
*	DOAJ and HTML ingest tweaks from QA run	Bryan Newbold	2020-11-10	2	-3/+3
\|
*	html: handle more traf error cases	Bryan Newbold	2020-11-08	1	-2/+2
\|
*	html: more adblock	Bryan Newbold	2020-11-08	1	-1/+3
\|
*	ingest: small html_bibli typo	Bryan Newbold	2020-11-08	1	-1/+1
\|
*	html: most small platform tweaks	Bryan Newbold	2020-11-08	1	-5/+4
\|
*	move fuzzy URL match method to misc	Bryan Newbold	2020-11-08	3	-19/+20
\|
*	move some PDF URL extraction into declarative format	Bryan Newbold	2020-11-08	5	-143/+177
\|
*	basic DOAJ ingest request conversion script	Bryan Newbold	2020-11-08	1	-0/+139
\|
*	ingest: default to html_biblio for PDF URL extraction	Bryan Newbold	2020-11-08	1	-24/+17
\|
*	ingest: shorted scope+platform keys; use html_biblio extraction for PDFs	Bryan Newbold	2020-11-08	1	-15/+35
\|
*	html: more robust ingest; better platform and scope detection	Bryan Newbold	2020-11-08	1	-32/+96
\|
*	html: more extraction patterns; bugfix; skip more crossmark	Bryan Newbold	2020-11-08	1	-1/+24
\|
*	ingest html: return better status based on sniffed scope	Bryan Newbold	2020-11-08	1	-9/+31
\|