index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
sql: grobid_refs table JSON as 'JSON' not 'JSONB'
Bryan Newbold
2021-11-04
2
-3
/
+3
*
grobid refs backfill progress
Bryan Newbold
2021-11-04
1
-1
/
+43
*
record SQL table sizes at start of crossref re-ingest
Bryan Newbold
2021-11-04
1
-0
/
+19
*
start notes on crossref refs backfill
Bryan Newbold
2021-11-04
1
-0
/
+54
*
crossref persist: make GROBID ref parsing an option (not default)
Bryan Newbold
2021-11-04
3
-9
/
+33
*
add grobid_refs and crossref_with_refs to sandcrawler-db SQL schema
Bryan Newbold
2021-11-04
1
-0
/
+21
*
glue, utils, and worker code for crossref and grobid_refs
Bryan Newbold
2021-11-04
4
-5
/
+212
*
update grobid refs proposal
Bryan Newbold
2021-11-04
1
-10
/
+72
*
iterated GROBID citation cleaning and processing
Bryan Newbold
2021-11-04
1
-27
/
+45
*
grobid citations: first pass at cleaning unstructured
Bryan Newbold
2021-11-04
1
-2
/
+34
*
initial proposal for GROBID refs table and pipeline
Bryan Newbold
2021-11-04
1
-0
/
+63
*
initial crossref-refs via GROBID helper routine
Bryan Newbold
2021-11-04
7
-6
/
+839
*
pipenv: bump grobid_tei_xml version to 0.1.2
Bryan Newbold
2021-11-04
2
-11
/
+11
*
pdftrio client: use HTTP session for POSTs
Bryan Newbold
2021-11-03
1
-1
/
+1
*
workers: use HTTP session for archive.org fetches
Bryan Newbold
2021-11-03
1
-3
/
+3
*
IA (wayback): actually use an HTTP session for replay fetches
Bryan Newbold
2021-11-03
1
-2
/
+3
*
SPN reingest: 6 hour minimum, 6 month max
Bryan Newbold
2021-11-03
1
-2
/
+2
*
sql: fix typo in quarterly (not weekly) script
Bryan Newbold
2021-11-03
1
-1
/
+1
*
sql: fixes to ingest_fileset_platform schema (from table creation)
Bryan Newbold
2021-11-01
2
-12
/
+12
*
updates/corrections to old small.json GROBID metadata example file
Bryan Newbold
2021-10-27
1
-6
/
+1
*
remove grobid2json helper file, replace with grobid_tei_xml
Bryan Newbold
2021-10-27
7
-224
/
+22
*
small type annotation things from additional packages
Bryan Newbold
2021-10-27
2
-5
/
+14
*
toolchain config updates
Bryan Newbold
2021-10-27
3
-10
/
+6
*
make fmt (black 21.9b0)
Bryan Newbold
2021-10-27
57
-3126
/
+3991
*
pipenv: flipflop from yapf back to black; more type packages; bump grobid_tei...
Bryan Newbold
2021-10-27
2
-27
/
+112
*
fileset: refactor out tables of helpers
Bryan Newbold
2021-10-27
3
-21
/
+19
*
gitlab-ci: copy env var in to place for tests
Bryan Newbold
2021-10-27
1
-0
/
+1
*
fix type annotations for petabox body fetch helper
Bryan Newbold
2021-10-26
5
-8
/
+11
*
small type annotation hack
Bryan Newbold
2021-10-26
1
-1
/
+1
*
fileset: fix field renaming bug (caught by mypy)
Bryan Newbold
2021-10-26
1
-2
/
+2
*
fileset ingest: fix table name typo (via mypy)
Bryan Newbold
2021-10-26
1
-1
/
+1
*
update 'XXX' notes from fileset ingest development
Bryan Newbold
2021-10-26
2
-9
/
+6
*
bugfix: setting html_biblio on ingest results
Bryan Newbold
2021-10-26
2
-2
/
+2
*
lint collection membership (last lint for now)
Bryan Newbold
2021-10-26
7
-32
/
+32
*
commit updated flake8 lint configuration
Bryan Newbold
2021-10-26
1
-6
/
+10
*
ingest fileset: fix silly import typo
Bryan Newbold
2021-10-26
1
-1
/
+1
*
type annotations for persist workers; required some work
Bryan Newbold
2021-10-26
1
-66
/
+59
*
ingest file HTTP API: fixes from type checking
Bryan Newbold
2021-10-26
1
-3
/
+3
*
more progress on type annotations
Bryan Newbold
2021-10-26
8
-34
/
+55
*
grobid: fix a bug with consolidate_mode header, exposed by type annotations
Bryan Newbold
2021-10-26
1
-1
/
+2
*
grobid: type annotations
Bryan Newbold
2021-10-26
1
-9
/
+19
*
type annotations on SandcrawlerWorker
Bryan Newbold
2021-10-26
1
-46
/
+57
*
more progress on type annotations and linting
Bryan Newbold
2021-10-26
11
-55
/
+87
*
live tests: FTP wayback replay now returns 200, not 226
Bryan Newbold
2021-10-26
1
-2
/
+2
*
ia: more tweaks to delicate code to satisfy type checker
Bryan Newbold
2021-10-26
1
-10
/
+12
*
ia helpers: enforce max_redirects count correctly
Bryan Newbold
2021-10-26
1
-1
/
+1
*
set CDX request params are str, not int or datetime
Bryan Newbold
2021-10-26
1
-3
/
+6
*
bugfix: was setting 'from' parameter as a tuple, not a string
Bryan Newbold
2021-10-26
1
-1
/
+1
*
start type annotating IA helper code
Bryan Newbold
2021-10-26
1
-37
/
+65
*
start adding python type annotations to db and persist code
Bryan Newbold
2021-10-26
2
-97
/
+124
[prev]
[next]