index
:
sandcrawler
bnewbold-args
bnewbold-backfill
bnewbold-persist-grobid-errors
bnewbold-refactor-loggging
master
trawler
[no description]
about
summary
refs
log
tree
commit
diff
stats
log msg
author
committer
range
Commit message (
Expand
)
Author
Age
Files
Lines
...
*
enqueue PLATFORM PDFs for crawl
Bryan Newbold
2022-01-07
1
-0
/
+23
*
document progress on re-GROBID-ing
Bryan Newbold
2022-01-05
1
-0
/
+89
*
filesets: handle weird figshare link-only case better
Bryan Newbold
2021-12-16
1
-1
/
+4
*
lint ('not in')
Bryan Newbold
2021-12-15
1
-2
/
+2
*
lint: ignore unused 'sentry_client'
Bryan Newbold
2021-12-15
1
-1
/
+1
*
fix type with --enable-sentry
Bryan Newbold
2021-12-15
1
-1
/
+1
*
ingest tool: allow enabling sentry (for exception debugging)
Bryan Newbold
2021-12-15
1
-0
/
+13
*
more fileset ingest tweaks
Bryan Newbold
2021-12-15
2
-0
/
+7
*
fileset ingest: more requests timeouts, sessions
Bryan Newbold
2021-12-15
3
-37
/
+68
*
fileset ingest: create tmp subdirectories if needed
Bryan Newbold
2021-12-15
1
-0
/
+5
*
fileset ingest: configure IA session from env
Bryan Newbold
2021-12-15
1
-1
/
+6
*
pipenv: add pymupdf; update trafilatura
Bryan Newbold
2021-12-15
2
-420
/
+644
*
fileset ingest: actually use spn2 CLI flag
Bryan Newbold
2021-12-11
2
-3
/
+4
*
notes on re-GROBID-ing (and re-extracting) some files
trawler
Bryan Newbold
2021-12-09
1
-0
/
+289
*
grobid: set a maximum file size (256 MByte)
Bryan Newbold
2021-12-07
1
-0
/
+8
*
worker: add kafka_group_suffix option
Bryan Newbold
2021-12-07
1
-3
/
+19
*
ingest tool: allow configuration of GROBID endpoint
Bryan Newbold
2021-12-07
1
-0
/
+7
*
2021-12-02 database table size stats
Bryan Newbold
2021-12-07
1
-0
/
+22
*
sandcrawler SQL dump and upload updates
Bryan Newbold
2021-12-07
1
-4
/
+12
*
update fatcat_file SQL table schema, and add backfill notes
Bryan Newbold
2021-12-07
1
-1
/
+3
*
update fatcat_file SQL table schema, and add backfill notes
Bryan Newbold
2021-12-01
1
-0
/
+13
*
commit old patch crawl notes
Bryan Newbold
2021-12-01
1
-0
/
+488
*
Revert "pipenv: update deps"
Bryan Newbold
2021-12-01
2
-574
/
+382
*
pipenv: update deps
Bryan Newbold
2021-12-01
2
-382
/
+574
*
add CDX sha1hex lookup/fetch helper script
Bryan Newbold
2021-11-30
1
-0
/
+170
*
sandcrawler SQL stats
Bryan Newbold
2021-11-27
2
-12
/
+425
*
codespell typos in README and original RFC
Bryan Newbold
2021-11-24
2
-2
/
+2
*
codespell typos in python (comments)
Bryan Newbold
2021-11-24
5
-5
/
+5
*
html_meta: actual typo in code (CSS selector) caught by codespell
Bryan Newbold
2021-11-24
1
-1
/
+1
*
codespell fixes in proposals
Bryan Newbold
2021-11-24
8
-16
/
+16
*
ingest tool: new backfill mode
Bryan Newbold
2021-11-16
1
-0
/
+76
*
make fmt
Bryan Newbold
2021-11-16
1
-1
/
+1
*
SPNv2: make 'resources' optional
Bryan Newbold
2021-11-16
1
-1
/
+1
*
grobid: handle XML parsing errors, and have them recorded in sandcrawler-db
Bryan Newbold
2021-11-12
1
-1
/
+5
*
ingest_file: more efficient GROBID metadata copy
Bryan Newbold
2021-11-12
1
-3
/
+3
*
wrap up crossref refs backfill notes
Bryan Newbold
2021-11-10
1
-0
/
+47
*
grobid_tool: helper to process a single file
Bryan Newbold
2021-11-10
1
-0
/
+15
*
ingest: start re-processing GROBID with newer version
Bryan Newbold
2021-11-10
1
-2
/
+6
*
simple persist worker/tool to backfill grobid_refs
Bryan Newbold
2021-11-10
2
-0
/
+62
*
grobid: extract more metadata in document TEI-XML
Bryan Newbold
2021-11-10
1
-0
/
+5
*
grobid: update 'TODO' comment based on review
Bryan Newbold
2021-11-04
1
-3
/
+0
*
update crossref/grobid refs generation notes
Bryan Newbold
2021-11-04
1
-4
/
+96
*
crossref grobid refs: another error case (ReadTimeout)
Bryan Newbold
2021-11-04
2
-5
/
+11
*
db (postgrest): actually use an HTTP session
Bryan Newbold
2021-11-04
1
-12
/
+24
*
grobid: use requests session
Bryan Newbold
2021-11-04
1
-3
/
+4
*
grobid crossref refs: try to handle HTTP 5xx and XML parse errors
Bryan Newbold
2021-11-04
2
-5
/
+33
*
grobid: handle weird whitespace unstructured from crossref
Bryan Newbold
2021-11-04
1
-1
/
+10
*
crossref persist: batch size depends on whether parsing refs
Bryan Newbold
2021-11-04
2
-2
/
+8
*
sql: grobid_refs table JSON as 'JSON' not 'JSONB'
Bryan Newbold
2021-11-04
2
-3
/
+3
*
grobid refs backfill progress
Bryan Newbold
2021-11-04
1
-1
/
+43
[prev]
[next]