Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | wrap up basic (locally testable) ingest refactor | Bryan Newbold | 2020-01-09 | 3 | -182/+267 |
| | |||||
* | cli: allow multiple ingest single types | Bryan Newbold | 2020-01-09 | 1 | -3/+4 |
| | |||||
* | example.env: more required vars | Bryan Newbold | 2020-01-09 | 1 | -2/+7 |
| | |||||
* | basic elife+plos extraction tests | Bryan Newbold | 2020-01-09 | 3 | -0/+4842 |
| | | | | | Ripped out some HTML, but these could have been minimized even further to keep repository from growing large. | ||||
* | fix grobid test (ISO-8859-1 encoding) | Bryan Newbold | 2020-01-09 | 1 | -6/+4 |
| | | | | Also changes for wayback refactor | ||||
* | fix grobid tests for new wayback refactors | Bryan Newbold | 2020-01-09 | 3 | -15/+17 |
| | |||||
* | more wayback and SPN tests and fixes | Bryan Newbold | 2020-01-09 | 4 | -52/+220 |
| | |||||
* | pipenv: mock testing library | Bryan Newbold | 2020-01-09 | 2 | -309/+325 |
| | |||||
* | refactor CdxApiClient, add tests | Bryan Newbold | 2020-01-08 | 2 | -40/+240 |
| | | | | | | - always use auth token and get full CDX rows - simplify to "fetch" (exact url/dt match) and "lookup_best" methods - all redirect stuff will be moved to a higher level | ||||
* | refactor SavePaperNowClient and add test | Bryan Newbold | 2020-01-07 | 2 | -28/+314 |
| | | | | | | - response as a namedtuple - "remote" errors (aka, SPN API was HTTP 200 but returned error) aren't an exception | ||||
* | remove SPNv1 code paths | Bryan Newbold | 2020-01-07 | 2 | -65/+25 |
| | |||||
* | sql: more cool random queries | Bryan Newbold | 2020-01-02 | 1 | -0/+5 |
| | |||||
* | TODO: check out python ratelimit library | Bryan Newbold | 2020-01-02 | 1 | -1/+1 |
| | |||||
* | minio version/compression notes | Bryan Newbold | 2020-01-02 | 1 | -0/+11 |
| | |||||
* | handle grobid2json errors in calling code instead | Bryan Newbold | 2020-01-02 | 1 | -1/+7 |
| | |||||
* | db: move duplicate row filtering into DB insert helpers | Bryan Newbold | 2020-01-02 | 2 | -15/+26 |
| | |||||
* | remove unused filter in grobid worker | Bryan Newbold | 2020-01-02 | 1 | -1/+0 |
| | |||||
* | fix dict typo | Bryan Newbold | 2020-01-02 | 1 | -1/+1 |
| | |||||
* | improvements to grobid persist worker | Bryan Newbold | 2020-01-02 | 1 | -13/+16 |
| | |||||
* | small (syntax?) changes to pig join script | Bryan Newbold | 2020-01-02 | 1 | -2/+2 |
| | |||||
* | update persist worker invocation to use batches | Bryan Newbold | 2020-01-02 | 1 | -15/+55 |
| | |||||
* | set mimetype when PUT to minio | Bryan Newbold | 2020-01-02 | 1 | -0/+4 |
| | |||||
* | fix DB import counting | Bryan Newbold | 2020-01-02 | 1 | -4/+5 |
| | |||||
* | include an example environment file | Bryan Newbold | 2020-01-02 | 1 | -0/+2 |
| | |||||
* | teixml2json test update for skipping null JSON keys | Bryan Newbold | 2020-01-02 | 1 | -10/+1 |
| | |||||
* | fix small errors found by pylint | Bryan Newbold | 2020-01-02 | 2 | -1/+2 |
| | |||||
* | fix sandcrawler persist workers | Bryan Newbold | 2020-01-02 | 2 | -8/+37 |
| | |||||
* | filter ingest results to not have key conflicts within batch | Bryan Newbold | 2020-01-02 | 1 | -1/+16 |
| | | | | | This handles a corner case with ON CONFLICT ... DO UPDATE where you can't do multiple such updates in the same batch transaction. | ||||
* | db: fancy insert/update separation using postgres xmax | Bryan Newbold | 2020-01-02 | 2 | -24/+45 |
| | |||||
* | add PersistGrobidDiskWorker | Bryan Newbold | 2020-01-02 | 2 | -0/+60 |
| | | | | To help with making dumps directly from Kafka (eg, for partner delivery) | ||||
* | flush out minio helper, add to grobid persist | Bryan Newbold | 2020-01-02 | 3 | -24/+91 |
| | |||||
* | update minio README | Bryan Newbold | 2020-01-02 | 1 | -10/+42 |
| | |||||
* | implement counts properly for persist workers | Bryan Newbold | 2020-01-02 | 1 | -15/+19 |
| | |||||
* | improve DB helpers | Bryan Newbold | 2020-01-02 | 1 | -26/+81 |
| | | | | | - return insert/update row counts - implement ON CONFLICT ... DO UPDATE on some tables | ||||
* | be more parsimonious with GROBID metadata | Bryan Newbold | 2020-01-02 | 2 | -3/+20 |
| | | | | | Because these are getting persisted in database (as well as kafka), don't write out empty keys. | ||||
* | start work on DB connector and minio client | Bryan Newbold | 2020-01-02 | 2 | -0/+200 |
| | |||||
* | have JsonLinePusher continue on JSON decode errors (but count) | Bryan Newbold | 2020-01-02 | 1 | -1/+5 |
| | |||||
* | start work on persist workers and tool | Bryan Newbold | 2020-01-02 | 3 | -5/+336 |
| | |||||
* | yet more tweaks to ingest proposal | Bryan Newbold | 2020-01-02 | 1 | -3/+2 |
| | |||||
* | SQL docs update for diesel change | Bryan Newbold | 2020-01-02 | 2 | -0/+48 |
| | |||||
* | move SQL schema to diesel migration pattern | Bryan Newbold | 2020-01-02 | 5 | -70/+157 |
| | |||||
* | hadoop job log rename and update | Bryan Newbold | 2019-12-27 | 1 | -0/+25 |
| | |||||
* | update job log with pig runs | Bryan Newbold | 2019-12-26 | 1 | -0/+10 |
| | |||||
* | update TODO | Bryan Newbold | 2019-12-26 | 1 | -1/+7 |
| | |||||
* | kafka topics: compress api-datacite | Bryan Newbold | 2019-12-24 | 1 | -1/+1 |
| | |||||
* | basic arabesque2ingestrequest script | Bryan Newbold | 2019-12-24 | 1 | -0/+69 |
| | |||||
* | commit grobid_tool transform mode | Bryan Newbold | 2019-12-22 | 1 | -0/+27 |
| | | | | Had some stale code on aitio with this change I forgot to commit. Oops! | ||||
* | pig: first rev of join-cdx-sha1 script | Bryan Newbold | 2019-12-22 | 3 | -0/+91 |
| | |||||
* | pig: move count_lines helper to pighelper.py | Bryan Newbold | 2019-12-22 | 3 | -7/+6 |
| | |||||
* | refactor: use print(..., file=sys.stderr) | Bryan Newbold | 2019-12-18 | 5 | -32/+34 |
| | | | | Should use logging soon, but this seems more idiomatic in the meanwhile. |