Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | add (skipped) live tests for wayback services | Bryan Newbold | 2020-01-09 | 1 | -0/+73 |
| | |||||
* | add ingest test file | Bryan Newbold | 2020-01-09 | 1 | -0/+120 |
| | | | | Forgot to commit earlier! | ||||
* | lots of progress on wayback refactoring | Bryan Newbold | 2020-01-09 | 3 | -51/+145 |
| | | | | | | - too much to list - canonical flags to control crawling - cdx_to_dict helper | ||||
* | location comes as a string, not list | Bryan Newbold | 2020-01-09 | 2 | -5/+5 |
| | |||||
* | fix http/https issue with GlobalWayback library | Bryan Newbold | 2020-01-09 | 1 | -1/+2 |
| | |||||
* | wayback fetch via replay; confirm hashes in crawl_resource() | Bryan Newbold | 2020-01-09 | 1 | -5/+40 |
| | |||||
* | wrap up basic (locally testable) ingest refactor | Bryan Newbold | 2020-01-09 | 3 | -182/+267 |
| | |||||
* | cli: allow multiple ingest single types | Bryan Newbold | 2020-01-09 | 1 | -3/+4 |
| | |||||
* | example.env: more required vars | Bryan Newbold | 2020-01-09 | 1 | -2/+7 |
| | |||||
* | basic elife+plos extraction tests | Bryan Newbold | 2020-01-09 | 3 | -0/+4842 |
| | | | | | Ripped out some HTML, but these could have been minimized even further to keep repository from growing large. | ||||
* | fix grobid test (ISO-8859-1 encoding) | Bryan Newbold | 2020-01-09 | 1 | -6/+4 |
| | | | | Also changes for wayback refactor | ||||
* | fix grobid tests for new wayback refactors | Bryan Newbold | 2020-01-09 | 3 | -15/+17 |
| | |||||
* | more wayback and SPN tests and fixes | Bryan Newbold | 2020-01-09 | 4 | -52/+220 |
| | |||||
* | pipenv: mock testing library | Bryan Newbold | 2020-01-09 | 2 | -309/+325 |
| | |||||
* | refactor CdxApiClient, add tests | Bryan Newbold | 2020-01-08 | 2 | -40/+240 |
| | | | | | | - always use auth token and get full CDX rows - simplify to "fetch" (exact url/dt match) and "lookup_best" methods - all redirect stuff will be moved to a higher level | ||||
* | refactor SavePaperNowClient and add test | Bryan Newbold | 2020-01-07 | 2 | -28/+314 |
| | | | | | | - response as a namedtuple - "remote" errors (aka, SPN API was HTTP 200 but returned error) aren't an exception | ||||
* | remove SPNv1 code paths | Bryan Newbold | 2020-01-07 | 2 | -65/+25 |
| | |||||
* | TODO: check out python ratelimit library | Bryan Newbold | 2020-01-02 | 1 | -1/+1 |
| | |||||
* | handle grobid2json errors in calling code instead | Bryan Newbold | 2020-01-02 | 1 | -1/+7 |
| | |||||
* | db: move duplicate row filtering into DB insert helpers | Bryan Newbold | 2020-01-02 | 2 | -15/+26 |
| | |||||
* | remove unused filter in grobid worker | Bryan Newbold | 2020-01-02 | 1 | -1/+0 |
| | |||||
* | fix dict typo | Bryan Newbold | 2020-01-02 | 1 | -1/+1 |
| | |||||
* | improvements to grobid persist worker | Bryan Newbold | 2020-01-02 | 1 | -13/+16 |
| | |||||
* | update persist worker invocation to use batches | Bryan Newbold | 2020-01-02 | 1 | -15/+55 |
| | |||||
* | set mimetype when PUT to minio | Bryan Newbold | 2020-01-02 | 1 | -0/+4 |
| | |||||
* | fix DB import counting | Bryan Newbold | 2020-01-02 | 1 | -4/+5 |
| | |||||
* | include an example environment file | Bryan Newbold | 2020-01-02 | 1 | -0/+2 |
| | |||||
* | teixml2json test update for skipping null JSON keys | Bryan Newbold | 2020-01-02 | 1 | -10/+1 |
| | |||||
* | fix small errors found by pylint | Bryan Newbold | 2020-01-02 | 2 | -1/+2 |
| | |||||
* | fix sandcrawler persist workers | Bryan Newbold | 2020-01-02 | 2 | -8/+37 |
| | |||||
* | filter ingest results to not have key conflicts within batch | Bryan Newbold | 2020-01-02 | 1 | -1/+16 |
| | | | | | This handles a corner case with ON CONFLICT ... DO UPDATE where you can't do multiple such updates in the same batch transaction. | ||||
* | db: fancy insert/update separation using postgres xmax | Bryan Newbold | 2020-01-02 | 2 | -24/+45 |
| | |||||
* | add PersistGrobidDiskWorker | Bryan Newbold | 2020-01-02 | 2 | -0/+60 |
| | | | | To help with making dumps directly from Kafka (eg, for partner delivery) | ||||
* | flush out minio helper, add to grobid persist | Bryan Newbold | 2020-01-02 | 3 | -24/+91 |
| | |||||
* | implement counts properly for persist workers | Bryan Newbold | 2020-01-02 | 1 | -15/+19 |
| | |||||
* | improve DB helpers | Bryan Newbold | 2020-01-02 | 1 | -26/+81 |
| | | | | | - return insert/update row counts - implement ON CONFLICT ... DO UPDATE on some tables | ||||
* | be more parsimonious with GROBID metadata | Bryan Newbold | 2020-01-02 | 2 | -3/+20 |
| | | | | | Because these are getting persisted in database (as well as kafka), don't write out empty keys. | ||||
* | start work on DB connector and minio client | Bryan Newbold | 2020-01-02 | 2 | -0/+200 |
| | |||||
* | have JsonLinePusher continue on JSON decode errors (but count) | Bryan Newbold | 2020-01-02 | 1 | -1/+5 |
| | |||||
* | start work on persist workers and tool | Bryan Newbold | 2020-01-02 | 3 | -5/+336 |
| | |||||
* | update TODO | Bryan Newbold | 2019-12-26 | 1 | -1/+7 |
| | |||||
* | basic arabesque2ingestrequest script | Bryan Newbold | 2019-12-24 | 1 | -0/+69 |
| | |||||
* | commit grobid_tool transform mode | Bryan Newbold | 2019-12-22 | 1 | -0/+27 |
| | | | | Had some stale code on aitio with this change I forgot to commit. Oops! | ||||
* | refactor: use print(..., file=sys.stderr) | Bryan Newbold | 2019-12-18 | 5 | -32/+34 |
| | | | | Should use logging soon, but this seems more idiomatic in the meanwhile. | ||||
* | refactor: sort keys in JSON output | Bryan Newbold | 2019-12-18 | 4 | -6/+7 |
| | | | | This makes debugging by tailing Kafka topics a lot more readable | ||||
* | refactor: improve argparse usage | Bryan Newbold | 2019-12-18 | 5 | -13/+27 |
| | | | | | use ArgumentDefaultsHelpFormatter and add help messages to all sub-commands | ||||
* | update ingest proposal source/link naming | Bryan Newbold | 2019-12-13 | 1 | -1/+1 |
| | |||||
* | fixes for large GROBID result skip | Bryan Newbold | 2019-12-02 | 1 | -2/+2 |
| | |||||
* | count empty blobs as 'failed' instead of crashing | Bryan Newbold | 2019-12-01 | 1 | -1/+2 |
| | | | | Might be better to record an artificial kafka response instead? | ||||
* | cleanup unused import | Bryan Newbold | 2019-12-01 | 1 | -1/+0 |
| |