aboutsummaryrefslogtreecommitdiffstats
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
...
* small tweak to americanarchivist.org URL extractionBryan Newbold2020-02-241-1/+1
|
* fetch_petabox_body: allow non-200 status code fetchesBryan Newbold2020-02-241-2/+10
| | | | | | But only if it matches what the revisit record indicated. This is mostly to enable better revisit fetching.
* allow fuzzy revisit matchesBryan Newbold2020-02-241-1/+26
|
* ingest: more revisit fixesBryan Newbold2020-02-221-4/+4
|
* html: more publisher-specific fulltext extraction tricksBryan Newbold2020-02-221-0/+47
|
* ia: improve warc/revisit implementationBryan Newbold2020-02-221-26/+46
| | | | | A lot of the terminal-bad-status seems to have due to not handling revisits correctly. They have status_code = '-' or None.
* html: degruyter extraction; disabled journals.lww.comBryan Newbold2020-02-221-0/+19
|
* ingest: include better terminal URL/status_code/dtBryan Newbold2020-02-221-0/+8
| | | | Was getting a lot of "last hit" metadata for these columns.
* ingest: skip more non-pdf, non-paper domainsBryan Newbold2020-02-221-0/+9
|
* cdx: handle empty/null CDX responseBryan Newbold2020-02-221-0/+2
| | | | Sometimes seem to get empty string instead of empty JSON list
* html: handle TypeError during bs4 parseBryan Newbold2020-02-221-1/+7
|
* filter out CDX rows missing WARC playback fieldsBryan Newbold2020-02-191-0/+4
|
* pdf_trio persist fixes from prodBryan Newbold2020-02-192-5/+9
|
* allow <meta property=citation_pdf_url>Bryan Newbold2020-02-181-0/+3
| | | | at least researchgate does this (!)
* X-Archive-Src more robust than X-Archive-Redirect-ReasonBryan Newbold2020-02-181-2/+3
|
* wayback: on bad redirects, log instead of assertBryan Newbold2020-02-181-2/+13
| | | | This is a different form of mangled redirect.
* attempt to work around corrupt ARC files from alexa issueBryan Newbold2020-02-181-0/+5
|
* unpaywall2ingestrequest transform scriptBryan Newbold2020-02-182-1/+104
|
* pdftrio: mode controlled by CLI argBryan Newbold2020-02-182-10/+14
|
* pdftrio: fix error nesting in pdftrio keyBryan Newbold2020-02-181-12/+20
|
* include rel and oa_status in ingest request 'extra'Bryan Newbold2020-02-182-2/+2
|
* ingest: bulk workers don't hit SPNv2Bryan Newbold2020-02-131-0/+2
|
* pdftrio fixes from testingBryan Newbold2020-02-131-3/+9
|
* move pdf_trio results back under key in JSON/KafkaBryan Newbold2020-02-132-7/+31
|
* pdftrio: small fixes from testingBryan Newbold2020-02-121-2/+2
|
* pdftrio basic python codeBryan Newbold2020-02-127-1/+393
| | | | This is basically just a copy/paste of GROBID code, only simpler!
* add ingestrequest_row2json.pyBryan Newbold2020-02-051-0/+48
|
* fix persist bug where ingest_request_source not savedBryan Newbold2020-02-051-0/+1
|
* fix bug where ingest_request extra fields not persistedBryan Newbold2020-02-051-1/+2
|
* handle alternative dt format in WARC headersBryan Newbold2020-02-051-2/+4
| | | | | If there is a UTC timestamp, with trailing 'Z' indicating timezone, that is valid but increases string length by one.
* decrease SPNv2 polling timeout to 3 minutesBryan Newbold2020-02-051-2/+2
|
* improvements to reliability from prod testingBryan Newbold2020-02-032-7/+20
|
* hack-y backoff ingest attemptBryan Newbold2020-02-032-3/+26
| | | | | | | | | | | | | | | The goal here is to have SPNv2 requests backoff when we get back-pressure (usually caused by some sessions taking too long). Lack of proper back-pressure is making it hard to turn up parallelism. This is a hack because we still timeout and drop the slow request. A better way is probably to have a background thread run, while the KafkaPusher thread does polling. Maybe with timeouts to detect slow processing (greater than 30 seconds?) and only pause/resume in that case. This would also make taking batches easier. Unlike the existing code, however, the parallelism needs to happen at the Pusher level to do the polling (Kafka) and "await" (for all worker threads to complete) correctly.
* grobid petabox: fix fetch body/contentBryan Newbold2020-02-031-1/+1
|
* wayback: try to resolve HTTPException due to many HTTP headersBryan Newbold2020-02-021-1/+9
| | | | | | | | | This is withing GWB wayback code. Trying two things: - bump default max headers from 100 to 1000 in the (global?) http.client module itself. I didn't think through whether we would expect this to actually work - catch the exception, record it, move on
* sandcrawler_worker: ingest worker distinct consumer groupsBryan Newbold2020-01-291-1/+3
| | | | | | I'm in the process of resetting these consumer groups, so might as well take the opportunity to split by topic and use the new canonical naming format.
* grobid worker: catch PetaboxError alsoBryan Newbold2020-01-281-2/+2
|
* worker kafka setting tweaksBryan Newbold2020-01-281-2/+4
| | | | These are all attempts to get kafka workers operating more smoothly.
* make grobid-extract worker batch size 1Bryan Newbold2020-01-281-0/+1
| | | | | This is part of attempts to fix Kafka errors that look like they might be timeouts.
* workers: yes, poll is necessaryBryan Newbold2020-01-281-1/+1
|
* grobid worker: always set a key in responseBryan Newbold2020-01-281-4/+25
| | | | | | | | | We have key-based compaction enabled for the GROBID output topic. This means it is an error to public to that topic without a key set. Hopefully this change will end these errors, which look like: KafkaError{code=INVALID_MSG,val=2,str="Broker: Invalid message"}
* fix kafka worker partition-specific errorBryan Newbold2020-01-281-1/+1
|
* fix WaybackError exception formatingBryan Newbold2020-01-281-1/+1
|
* fix elif syntax errorBryan Newbold2020-01-281-1/+1
|
* block springer page-one domainBryan Newbold2020-01-281-0/+3
|
* clarify petabox fetch behaviorBryan Newbold2020-01-281-3/+6
|
* re-enable figshare and zenodo crawlingBryan Newbold2020-01-211-8/+0
| | | | For daily imports
* persist grobid: actually, status_code is requiredBryan Newbold2020-01-212-3/+10
| | | | | | | Instead of working around when missing, force it to exist but skip in database insert section. Disk mode still needs to check if blank.
* ingest: check for null-body before file_metaBryan Newbold2020-01-211-0/+3
| | | | | gen_file_metadata raises an assert error if body is None (or false-y in general)
* wayback: replay redirects have X-Archive-Redirect-ReasonBryan Newbold2020-01-211-2/+4
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-16 03:20:31 +0000