| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | ingest: always try some lancet journals | Bryan Newbold | 2020-03-19 | 1 | -0/+3 |
| | | |||||
| * | entity worker: ingest more releases | Bryan Newbold | 2020-02-22 | 1 | -1/+37 |
| | | | | | | | | | If release is a dataset or image, don't do a pdf ingest request. If release is a datacite DOI, and release_type is a "document", crawl regardless of is_oa detection. This is mostly to crawl repositories (institutional or subject). | ||||
| * | always crawl researchgate DOIs | Bryan Newbold | 2020-02-18 | 1 | -0/+2 |
| | | | | | Now that ingest is fixed | ||||
| * | add acceptlist override for biorxiv/medrxiv | Bryan Newbold | 2020-02-10 | 1 | -2/+12 |
| | | |||||
| * | fix KafkaError worker reporting for partition errors | Bryan Newbold | 2020-01-29 | 1 | -1/+1 |
| | | |||||
| * | additional DOI prefix filters | Bryan Newbold | 2020-01-28 | 1 | -0/+8 |
| | | | | | From martin, thanks. | ||||
| * | apply ingest request filtering in entity worker | Bryan Newbold | 2020-01-28 | 1 | -3/+34 |
| | | | | | | | | `ingest_oa_only` behavior, and other filters, now handled in the entity update worker, instead of in the transform function. Also add a DOI prefix blocklist feature. | ||||
| * | update ingest request schema | Bryan Newbold | 2019-12-13 | 1 | -1/+1 |
| | | | | | | This is mostly changing ingest_type from 'file' to 'pdf', and adding 'link_source'/'link_source_id', plus some small cleanups. | ||||
| * | project -> ingest_request_source | Bryan Newbold | 2019-11-15 | 1 | -1/+1 |
| | | |||||
| * | add ingest request feature to entity_updates worker | Bryan Newbold | 2019-11-15 | 1 | -4/+20 |
| | | | | | | | | | | | | | | Initially was going to create a new worker to consume from the release update channel, but couldn't get the edit context ("is this a new release, or update to an existing") from that context. Currently there is a flag in source code to control whether we only do OA releases or all releases. Starting with OA only to start slow, but should probably default to all, and make this a config flag. Should probably also have a config flag to control this entire feature. Tested locally in dev. | ||||
| * | review/fix all confluent-kafka produce code | Bryan Newbold | 2019-09-20 | 1 | -4/+12 |
| | | |||||
| * | small fixes to confluent-kafka importers/workers | Bryan Newbold | 2019-09-20 | 1 | -4/+10 |
| | | | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again) | ||||
| * | convert pipeline workers from pykafka to confluent-kafka | Bryan Newbold | 2019-09-20 | 1 | -67/+116 |
| | | |||||
| * | fix typo in typo | Bryan Newbold | 2019-06-24 | 1 | -1/+1 |
| | | |||||
| * | fix typo in changelog worker | Bryan Newbold | 2019-06-24 | 1 | -1/+1 |
| | | |||||
| * | more links on new homepage | Bryan Newbold | 2019-06-19 | 1 | -1/+1 |
| | | | | | | matching produce sizes. may want to tweak this config in the future for throughput. | ||||
| * | fix and workaround container entities in release topic | Bryan Newbold | 2019-05-30 | 1 | -2/+2 |
| | | |||||
| * | file and container update kafka topics | Bryan Newbold | 2019-05-30 | 1 | -54/+69 |
| | | |||||
| * | update elastic for releases when files added | Bryan Newbold | 2019-05-30 | 1 | -1/+36 |
| | | | | | A bunch of remaining TODOs here | ||||
| * | 10 MByte default Kafka produce (workers) | Bryan Newbold | 2019-03-06 | 1 | -2/+6 |
| | | |||||
| * | bunch of lint/whitespace cleanups | Bryan Newbold | 2019-02-22 | 1 | -2/+1 |
| | | |||||
| * | include filesets and webcaptures in exports | Bryan Newbold | 2019-01-18 | 1 | -1/+1 |
| | | |||||
| * | workers do API-passing (not URI-passing) | Bryan Newbold | 2019-01-08 | 1 | -4/+4 |
| | | |||||
| * | not as strong a todo (timestamps) | Bryan Newbold | 2018-11-19 | 1 | -1/+1 |
| | | |||||
| * | bunch of pylint cleanup | Bryan Newbold | 2018-11-15 | 1 | -1/+1 |
| | | |||||
| * | large refactor of python names/paths | Bryan Newbold | 2018-11-15 | 1 | -3/+4 |
| | | | | | | | | - Add __init__.py files for fatcat_tools submodules, and use them in imports - Add a bunch of comments to files. - rename a number of classes and functions to be less verbose | ||||
| * | fix worker code | Bryan Newbold | 2018-11-14 | 1 | -2/+3 |
| | | |||||
| * | most_recent_message as reusable function | Bryan Newbold | 2018-11-14 | 1 | -26/+1 |
| | | |||||
| * | switch to auto consumer offset updates | Bryan Newbold | 2018-11-13 | 1 | -1/+6 |
| | | | | | | | This is the classic/correct way to do consumer group updates for higher throughput, when "at least once" semantics are acceptible (as they are here; double processing should be safe/fine). | ||||
| * | shuffle around fatcat_tools layout | Bryan Newbold | 2018-11-13 | 1 | -0/+122 |
 |
index : fatcat | |
| [no description] |
| summaryrefslogtreecommitdiffstats |