| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The goal here is to have SPNv2 requests backoff when we get
back-pressure (usually caused by some sessions taking too long). Lack of
proper back-pressure is making it hard to turn up parallelism.
This is a hack because we still timeout and drop the slow request. A
better way is probably to have a background thread run, while the
KafkaPusher thread does polling. Maybe with timeouts to detect slow
processing (greater than 30 seconds?) and only pause/resume in that
case. This would also make taking batches easier. Unlike the existing
code, however, the parallelism needs to happen at the Pusher level to do
the polling (Kafka) and "await" (for all worker threads to complete)
correctly.
|
| |
|
|
| |
These are all attempts to get kafka workers operating more smoothly.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Should use logging soon, but this seems more idiomatic in the meanwhile.
|
| |
|
|
|
| |
Scalding test is broken :(
But we aren't even using that code much these days.
|
| |
|
|
|
| |
The ingest worker keeps timing out at just over 5 minutes, so bump it
just a bit.
|
| |
|
|
|
|
|
|
| |
Was defaulting to 100, which I think was resulting in lots of consumer
group timeouts, resulting in UNKNOWN_MEMBER_ID errors.
Will probably switch back to batches of 10 or so, but multi-processing
or some other concurrent dispatch/processing.
|
| |
|
|
| |
Best to do this in wrapping code for full flexibility.
|
| | |
|
| | |
|
| | |
|
| |
|
