FS2 Kafka FS2 Kafka

Technical Details

In the following sections, technical aspects of the library are detailed.

Implementation Notes

Following are some general library implementation notes.

Consumer Streaming

To enable backpressured streaming of Kafka records, there are three pieces at work.

  1. The 'consumer stream' continually issues fetch requests as long as there is demand. There is some room to issue fetch requests even when there isn't demand, to allow prefetching of records. This is controlled using the maxPrefetchBatches setting. Fetch requests will only be issued as long as processing is fast enough, so that less than maxPrefetchBatches batches are prefetched.
  2. The 'consumer loop' handles fetch requests issued by the stream, as well as poll requests, and other requests (e.g. commit). Records are only fetched for partitions where there is demand (pull-based). KafkaConsumerActor mostly implements the consumer loop internally.
  3. The 'poll scheduler' which schedules poll requests via a poll queue (bounded to 1 element).

These pieces are assembled in KafkaConsumer and run concurrently in independent Fibers.

Backpressure

There are a few important points to understanding how the backpressure works.