KafkaProducer

fs2.kafka.KafkaProducer
See theKafkaProducer companion object
abstract class KafkaProducer[F[_], K, V]

KafkaProducer represents a producer of Kafka records, with the ability to produce ProducerRecords using produce.

Attributes

Companion
object
Source
KafkaProducer.scala
Graph
Supertypes
class Object
trait Matchable
class Any
Known subtypes
trait WithSettings[F, K, V]
Self type
KafkaProducer[F, K, V]

Members list

Value members

Abstract methods

Enables transactions for the producer.

Enables transactions for the producer.

If using KafkaProducer.transactional or KafkaProducer.transactionalStream, then this will automatically be done when the producer is created. If not, then this function has to be called manually before using any of the transaction methods:

or an IllegalStateException exception will be raised.

Attributes

Source
KafkaProducer.scala
def metrics: F[Map[MetricName, Metric]]

Returns producer metrics.

Returns producer metrics.

Attributes

See also

org.apache.kafka.clients.producer.KafkaProducer#metrics

Source
KafkaProducer.scala
def partitionsFor(topic: String): F[List[PartitionInfo]]

Returns partition metadata for the given topic.

Returns partition metadata for the given topic.

Attributes

See also

org.apache.kafka.clients.producer.KafkaProducer#partitionsFor

Source
KafkaProducer.scala
def produce(records: ProducerRecords[K, V]): F[F[ProducerResult[K, V]]]

Produces the specified ProducerRecords in two steps: the first effect puts the records in the buffer of the producer, and the second effect waits for the records to send.

Produces the specified ProducerRecords in two steps: the first effect puts the records in the buffer of the producer, and the second effect waits for the records to send.

It's possible to flatten the result from this function to have an effect which both sends the records and waits for them to finish sending.

Waiting for individual records to send can substantially limit performance. In some cases, this is necessary, and so we might want to consider the following alternatives.

  • Wait for the produced records in batches, improving the rate at which records are produced, but loosing the guarantee where produce >> otherAction means otherAction executes after the record has been sent.

  • Run several produce.flatten >> otherAction concurrently, improving the rate at which records are produced, and still have otherAction execute after records have been sent, but losing the order of produced records.

Attributes

Source
KafkaProducer.scala

Produces the ProducerRecords in the specified TransactionalProducerRecords in four steps: first a transaction is initialized, then the records are placed in the buffer of the producer, then the offsets of the records are sent to the transaction, and lastly the transaction is committed.

Produces the ProducerRecords in the specified TransactionalProducerRecords in four steps: first a transaction is initialized, then the records are placed in the buffer of the producer, then the offsets of the records are sent to the transaction, and lastly the transaction is committed. If errors or cancellation occurs, the transaction is aborted. The returned effect succeeds if the whole transaction completes successfully.

Attributes

Source
KafkaProducer.scala
def produceTransactionally(records: ProducerRecords[K, V]): F[ProducerResult[K, V]]

Produces the ProducerRecords in the specified ProducerRecords in three steps: first a transaction is initialized, then the records are placed in the buffer of the producer, and lastly the transaction is committed.

Produces the ProducerRecords in the specified ProducerRecords in three steps: first a transaction is initialized, then the records are placed in the buffer of the producer, and lastly the transaction is committed. If errors or cancellation occurs, the transaction is aborted. The returned effect succeeds if the whole transaction completes successfully.

Attributes

Source
KafkaProducer.scala
def sendOffsetsToTransaction(offsets: Map[TopicPartition, OffsetAndMetadata], groupMetadata: ConsumerGroupMetadata): F[Unit]

Sends the specified offsets and KafkaConsumer.groupMetadata to be committed as part of a transaction.

Sends the specified offsets and KafkaConsumer.groupMetadata to be committed as part of a transaction.

Attributes

Source
KafkaProducer.scala

Return a resource which handles the transaction lifecycle.

Return a resource which handles the transaction lifecycle.

The returned resource begins a transaction on use and:

  • commits the transaction if the use finishes successfully, or

  • aborts the transaction if an error occurs or the process is canceled.

Note initTransactions must have been called before using this method, either manually or automatically through the use of KafkaProducer.transactional or KafkaProducer.transactionalStream.

Also note only one transaction can be open at any time. If a second transaction is started within the lifecycle of a first transaction, the second transaction will deadlock.

producer.transaction.surround {
 producer.transaction.surround { // deadlocks waiting for the outer transaction
   IO.unit
 }
}

Attributes

Source
KafkaProducer.scala
def withSerializers[K2, V2](keySerializer: KeySerializer[F, K2], valueSerializer: ValueSerializer[F, V2]): KafkaProducer[F, K2, V2]

Returns a new KafkaProducer using the same underlying producer but with different key and value serializers.

Returns a new KafkaProducer using the same underlying producer but with different key and value serializers.

Attributes

Source
KafkaProducer.scala

Concrete methods

final def imapK[G[_] : MonadCancelThrow](fk: FunctionK[F, G], gk: FunctionK[G, F])(implicit evidence$1: MonadCancelThrow[G], F: MonadCancelThrow[F]): KafkaProducer[G, K, V]

Creates a new KafkaProducer in which the effect type has been changed using the specified FunctionKs.

Creates a new KafkaProducer in which the effect type has been changed using the specified FunctionKs.

Attributes

Source
KafkaProducer.scala
def produceOne(topic: String, key: K, value: V): F[F[ProducerResult[K, V]]]
Implicitly added by ProducerOps

Produce a single record to the specified topic using the provided key and value, see KafkaProducer.produce for general semantics.

Produce a single record to the specified topic using the provided key and value, see KafkaProducer.produce for general semantics.

Attributes

Source
KafkaProducer.scala
def produceOne(record: ProducerRecord[K, V]): F[F[ProducerResult[K, V]]]
Implicitly added by ProducerOps

Produce a single ProducerRecord, see KafkaProducer.produce for general semantics.

Produce a single ProducerRecord, see KafkaProducer.produce for general semantics.

Attributes

Source
KafkaProducer.scala
def produceOne_(record: ProducerRecord[K, V])(implicit F: Functor[F]): F[F[RecordMetadata]]
Implicitly added by ProducerOps

Produce a single ProducerRecord, see KafkaProducer.produce for general semantics.

Produce a single ProducerRecord, see KafkaProducer.produce for general semantics.

Attributes

Source
KafkaProducer.scala
def produceOne_(topic: String, key: K, value: V)(implicit F: Functor[F]): F[F[RecordMetadata]]
Implicitly added by ProducerOps

Produce a single record to the specified topic using the provided key and value, see KafkaProducer.produce for general semantics.

Produce a single record to the specified topic using the provided key and value, see KafkaProducer.produce for general semantics.

Attributes

Source
KafkaProducer.scala