Kafka Offsets example

Category : Apache Kafka | Sub Category : Apache Kafka | By Prasad Bonam Last updated: 2023-08-05 09:42:18 Viewed : 339

Kafka Offsets example:

In Apache Kafka, offsets are used to represent the position of a consumer within a partition. Offsets are a critical aspect of Kafka`s design, as they allow consumers to keep track of their progress in consuming messages from a topic. Each partition maintains its own offset for each consumer group.

Lets go through an example to understand Kafka offsets:

1. Setting Up Kafka: For this example, assume you have set up a Kafka cluster with a single broker: localhost:9092.

2. Topic Creation: Create a Kafka topic named "my_topic" with three partitions and a replication factor of 1. This means there will be three partitions, and each partition will have a single replica.

   Using the command-line tool on Unix/Linux/Mac:

   bashCopy code
   bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my_topic --partitions 3 --replication-factor 1
   

   Using the command-line tool on Windows:

   batchCopy code
   binwindowskafka-topics.bat --bootstrap-server localhost:9092 --create --topic my_topic --partitions 3 --replication-factor 1
   

3. Producing Messages: Start a producer to send messages to the "my_topic" topic.

   Using the command-line tool on Unix/Linux/Mac:

   bashCopy code
   bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic
   

   Using the command-line tool on Windows:

   batchCopy code
   binwindowskafka-console-producer.bat --broker-list localhost:9092 --topic my_topic
   

   Now, you can enter messages in the console, and Kafka will publish them to one of the three partitions in the "my_topic" topic.

4. Consuming Messages: Start a consumer to read messages from the "my_topic" topic.

   Using the command-line tool on Unix/Linux/Mac:

   bashCopy code
   bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_topic --group my_group
   

   Using the command-line tool on Windows:

   batchCopy code
   binwindowskafka-console-consumer.bat --bootstrap-server localhost:9092 --topic my_topic --group my_group
   

   The consumer will read messages from one of the partitions in the "my_topic" topic. Kafka will automatically assign a partition to the consumer within the consumer group and start reading from the offset where it left off.

5. Offset Tracking: Kafka maintains the offset for each consumer group and partition. As the consumer reads messages, it updates its current offset. If the consumer is stopped and restarted, it will resume reading from the last committed offset for each partition.

6. Manual Offset Control: In some cases, you might want to manually control the offset position. Kafka allows you to commit offsets manually, which can be useful for scenarios like at-least-once message processing. You can commit offsets after successfully processing a batch of messages to ensure that they are not reprocessed in case of failures.

   In code (Java) for manual offset control with the KafkaConsumer:

   javaCopy code
   // Assuming you have a KafkaConsumer instance
   consumer.subscribe(Arrays.asList("my_topic"));
   
   try {
       while (true) {
           ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
           for (ConsumerRecord<String, String> record : records) {
               // Process the record
               System.out.printf("Offset: %d, Key: %s, Value: %s%n", record.offset(), record.key(), record.value());
           }
           // Manually commit the offsets after successful processing
           consumer.commitSync();
       }
   } finally {
       consumer.close();
   }
   

Kafka offsets play a crucial role in ensuring reliable message processing and data integrity. By maintaining the offset position, Kafka allows consumers to consume messages efficiently and handle various scenarios, including restarts, failures, and load balancing within consumer groups.