Kafka tips from banking work

Getting a Kafka producer and consumer working takes an afternoon. Keeping them correct in production, where a duplicate message can mean a duplicate payment, takes a lot more care. These are the lessons I treat as non-negotiable after years on banking and payments backends.

1. Assume every message will arrive more than once

Kafka gives you at-least-once delivery by default. Rebalances, retries and restarts all cause redelivery. So every consumer must be idempotent.

The simplest reliable pattern: store a unique business ID (payment ID, event ID) in a table with a unique constraint, in the same database transaction as your business change.

@Transactional
public void handle(PaymentEvent event) {
    // Insert fails with a constraint violation if we've seen this event already
    if (!processedEvents.tryInsert(event.eventId())) {
        log.info("Duplicate event {} ignored", event.eventId());
        return;
    }
    accountService.apply(event);
}

Database first, Kafka offset second. If the app crashes in between, the redelivered message is simply ignored.

2. Make the producer safe too

Modern Kafka clients (3.0+) default to acks=all and enable.idempotence=true. Check you haven't overridden them with older settings copied from a 2018 blog post. For financial events, also send a stable event ID in the payload or headers, so consumers can deduplicate (see lesson 1).

3. Pick the message key for ordering, on purpose

Kafka only guarantees order within a partition, and the key decides the partition. If debits and credits for the same account must be processed in order, the key must be the account ID, not a random UUID. Changing the partition count later reshuffles keys, so plan partitions up front.

4. Separate "retry" from "give up"

A temporary database timeout should be retried. A malformed message will never succeed, and retrying it forever blocks the whole partition. In Spring Kafka, use a DefaultErrorHandler with a limited back-off and a dead-letter topic:

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);   // sends to <topic>-dlt
    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3)); // 3 retries
    handler.addNotRetryableExceptions(ValidationException.class);  // straight to DLT
    return handler;
}

Then alert on messages landing in the DLT, and have a documented way to replay them. A DLT nobody watches is just a slower way of losing data.

5. Watch out for slow processing and rebalances

If processing a batch takes longer than max.poll.interval.ms (default 5 minutes), Kafka decides the consumer is dead and rebalances. Its messages then get processed again by someone else, which makes things slower still. Symptoms: rising duplicates and constant "rebalancing" logs. Fixes: smaller max.poll.records, faster downstream calls, or moving slow work to its own topic.

6. Consumer lag is your most important metric

CPU and memory can look perfect while a consumer is hours behind. Monitor consumer lag per group and partition (Micrometer/Actuator, Burrow, or your platform's tools), and alert on lag that keeps growing, not on a single spike. In payments, "late" can be as bad as "wrong".

7. Treat message formats as a public API

Producers and consumers deploy at different times. Only make backward-compatible changes: add optional fields, never rename or remove fields in place. A schema registry (Avro/Protobuf/JSON Schema) enforces this automatically. Without one, a code review rule plus contract tests is the minimum.

Quick production checklist

• ☐ Idempotent consumer (unique business ID + DB constraint)
• ☐ Producer: acks=all, idempotence on, stable event ID
• ☐ Key chosen for the ordering you need
• ☐ Limited retries + dead-letter topic + alert + replay process
• ☐ max.poll.records / max.poll.interval.ms tuned to real processing time
• ☐ Consumer lag dashboards and alerts
• ☐ Backward-compatible schema rules

Kafka losing, duplicating or delaying messages?
I design and fix Kafka integrations for Spring Boot systems: retries, dead-letter topics, idempotency, lag and ordering issues. I can review your setup and give you a prioritised fix list, or implement the fixes myself.
See how I work → or email sheikhhameed007@gmail.com