- Home
- Skills
- APIs & Backend
- Event Driven Kafka Producer Consumer
Works with the AI Agents you already use
Event Driven Kafka Producer Consumer
Build reliable Kafka pipelines in Node.js and Python with idempotent producers, strict ordering, and DLQ support.
$15
Event Driven Kafka Producer Consumer
Example session with this skill installed
Generate a reliable Node.js Kafka producer and consumer for a payment processing system. I need strict message ordering by paymentId and a way to handle failed messages without losing them.
- Read your context and instructions
- Compiled the event driven kafka
I have architected a robust Node.js Kafka pipeline using KafkaJS. The producer is configured for idempotency to prevent duplicate payments, and the consumer implements a manual commit strategy with a Dead Letter Queue (DLQ) for unrecoverable failures. ## Kafka Payment Processing Implementation ```javascript // producer.js const { Kafka } = require('kafkajs'); const kafka = new Kafka({ clientId: 'payment-service', brokers: [process.env.KAFKA_BROKER] }); const producer = kafka.producer({ allowAutoTopicCreation: false, transactionalId: 'payment-producer-v1', maxInFlightRequests: 1, idempotent: true // enable.idempotence=true }); async function sendPaymentEvent(payment) { await producer.connect(); await producer.send({ topic: 'payment-events', acks: -1, // acks=all messages: [ { key: payment.paymentId, // Partitioning by entity key for ordering value: JSON.stringify(payment) } ], }); } // consumer.js const consumer = kafka.consumer({ groupId: 'payment-group' }); async function startConsumer() { await consumer.connect(); await consumer.subscribe({ topic: 'payment-events', fromBeginning: true }); await consumer.run({ autoCommit: false, // Manual commit management eachMessage: async ({ topic, partition, message }) => { try { const paymentData = JSON.parse(message.value.toString()); await processPayment(paymentData); // Commit only after successful processing await consumer.commitOffsets([ { topic, partition, offset: (BigInt(message.offset) + 1n).toString() } ]); } catch (error) { console.error('Processing failed, routing to DLQ', error); await producer.send({ topic: 'payment-events-dlq', messages: [{ key: message.key, value: message.value }] }); } }, }); } ``` **Next steps** 1. Add the `KAFKA_BROKER` address to your `.env` file based on the `.env.example` provided. 2. Verify that the `payment-events-dlq` topic is created in your Kafka cluster. 3. Run the automated evaluation suite to test the consumer group rebalancing behavior.
Connects securely to your tools. The creator never sees your data.
About this skill
Distributed messaging often fails due to lost messages, out-of-order processing, or poison pills that crash consumers. This skill provides production-ready Apache Kafka producer and consumer implementations in Node.js or Python, enforcing reliability through idempotent delivery and dead-letter queues. ## What it does - **Idempotent Producers** ensures messages are delivered exactly once even during retries by using `acks=all`. - **Key-Based Partitioning** maintains strict message ordering for specific entities like `userId` or `orderId`. - **Manual Commit Management** prevents data loss by acknowledging offsets only after successful message processing. - **Dead Letter Queues (DLQ)** isolates unrecoverable messages to prevent pipeline blockage and enable debugging. ## How it works 1. **Initialize Producer** with `enable.idempotence=true` and automated retry logic for high-availability writes. 2. **Define Routing** by business entity keys to ensure sequential processing within specific partitions. 3. **Configure Consumer Groups** for horizontal scaling and fault-tolerant offset tracking. 4. **Implement Error Handling** that automatically routes failed payloads to a designated DLQ topic. ## Frameworks & tools Uses KafkaJS for Node.js environments or Confluent-Kafka for Python. It follows standard patterns for distributed systems messaging and pub/sub architectures. ## Why this beats prompting it yourself Writing reliable Kafka code requires deep knowledge of configuration flags that prevent silent failures. This skill enforces strict ordering and fault tolerance by default, saving you from troubleshooting partition skews or data duplication in production. ## Use cases - Build resilient order processing pipelines for e-commerce backends. - Implement real-time user activity tracking with strict sequential requirements. - Create fault-tolerant data ingestion layers for analytics platforms. - Manage unrecoverable message failures without stopping the entire stream. ## Known limitations Focuses on application-level logic for KafkaJS and Python. It does not handle infrastructure provisioning or Kafka cluster management.
Focuses on KafkaJS (Node.js) and Python (Confluent-Kafka). Does not perform infrastructure setup, cluster provisioning, or global system package installation.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 days ago
Permissions
Target Use Cases: - Build an event-driven Kafka producer and consumer in Node.js. - Configure Kafka consumer group with Dead Letter Queue handling. - Implement idempotent message publishing in Apache Kafka.