log analyzer kafka

    1

    Expert diagnostic tool for Apache Kafka logs, identifying broker failures, consumer lag, and KRaft/ZooKeeper issues.

    $1

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    log analyzer kafka

    Example session with this skill installed

    Analiza este log de un broker en PROD: [2023-10-27 10:15:01,234] ERROR [ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error in response with solr-logs-0 (kafka.server.ReplicaFetcherThread)

    • Read your context and instructions
    • Compiled the log analyzer kafka

    Analicé los logs del broker proporcionados. Se identifica un problema de conectividad entre réplicas en un entorno de producción.

    REPORTE DE ANÁLISIS DE LOG — APACHE KAFKA

    ╔══════════════════════════════════════════════════════════════╗
    ║ REPORTE DE ANÁLISIS DE LOG — APACHE KAFKA ║
    ╚══════════════════════════════════════════════════════════════╝

    📋 RESUMEN EJECUTIVO
    Tecnología: Apache Kafka [3.4.0] / [ZooKeeper]
    Entorno: PROD
    Componente(s): Broker (ReplicaFetcherThread)
    Período log: 2023-10-27 10:15:01
    Total eventos: 1 crítico | 0 warnings | 0 notables
    Diagnóstico breve: El seguidor (Broker 1) no puede buscar datos del líder (Broker 2) para el topic 'solr-logs'.

    🔴 HALLAZGOS CRÍTICOS

    ReplicaFetcher Error: El hilo de recuperación de réplicas falló al intentar sincronizar la partición solr-logs-0. Esto resultará en under-replicated partitions si persiste.

    Network/Auth: El error en ReplicaFetcherThread suele indicar problemas de red entre brokers o fallos en la autenticación SASL interna.

    ⚠️ DATOS SENSIBLES

    • Endpoint interno detectado: <span style="color:red;font-weight:bold">leaderId=2</span> (Requiere mapear a hostname interno).

    ❓ PREGUNTAS ABIERTAS / 🛠️ PRÓXIMOS PASOS

    1. ¿Cuál es el replication.factor del topic solr-logs?
    2. Revisar logs del Broker 2 para verificar si el líder está rechazando conexiones o sufriendo pausas de GC.
    3. Verificar conectividad en el puerto 9092/9093 entre el Broker 1 y el Broker 2.

    Next steps

    • Verifique si el Broker 2 está bajo carga extrema o si el proceso ha muerto.
    • Ejecute kafka-topics --describe para confirmar el estado de ISR de la partición afectada.
    • ¿Deseas el reporte como documento entregable?

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Diagnosing distributed systems like Apache Kafka through raw logs is a manual, error-prone grind. This skill automates the identification of cluster health issues, consumer lag, and connectivity failures across the entire Kafka ecosystem.

    What it does

    • Multi-component parsing processes logs from brokers, controllers, Connect, Streams, and clients (producer/consumer).
    • Architecture detection identifies whether the cluster uses ZooKeeper or KRaft to contextualize leader election and quorum issues.
    • Error pattern matching detects ISR shrinkage, offset ranges, rebalancing loops, and SASL/SSL handshake failures.
    • PII/Security auditing flags sensitive data like JAAS configs, keystore passwords, and internal endpoints in red.
    • Resource monitoring spots Disk I/O bottlenecks, GC pauses, and OOM errors affecting broker performance.

    How it works

    1. Log Ingestion involves pasting raw logs containing markers like kafka.server or org.apache.kafka.clients.
    2. Context Elicitation triggers specific questions about environment (PROD/DEV), replication factors, and coordination modes.
    3. Structured Analysis maps log events to known patterns like CommitFailedException or under-replicated partitions.
    4. Report Generation outputs a formal diagnostic report including critical findings and open questions.

    Frameworks & tools

    Apache Kafka (2.x/3.x), ZooKeeper, KRaft, Kafka Connect, Kafka Streams, Confluent Platform, Strimzi, and Java-based Kafka clients.

    Why this beats prompting it yourself

    Standard LLM prompts often hallucinate the severity of under-replicated partitions without knowing the replication factor. This skill follows a strict diagnostic framework that checks for environment-specific nuances and specific Kafka configuration limits.

    Use cases

    • Troubleshooting sudden consumer group rebalancing loops in production.
    • Validating KRaft migration leader election issues.
    • Auditing Kafka Connect task failures and Schema Registry incompatibilities.
    • Identifying PII leakage in DEBUG logs before they hit storage.

    Known limitations

    Requires specific log markers to trigger; cannot analyze binary data or direct JMX metrics without log-based representation.

    Requires log markers like kafka.server or org.apache.kafka.clients. Cannot diagnose under-replicated partitions accurately without user-provided replication factor data.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 28 days ago

    Permissions

    No special permissions declared or detected

    Listed28 days ago

    What's inside

    Frequently Asked Questions

    This skill acts as a specialized diagnostic engine that maps raw Kafka logs to specific distributed system failures, such as ISR shrinkage or KRaft quorum issues, which generic AI prompts often misinterpret or hallucinate.

    The analyzer supports logs from Apache Kafka versions 2.x through 3.x, including both legacy ZooKeeper-based clusters and modern KRaft architectures.

    No complex installation is required; simply provide the skill with raw log output containing standard Kafka markers (like 'kafka.server') to trigger the structured diagnostic workflow.

    Yes, the skill automatically scans for and red-flags sensitive internal data such as JAAS configurations, keystore passwords, and internal endpoints to ensure your diagnostic reports are audit-ready.

    The skill is designed to process textual log data and cannot directly ingest binary files, JMX metrics, or live telemetry streams unless they are represented in a log format.

    Purchase includes full access to the diagnostic skill, supporting analysis for Kafka Brokers, Connect, Streams, and Java-based clients across production and development environments.

    New in Testing & Debugging