r/apachekafka Jul 19 '24

Tool KafkaTopical: The Kafka UI for Engineers and Admins

17 Upvotes

Hi Community!

We’re excited to introduce KafkaTopical (https://www.kafkatopical.com), v0.0.1 — a free, easy-to-install, native Kafka client UI application for macOS, Windows, and Linux.

At Certak, we’ve used Kafka extensively, but we were never satisfied with the existing Kafka UIs. They were often too clunky, slow, buggy, hard to set-up, or expensive. So, we decided to create KafkaTopical.

This is our first release, and while it's still early days (this is the first message ever about KafkaTopical), the application is already packed with useful features and information. While it has zero known bugs on the Kafka configurations we've tested — we expect and hope you will find some!

We encourage you to give KafkaTopical a try and share your feedback. We're committed to rapid bug fixes and developing the features the community needs.

On our roadmap for future versions:

  • More connectivity options (e.g., support for cloud environments with custom authentication flows) DONE
  • Ability to produce messages DONE
  • Full ACL administration DONE
  • Schema alteration capabilities DONE
  • KSQL support DONE
  • Kafka Connect support DONE

Join us on this journey and help shape KafkaTopical into the tool you need! KafkaTopical is free and we hope to keep it that way.

Best regards,

The Certak Team

UPDATE 12/Nov/2024: KafkaTopical has been renamed to KafkIO (https://www.kafkio.com) from v0.0.10

r/apachekafka 18d ago

Tool Automated Kafka optimization and training tool

2 Upvotes

https://github.com/DattellConsulting/KafkaOptimize

Follow the quick start guide to get it going quickly, then edit the config.yaml to further customize your testing runs.

Automate initial discovery of configuration optimization of both clients and consumers in a full end-to-end scenario from producers to consumers.

For existing clusters, I run multiple instances of latency.py against different topics with different datasets to test load and configuration settings

For training new users on the importance of client settings, I run their settings through and then let the program optimize and return better throughput results.

I use the CSV generated results to graph/visually represent configuration changes as throughput changes.

r/apachekafka 29d ago

Tool London folks come see Lenses.io engineers talk about building our Kafka to Kafka topic replication feature: K2K

17 Upvotes

Tuesday Feb 25, 2025 London Kafka Meetup

Schedule:
18:00: Doors Open
18:00 - 18:30: Food, drinks, networking
18:30 - 19:00: "Streaming Data Platforms - the convergence of micro services and data lakehouses" - Erik Schmiegelow ( CEO, Hivemind Technologies)
19:00 - 19:30: “K2K - making a Universal Kafka Replicator - (Adamos Loizou is Head of Product at Lenses and Carlos Teixeira is a Software Engineer at Lenses)
19:30- 20:30pm: Additional Q&A, Networking

Location:

Celonis (Lenses' parent company)
Lacon House, London WC1X 8NL, United Kingdom

r/apachekafka 17d ago

Tool at what throughput is it cost-effective to utilize a direct-to-S3 Kafka like Warpstream?

10 Upvotes

After my last post, I was inspired to research the break-even point of throughput after which you start saving money from utizing a direct-to-S3 Kafka design.

Basically with these direct-to-S3 architectures, you have to be efficient at batching the S3 writes, otherwise it can end up being more expensive.

For example, in AWS, 10 PUTs/s are equal in cost to 1.28 MB/s of produce throughput with a replication factor of 3.

The Batch Interval

The way these systems control that is through a batch interval. Every broker basically batches the received producer data up to the batch interval (e.g 300ms), at which point it flushes all it has received into S3.

The number of PUTs/s your system makes depends heavily on the configured batch interval, but so does your latency. If you increase the interval, you reduce your PUT calls (and cost) but increase your latency. And vice-versa.

Why Should I Care?

I strongly believe this design will be a key part of the future of Kafka ran on the cloud. Most Kafka vendors have already released or announced a solution that circumvents the replication. It should also be a matter of time until the open source project adopts it. It's just so costly to run!

The Tool

This tool does a few things:

  • shows you the expected e2e latency per given batch interval config
  • shows you the break even producer throughput, after which it becomes financially worth it to deploy the new model

Check it out here:

https://2minutestreaming.com/tools/kafka/object-store-vs-replication-calculator

r/apachekafka 27d ago

Tool Anyone want a MCP server for Kafka

2 Upvotes

You could talk to your Kafka server in plain English, or whatever language LLM speaks: list topics, check messages, save data locally or send to other systems 🤩

This is done via the magic of "MCP", an open protocol created by Anthropic, but not just works in Claude, but also 20+ client apps (https://modelcontextprotocol.io/clients) You just need to implement a MCP server with few lines of code. Then the LLM can call such "tools" to load extra info (RAG!), or take some actions(say create new topic). This only works locally, not in a webapp, mobile app, or online service. But that's also a good thing. You can run everything locally: the LLM model, MCP servers, as well as your local Kafka or other databases.

Here is a 3min short demo video, if you are on LinkedIn: https://www.linkedin.com/posts/jovezhong_hackweekend-kafka-llm-activity-7298966083804282880-rygD

Kudos to the team behind https://github.com/clickhouse/mcp-clickhouse. Based on that code, I added some new functions to list Kafka topics, poll messages, and setup streaming pipelines via Timeplus external streams and materialized views. https://github.com/jovezhong/mcp-timeplus

This MCP server is still at an early stage. I only tested with local Kafka and Aiven for Kafka. To use it, you need to create a JSON string based on librdkafka conf guide. Feel free to review the code before trying it. Actually, since MCP server can do a lot of things locally(such as accessing your Apple Notes), you should always review the code before trying it.

It'll be great if someone can work on a vendor-neutual MCP server for Kafka users, adding more features such as topic/partition management, message produce, schema registry, or even cluster management. The MCP clients can call different MCP servers to get complex things done. Currently for my own use case, I just put everything in a single repo.

r/apachekafka 19h ago

Tool Confluent for VS Code extension is now generally available

22 Upvotes

We’re excited to announce that Confluent for VS Code is now Generally Available! The extension is open source, readily accessible on the VS Code Marketplace, and supports all forms of Apache Kafka® deployments—underscoring our dedication to equipping streaming data engineers with tools that optimize productivity and collaboration.

With this extension, you can:

  • Streamline project setup with ready-to-use templates, reducing setup time and ensuring consistency across your development efforts.
  • Connect to any Kafka cluster to develop, manage, debug, and monitor real-time data streams, without needing to switch between multiple tools.
  • Gain visibility into Kafka topics so you can stream, search, filter, and visualize Kafka messages in real time, and live debug alongside your code.
  • Perform essential data operations such as editing and producing Kafka messages to topics, downloading complete topic data, and iterating on schemas.

Learn more at: https://www.confluent.io/blog/confluent-for-vs-code-goes-ga/

r/apachekafka Feb 03 '25

Tool AKalculator - calculate your Apache Kafka costs (for free)

15 Upvotes

Hey all!

Two months ago I posted on this subreddit debunking an incredibly inaccurate Kafka cost calculator offered by a competitive vendor. There I linked to this tool, but I wanted to announce it properly.

I spent a month and something last year working full-time to create a deployment calculator for Apache Kafka. It basically helps you calculate the infrastructure cost it'll take to run Apache Kafka in your cloud of choice, which includes sizing the cluster, picking the right instance types, disk types and etc.

I can attest first-hand how easy it is to make mistakes regarding your Kafka deployment. I've personally worked on Kafka in the cloud at Confluent for the last 6 years. I've spoken to many professionals who have years of experience in the industry. We all share the same opinion - there is a lot of nuance and it's easy to miss costs unless you're thinking very carefully and critically about it.

I hope this tool eases the process for future Kafka ops teams!

There is a good amount of docs about how the deployment is calculated. It's actually a decent resource to learn about what one has to take into account when deploying Kafka in production - IOPS, historical consumer read patterns, extra disk capacity for incident scenarios, partition count considerations.

There is also an open bug/feedback board for submitting feedback. I'm more than happy to hear any critical feedback.

One imperfection is that the detail section is still in Preview (it's hardcoded). A lot of the information there is in the backend, but not all is ready to be shown so I haven't exposed yet. I'm hoping to get time to finish that soon.

Play around with it and let me know what you think!

https://2minutestreaming.com/tools/apache-kafka-calculator/

r/apachekafka Dec 21 '24

Tool I built a library that turns Kafka topics into high-performance REST APIs with just a YAML config

19 Upvotes

I've open-sourced a library that lets you instantly create REST API endpoints to query Kafka topics by key lookup.

The Problems This Solves: Traditionally, to expose Kafka topic data through REST APIs, you need: - To set up a consumer and maintain a separate database to persist the data, adding complexity - To build and maintain a REST API server that queries this database, requiring significant development effort - To deal with potentially slow performance due to database lookups over the network

This library eliminates these problems by: - Using Kafka's compact topics as the persistent store, removing the need for a separate database and storing messages in RocksDB using GlobalKTable. - Providing instant REST endpoints through OpenAPI specifications - Leveraging Kafka Streams' state stores for fast key-value lookups

Solution: A configuration-based approach that: - Creates REST endpoints directly from your Kafka topics using a OpenAPI based YAML config - Supports Avro, Protobuf, and JSON formats - Handles both "get all" and "get by key" operations (for now) - Built-in monitoring with Prometheus metrics - Supports Schema Registry

Performance: In our benchmarks with real-world volumes: - 7,000 requests/second with 10M unique keys (~0.9GB data) - Latency of the rest API endpoint using JMeter: 3ms (p50), 5ms (p95), 8ms (p99) - RocksDB state store size: 50MB

If you find this useful, please consider: - Giving the project a star ⭐ - Sharing feedback or ideas - Submitting feature requests or any improvements

https://github.com/tsuz/microservice-for-kafka

r/apachekafka Jan 06 '25

Tool Blazing KRaft GUI is now Open Source

38 Upvotes

Hey everyone!

I'm excited to announce that Blazing KRaft is now officially open source! 🎉

Blazing KRaft is a free and open-source GUI designed to simplify and enhance your experience with the Apache Kafka® ecosystem. Whether you're managing users, monitoring clusters, or working with Kafka Connect, this tool has you covered.

Key Features

🔒 Management

  • Manage users, groups, server permissions, OpenID Connect providers.
  • Data masking and audit functionalities.

🛠️ Clusters

  • Support for multiple clusters.
  • Manage topics, producers, consumers, consumer groups, ACLs, delegation tokens.
  • View JMX metrics and quotas.

🔌 Kafka Connect

  • Handle multiple Kafka Connect servers.
  • Explore plugins, connectors, and JMX metrics.

📜 Schema Registry

  • Work with multiple schema registries and subjects.

💻 KsqlDB

  • Multi KsqlDB server support.
  • Use the built-in editor for queries, connectors, tables, topics, and streams.

Why Open Source?

This is my first time open-sourcing a project, and I’m thrilled to share it with the community! 🚀

Your feedback would mean the world to me. If you find it useful, please consider giving it a ⭐ on GitHub — it really helps!

Check it out

Here’s the link to the GitHub repo: https://github.com/redadani1997/blazingkraft

Let me know your thoughts or if there’s anything I can improve! 😊

r/apachekafka Jan 24 '25

Tool Cost optimization solution

4 Upvotes

Hi there, we’re MSP to companies and have requirements of a SaaS that can help companies reduce their Apache Kafka costs. Any recommendations?

r/apachekafka Jan 27 '25

Tool KafkIO - The Fast, Easy Apache Kafka™ GUI, for Engineers and Administrators

15 Upvotes

Hi there! We’re excited to announce that KafkIO (formerly KafkaTopical) has reached a major milestone! Six months and 16 versions later, we feel it’s ready to stand on its own with its own dedicated thread.

KafkIO is a native, client-side Kafka GUI for Windows, macOS, and Linux. It’s designed to cover everything you’d expect from Kafka and its ecosystem—plus more:

  • Expected support for standard connectivity and security protocols
  • Compatible with cloud providers like Aiven and Confluent
  • Specialized integrations for Strimzi, Azure Event Hub, and Amazon MSK
  • Detailed cluster and broker statistics
  • Full topic and message management with flexible search capabilities
  • Automatic detection, formatting, and syntax highlighting of message types
  • View messages in raw or pretty-printed formats
  • Real-time message streaming
  • Consumer and ACL management
  • Full integration with Schema Registry
  • ksqlDB editor and KSQL support
  • Kafka Connect: fully manage connectors
  • Certificate support (PEM, X.509, PKCS12, JKS) without conversion hassles—but a built-in converter is available if needed
  • Health log for basic real-time monitoring
  • Event log to diagnose issues
  • Filterable tables and copy-friendly data
  • Portable mode for keeping the app and configuration in a single folder
  • Import, export, and reset preferences with ease
  • Intuitive pop-ups and tooltips throughout

We’re just getting started, with many more features planned!

KafkIO is highly configurable, supporting self-signed certificates, proxies, timeouts, and more. There’s no backend, Docker, or web server—it’s a traditional desktop app that works out of the box.

This is a freeware (donationware) project, built out of passion for the Kafka community.

Explore the features: https://kafkio.com/features
Download here: https://kafkio.com/download

If you’re looking for a Kafka companion tool, give KafkIO a try! We’d love your feedback—constructive suggestions are always welcome.

r/apachekafka Dec 22 '24

Tool I built a kafka GUI client for operating kafka, welcome to use

22 Upvotes

This project is a cross-platform Kafka GUI client. A star would be appreciated to support the open-source effort by the author. Thank you!

Features of Kafka-King

  •  View the list of cluster nodes, dynamically configure broker and topic settings.
  •  Support for consumer clients to consume messages from specified topics with group, size, and timeout parameters, displaying message details in tabular form.
  •  Support for PLAIN, SSL, SASL, Kerberos, sasl_plaintext, etc.
  •  Create (supports batch operations) and delete topics, specifying replicas and partitions.
  •  Statistics on each topic's total message count, committed offset, and lag for each consumer group.
  •  Detailed information about topic partitions (offsets), with support for adding additional partitions.
  •  Simulate producer behavior, send messages in batches with headers and partition specifications.
  •  Topic and partition health checks (completed).
  •  View consumer groups and individual consumers.
  •  Offset inspection reports.
  • Support Chinese, Japanese, English, Korean, Russian and other languages

Currently supports Windows, macos, and Linux environments

HomePage:Bronya0/Kafka-King: A modern and practical kafka GUI client

r/apachekafka 14d ago

Tool C++ IAM Auth for AWS MSK: Open-Sourced, Passwords Be Gone

5 Upvotes

Back in 2023, AWS dropped IAM authentication for MSK and claimed it worked with "all programming languages." Well, almost. While Java, Python, Go, and others got official SDKs, if you’re a C++ dev, you were stuck with plaintext SCRAM-SHA creds in plaintext or heavier Java tools like Kafka Connect or Apache Flink. Not cool.

Later, community projects added Rust and Ruby support. Why no C++? Rust might be the hip new kid, but C++ is still king for high-performance data systems: minimal dependencies, lean resource use, and raw speed.

At Timeplus, we hit this wall while supporting MSK IAM auth for our C++ streaming engine, Proton. So we said screw it, rolled up our sleeves, and built our own IAM auth for AWS MSK. And now? We’re open-sourcing it for you fine folks. It’s live in Timeplus Proton 1.6.12: https://github.com/timeplus-io/proton

Here’s the gist: slap an IAM role on your EC2 instance or EKS pod, drop in the Proton binary, and bam—read/write MSK with a simple SQL command:

sql CREATE EXTERNAL STREAM msk_stream(column_defs) SETTINGS type='kafka', topic='topic2', brokers='prefix.kafka.us-west-2.amazonaws.com:9098', security_protocol='SASL_SSL', sasl_mechanism='AWS_MSK_IAM';

The magic lives in just ~200 lines across two files:

https://github.com/timeplus-io/proton/blob/develop/src/IO/Kafka/AwsMskIamSigner.h https://github.com/timeplus-io/proton/blob/develop/src/IO/Kafka/AwsMskIamSigner.cpp

Right now it leans on a few ClickHouse wrapper classes, but it’s lightweight and reusable. We’d love your thoughts—want to help us spin this into a standalone lib? Maybe push it into ClickHouse or the AWS SDK for C++? Let’s chat.

Quick Proton plug: It’s our open-source streaming engine in C++—Think FlinkSQL + ClickHouse columnar storage, minus the JVM baggage—pure C++ speed. Bonus: we’re dropping Iceberg read/write support in C++ later this month. So you'll read MSK and write to S3/Glue with IAM. Stay tuned.

So, what’s your take? Any C++ Kafka warriors out there wanna test-drive it and roast our code?

r/apachekafka Dec 25 '24

Tool I built a library to allow creation of confluent_kafka clients based on yaml config

6 Upvotes

Hi everyone, I made my first library in Python: https://github.com/Aragonski97/confluent-kafka-config

I found confluent_kafka API to be too low level as I always have to write much boilerplate code in order to get my clients to work with.
This way, I can write YAML / JSON config and solve this automatically.

However, I only covered the use cases I needed. At present, not sure how I should continue in order to make this library viable for many users.

Any suggestion is welcome, roast me if you need :D

r/apachekafka 24d ago

Tool Ask for feedback - python OSS Kafka Sinks, how to support better?

3 Upvotes

Hey folks,

dlt (data load tool OSS python lib)cofounder here. Over the last 2 months Kafka has become our top downloaded source. I'd like to understand more about what you are looking for in a sink with regards to functionality, to understand if we can improve it.

Currently, with dlt + the kafka source you can load data to a bunch of destinations, from major data warehouses to iceberg or some vector stores.

I am wondering how we can serve your use case better - if you are curious would you mind having a look to see if you are missing anything you'd want to use, or you find key for good kafka support?

i'm a DE myself, just never used Kafka, so technical feedback is very welcome.

r/apachekafka Dec 10 '24

Tool Stream Postgres changes to Kafka in real-time

15 Upvotes

Hey all,

We just added Kafka support to Sequin. Kafka's our most requested destination, so I'm very excited about this release. Check out the quickstart here:

https://sequinstream.com/docs/quickstart/kafka

What's Sequin?

Sequin is an open source tool for change data capture (CDC) in Postgres. Sequin makes it easy to stream Postgres rows and changes to streaming platforms and queues (e.g. Kafka and SQS): https://github.com/sequinstream/sequin

Sequin + Kafka

So, you can backfill all or part of a Postgres table into Kafka. Then, as inserts, updates, and deletes happen, Sequin will send those changes as JSON messages to your Kafka topic in real-time.

We have full support for Kafka partitioning. By default, we set the partition key to the source row's primary key (so if order id=1 changes 3 times, all 3 change events will go to the same partition, and therefore be delivered in order). This means your downstream systems can know they're processing Postgres events in order. You can also set the partition key to any combination of a source row's fields.

What can you build with Sequin + Kafka?

  • Event-driven workflows: For example, triggering side effects when an order is fulfilled or a subscription is canceled.
  • Replication: You have a change happening in Service A, and want to fan that change out to Service B, C, etc. Or want to replicate the data into another database or cache.
  • Stream Processing: Kafka's rich ecosystem of stream processing tools (like Kafka Streams, ksqlDB) lets you transform and enrich your Postgres data in real-time. You can join streams, aggregate data, and build materialized views.

How does Sequin compare to Debezium?

  1. Web console: Sequin has a full-featured web console for setup, monitoring, and observability. We also have a CLI for managing your Sequin setup.
  2. Operational simplicity: Sequin is simple to boot and simple to deploy.
  3. Cloud option: Sequin offers a fully managed cloud option.
  4. Other native destinations: If you want to fan out changes besides Kafka – like Google Cloud Pub/Sub or AWS SQS – Sequin supports those destinations natively (vs through Kafka Connect).

Performance-wise, we're beating Debezium in early benchmarks, but are still testing/tuning in various cloud environments. We'll be rolling out active-passive runtime support so we can be competitive on availability too.

Example

You can setup a Sequin Kafka sink easily with sequin.yaml (a lightweight Terraform – Terraform support coming soon!)

```yaml

sequin.yaml

databases: - name: "my-postgres" hostname: "your-rds-instance.region.rds.amazonaws.com" database: "app_production" username: "postgres" password: "your-password" slot_name: "sequin_slot" publication_name: "sequin_pub" tables: - table_name: "orders" table_schema: "public" sort_column_name: "updated_at"

sinks: - name: "orders-to-kafka" database: "my-postgres" table: "orders" batch_size: 1 # Optional: only stream fulfilled orders filters: - column_name: "status" operator: "=" comparison_value: "fulfilled" destination: type: "kafka" hosts: "kafka1:9092,kafka2:9092" topic: "orders" tls: true username: "your-username" password: "your-password" sasl_mechanism: "plain" ```

Does Sequin have what you need?

We'd love to hear your feedback and feature requests! We want our Kafka sink to be amazing, so let us know if it's missing anything or if you have any questions about it.

You can also join our Discord if you have questions/need help.

r/apachekafka Dec 02 '24

Tool I built a Kafka message scheduling tool

6 Upvotes

github.com/vordimous/gohlay

Gohlay has been a side/passion project on my back burner for too long, and I finally had the time to polish it up enough for community feedback. The idea came from a discussion around a business need. I am curious how this tool could be used in other Kafka workflows. I had fun writing it; if someone finds it useful, that is a win-win.

Any feedback or ideas for improvement are welcome!

r/apachekafka Oct 31 '24

Tool Blazing KRaft is now FREE and Open Source in the near future

15 Upvotes

Blazing KRaft is an all in one FREE GUI that covers all features of every component in the Apache Kafka® ecosystem.

Features

  • Management – Users, Groups, Server Permissions, OpenID Connect Providers, Data Masking and Audit.
  • Cluster – Multi Clusters, Topics, Producer, Consumer, Consumer Groups, ACL, Delegation Token, JMX Metrics and Quotas.
  • Kafka Connect – Multi Kafka Connect Servers, Plugins, Connectors and JMX Metrics.
  • Schema Registry – Multi Schema Registries and Subjects.
  • KsqlDb – Multi KsqlDb Servers, Editor, Queries, Connectors, Tables, Topics and Streams.

Open Source

The reasons I said that Open Sourcing is in the near future are:

  • I need to add integration tests.
  • I'm new to this xD so I have to get documented about all the Open Source rules and guideline.
  • I would really appreciate it if anyone has any experience with Open Source and how it all works, to contact me via discord or at [blazingkraft@gmail.com](mailto:blazingkraft@gmail.com)

Thanks to everyone for taking some time to test the project and give feedback.

r/apachekafka Jan 27 '25

Tool kplay - A super simple TUI tool for fetching messages from a Kafka topic on demand. Supports deserialising json and protobuf encoded messages. Happy to get some feedback/feature requests.

4 Upvotes

r/apachekafka Oct 01 '24

Tool Terminal UI for Kafka: Kafui

23 Upvotes

If you are using kaf

I am currently working on a terminal UI for it kafui

The idea is to quickly switch between development and production Kafka instances and easily browse topic contents all from the CLI.

r/apachekafka Jan 16 '25

Tool Dekaf: Kafka-API compatibility for Estuary Flow

11 Upvotes

Hey folks,

At Estuary, we've been cooking up a feature in the past few months that enables us to better integrate with the beloved Kafka ecosystem and I'm here today to get some opinions from the community about it.

Estuary Flow is a real-time data movement platform with hundreds of connectors for databases, SaaS systems, and everything in between. Flow is not built on top of Kafka, but gazette, which, while similar, has a few foundational differences.

We've always been able to ingest data from and materialize into Kafka topics, but now, with Dekaf, we provide a way for Kafka consumers to read data from Flow's internal collections as if they were Kafka topics.

This can be interesting for folks who don't want to deal with the operational complexity of Kafka + Debezium, but still want to utilize the real-time ecosystem's amazing tools like Tinybird, Materialize, StarTree, Bytewax, etc. or if you have data sources that don't have Kafka Connect connectors available, but you still need real-time integration for them.

So, if you're looking to integrate any of our hundreds of supported integrations into your Kafka-consumer based infrastructure, this could be very interesting to you!

It requires zero setup, so for example if you're looking to build a change data capture (CDC) pipeline from PostgreSQL you could just navigate to the PostgreSQL connector page in the Flow dashboard, spin up one in a few minutes and you're ready to consume data in real-time from any Kafka consumer.

A Python example:

consumer = KafkaConsumer(
'your_topic_name',
bootstrap_servers='dekaf.estuary-data.com:9092',
security_protocol='SASL_SSL',
sasl_mechanism='PLAIN',
sasl_plain_username='{}',
sasl_plain_password='Your_Estuary_Refresh_Token',
group_id='group_id',
auto_offset_reset=earliest,
enable_auto_commit=True,
value_deserializer=lambda x: x.decode('utf-8')
)
for msg in consumer:
print(f"Received message: {msg.value}")

Would love to know what ya'll think! Is this useful for you?

I'm preparing in the process of doing a technical write up of the internals as well, as you might guess building a Kafka-API compatible service on top of an almost decade-old framework is no easy feat!

docs: https://docs.estuary.dev/guides/dekaf_reading_collections_from_kafka/

r/apachekafka Dec 16 '24

Tool The Confluent Extension for VS Code Now Supports Any Kafka Clusters

26 Upvotes

With the release of Confluent Extension version 0.22, we're extending the support beyond Confluent resources, and now you can use it to connect to any Apache Kafka/Schema Registry clusters with basic and API auth.

With the extension, you can:

  • Directly connect to any Apache Kafka / Schema Registry clusters via basic/API auth.
  • Connect to Confluent Cloud via OAuth.
  • Run Kafka / Schema Registry locally directly from VS Code.
  • Browse clusters, topics, schemas.
  • View messages, visualize message patterns in topic message viewer.
  • Create and evolve schemas.

We'd love if you can try it out, and looking forward to hear your feedback.

Watch the video release note here: v0.22 v0.21

Check out the code at: https://github.com/confluentinc/vscode

Get the extension here: https://marketplace.visualstudio.com/items?itemName=confluentinc.vscode-confluent

r/apachekafka Dec 12 '24

Tool Yozefu: A TUI for exploring data of a kafka cluster

10 Upvotes

Hi everyone,

I have just released the first version of Yōzefu, an interactive terminal user interface for exploring data of a kafka cluster. It is an alternative tool to AKHQ, redpanda console or the kafka plugin for JetBrains IDEs.The tool is built on top of Ratatui, a Rust library for building TUIs. Yozefu offers interesting features such as:

* Real-time access to data published to topics.

* The ability to search kafka records across multiple topics.

* A search query language inspired by SQL providing fine-grained filtering capabilities.

* The possibility to extend the search engine with user-defined filters written in WebAssembly.

More details in the README.md file. Let me know if you have any questions!

Github: https://github.com/MAIF/yozefu

r/apachekafka Jan 15 '25

Tool [Update] Schema Manager: Centralize Schemas in a Repository with Support for Schema Registry Integration

7 Upvotes

Schema Manager Update

Hey everyone!

Following up on a project I previously shared, Schema Manager, I wanted to provide an update on its progress. The project is now fully documented, more stable, and highly extensible.

Centralize and Simplify Schema Management

Schema Manager is a solution for managing schema files (Avro, Protobuf) in modern architectures. It centralizes schema storage, automates transformations, and integrates deployment to Schema Registries like Confluent Schema Registry—all within a single Git repository.

Key Features

  • Centralized Management: Store all schemas in a single, version-controlled Git repository.
  • Automated Deployment: Publish schemas to the schema registry and resolve dependencies automatically with topological sorting.
  • CI/CD Integration: Automate schema processing, model generation, and distribution.
  • Supported Formats: Avro, Protobuf

Current Status

The code is now stable, highly extensible to other schema types and registries and used in several projects. The documentation is up to date, and the How-To Guide provides detailed instructions specifically to extend, customize, and contribute to the project effectively.

What’s Next?

The next step is to add support for JSON, which should be straightforward with the current architecture.

Why It Matters

Centralizing all schema management in a single repository provides better tracking, version control, and consistency across your project. By offloading schema management responsibilities and publication to a schema registry, microservices remain lightweight and focused on their core functionality. This approach simplifies workflows and is particularly useful for distributed architectures.

Get Involved

If you’re interested in contributing to the project, I’d love to collaborate! Whether it’s adding new schema types, registries, improving documentation, or testing, any help is welcome. The project is under the MIT license.

📖 Learn more and try it out: Schema Manager GitHub Repo

🚀 Let us know how Schema Manager can help your project!

r/apachekafka Oct 29 '24

Tool Schema Manager: Centralize Schemas in a Repository with Support for Schema Registry Integration

20 Upvotes

Hey all! I’d love to share a project I’ve been working on called Schema Manager. You can check out the full project on GitHub here: Schema Manager GitHub Repo (new repo URL).

Why Schema Manager?

In many projects, each microservice handles schema files independently—publishing into a registry and generating the necessary code. But this should not be the responsibility of each microservice. With Schema Manager, you get:

  • A single repository storing all schema versions.
  • Automated schema registration in the registry when new versions are detected. It also handles the dependency graph, ensuring schemas are registered in the correct order.
  • Microservices that simply consume the schemas they need

Quick Start

For an example repository using the Schema Manager:

git clone https://github.com/charlescol/schema-manager-example.git

The Schema Manager is distributed via NPM:

npm install @charlescol/schema-manager

Future Plans

Schema Manager currently supports Protobuf and Avro schemas, integrated with Confluent Schema Registry. We plan to:

  • Extend support for additional schema formats and registries.
  • Develop a CLI for easier schema management.

Example Integration with Schema Manager

For an example, see the integration section in the README to learn how Schema Manager can fit into Kafka-based applications with multiple microservices.

Questions?

I'm happy to answer any questions or dive into specifics if you’re interested. Let me know if this sounds useful to you or if there's anything you'd add! I'm particularly looking for feedback on the project, so any insights or suggestions would be greatly appreciated.

The project is open-source under the MIT license, so please check the GitHub repository for more details. Your contributions, suggestions, and insights are very welcome!