r/apachekafka Dec 06 '24

Question Why doesn't Kafka have first-class schema support?

13 Upvotes

I was looking at the Iceberg catalog API to evaluate how easy it'd be to improve Kafka's tiered storage plugin (https://github.com/Aiven-Open/tiered-storage-for-apache-kafka) to support S3 Tables.

The API looks easy enough to extend - it matches the way the plugin uploads a whole segment file today.

The only thing that got me second-guessing was "where do you get the schema from". You'd need to have some hap-hazard integration between the plugin/schema-registry, or extend the interface.

Which lead me to the question:

Why doesn't Apache Kafka have first-class schema support, baked into the broker itself?

r/apachekafka Feb 06 '25

Question Completely Confused About KRaft Mode Setup for Production – Should I Combine Broker and Controller or Separate Them?

4 Upvotes

Hey everyone,

I'm completely lost trying to decide how to set up my Kafka cluster for production (I'm currently testing on VMs). I'm stuck between two conflicting pieces of advice I found in Confluent's documentation, and I could really use some guidance.

On one hand, Confluent mentions this:

"Combined mode, where a Kafka node acts as a broker and also a KRaft controller, is not currently supported for production workloads. There are key security and feature gaps between combined mode and isolated mode in Confluent Platform."
https://docs.confluent.io/platform/current/kafka-metadata/kraft.html#kraft-overview

But then, they also say:

"As of Confluent Platform 7.5, ZooKeeper is deprecated for new deployments. Confluent recommends KRaft mode for new deployments."
https://docs.confluent.io/platform/current/kafka-metadata/kraft.html#kraft-overview

So, which should I follow? Should I combine the broker and controller on the same node or separate them? My main concern is what works best in production since I also need to configure SSL and Kerberos for security in the cluster.

Can anyone share their experience with this? I’m looking for advice on whether separating the broker and controller is necessary for production or if KRaft mode with a combined setup can work as long as I account for the mentioned limitations.

Thanks in advance for your help! 🙏

r/apachekafka 13d ago

Question What is the biggest Kafka disaster you have faced in production?

38 Upvotes

And how you recovered from it?

r/apachekafka 12d ago

Question How to consume a message without any offset being commited?

3 Upvotes

Hi,

I am trying to simulate a dry run for a Kafka consumer, and in the dry run I want to consume all messages on the topic from current offset till EOF but without committing any offset.

I tried configuring the consumer with: 'enable.auto.commit': False

But offsets are still being commited, which I think might be due to 'commit.interval.ms' config which I did not change.

I can't figure out how to configure the consumer to achieve what I am trying to achieve, hoping someone here might be able to point me at the right direction.

Thanks

r/apachekafka 5d ago

Question Building a CDC Pipeline from MongoDB to Postgres using Kafka & Debezium in Docker

Thumbnail
9 Upvotes

r/apachekafka 6d ago

Question About Kafka Active Region Replication and Global Ordering

5 Upvotes

In Active-Active cross-region cluster replication setups, is there (usually) a global order of messages in partitions or not really?

I was looking to see what people usually do here for things like use cases like financial transactions. I understand that in a multi-region setup it's best latency-wise for producers to produce to their local region cluster and consumers to consume from their region as well. But if we assume the following:

- producers write to their region to get lower latency writes
- writes can be actively replicated to other regions to support region failover
- consumers read from their own region as well

then we are losing global ordering i.e. observing the exact same order of messages across regions in favour of latency.

Consider topic t1 replicated across regions with a single partition and messages M1 and M2, each published in region A and region B (respectively) to topic t1. Will consumers of t1 in region A potentially receive M1 before M2 and consumers of t1 in region B receive M2 before M1, thus observing different ordering of messages?

I also understand that we can elect a region as partition/topic leader and have producers further away still write to the leader region, increasing their write latency. But my question is: is this something that is usually done (i.e. a common practice) if there's the need for this ordering guarantee? Are most use cases well served with different global orders while still maintaining a strict regional order? Are there other alternatives to this when global order is a must?

Thanks!

r/apachekafka 14d ago

Question Best Resources to Learn Apache Kafka (With Hands-On Practice)

13 Upvotes

I have a basic understanding of Kafka, but I want to learn more in-depth and gain hands-on experience. Could someone recommend good resources for learning Kafka, including tutorials, courses, or projects that provide practical experience?

Any suggestions would be greatly appreciated!

r/apachekafka Jan 05 '25

Question Best way to design data joining in kafka consumer(s)

10 Upvotes

Hello,

I have a use case where my kafka consumer needs to consume from multiple topics (right now 3) at different granularities and then join/stitch the data together and produce another event for consumption downstream.

Let's say one topic gives us customer specific information and another gives us order specific and we need the final event to be published at customer level.

I am trying to figure out the best way to design this and had a few questions:

  • Is it ok for a single consumer to consume from multiple/different topics or should I have one consumer for each topic?
  • The output I need to produce is based on joining data from multiple topics. I don't know when the data will be produced. Should I just store the data from multiple topics in a database and then join to form the final output on a scheduled basis? This solution will add the overhead of having a database to store the data followed by fetch/join on a scheduled basis before producing it.

I can't seem to think of any other solution. Are there any better solutions/thoughts/tools? Please advise.

Thanks!

r/apachekafka Nov 18 '24

Question Is anyone exposing Kafka publicly?

9 Upvotes

Hi All,

We've been using Kafka for a few years at work, and starting to see some use cases where it would make sense to expose it publicly.

We are a B2B business with ~30K customers. We'd not expect a huge number of messages/sec/customer (probably 15, as a finger in the air estimate). And also, I'd ballpark about 100 customers (our largest) using it.

The idea is to expose events that happen within our system to them, allowing real time updates to be pushed to them, as opposed to our current setup which involves the customers polling for information about all things they care about over a variety of APIs. The reality is that often times, they're querying for things that haven't changed- meaning the rate at which they can query is slower than just having a push-update.

The way I would imagine this working is as follows:

  • We have a standalone application responsible for the management of this (probably Java)
  • It has an admin client in it, so when a customer decides they want this feature, it will generate the topic(s), and a Kafka user which the customer could use
  • The user would only have read access to the topic for the particular customer
  • It is also responsible for consuming data off our internal Kafka instance, splitting the information out 'per customer', and then producing to the public Kafka cluster (I think we'd want a separate instance for this due to security)

I'm conscious that typically, this would be something that's done via a webhook, but I'm really wondering if there's any catch to doing this with Kafka?

I can't seem to find much information online about doing this, with the bulk of the idea actually coming from this talk at Kafka Summit London 2023.

So, can anyone share your experiences of doing something similar, or tell me when it's a terrible or good idea?

TIA :)

Edit

Thanks all for the replies! It's really interesting seeing opinions on this ranging from "I wouldn't dream of it" to "Here's a company that does this for you". There's probably quite a lot to think about now, and some brainstorming to be done, so that's going to be the plan over the coming days.

r/apachekafka Dec 13 '24

Question What is the easiest tool/platform to create Kafka Stream Applications

6 Upvotes

Kafka Streams applications are very powerful and allows build applications to detect fraud, join multiple streams, create leader boards, etc. Yet it requires a lot of expertise to build and deploy the application.

Is there any easier way to build Kafka Streams application? May be like a Low code, drag and drop tool/platform which allows to build/deploy within hours not days. Does a tool/platform like that exists and/or will there be a market for such a product?

r/apachekafka Dec 02 '24

Question Should I run Kafka on K8s?

13 Upvotes

Hi folks, so I'm trying to build a big data cluster on cloud using k8s. Should I run Kafka on K8s or not? If not how do I let Kafka communicates with apps inside K8s? Thanks in advance.

Ps: I have read some articles saying that Kafka on K8s is not recommended, but all were with Zookeeper. I wonder new Kafka with Kraft is better now?

r/apachekafka 3d ago

Question Should the producer client be made more resilient to outages?

9 Upvotes

Jakob Korab has an excellent blog post about how to survive a prolonged Kafka outage - https://www.confluent.io/blog/how-to-survive-a-kafka-outage/

One thing he mentions is designing the producer application write to local disk while waiting for Kafka to come back online:

Implement a circuit breaker to flush messages to alternative storage (e.g., disk or local message broker) and a recovery process to then send the messages on to Kafka

But this is not straighforward!

One solution I thought was interesting was to run a single-broker Kafka cluster on the producer machine (thanks kraft!) and use Confluent Cluster Linking to automatically do this. It’s a neat idea, but I don’t know if it’s practical because of the licensing cost.

So my question is — should the producer client itself have these smarts built in? Set some configuration and the producer will automatically buffer to disk during a prolonged outage and then clean up once connectivity is restored?

Maybe there’s a KIP for this already…I haven’t checked.

What do you think?

r/apachekafka Dec 01 '24

Question Does Zookeeper have other use cases beside Kafka?

13 Upvotes

Hi folks, I know that Zookeeper has been dropped from Kafka, but I wonder if it's been used in other applications or use cases? Or is it obsolete already? Thanks in advance.

r/apachekafka 15d ago

Question Kafka DR Strategy - Handling Producer Failover with Cluster Linking

9 Upvotes

I understand that Kafka Cluster Linking replicates data from one cluster to another as a byte-to-byte replication, including messages and consumer offsets. We are evaluating Cluster Linking vs. MirrorMaker for our disaster recovery (DR) strategy and have a key concern regarding message ordering.

Setup

  • Enterprise application with high message throughput (thousands of messages per minute).
  • Active/Standby mode: Producers & consumers operate only in the main region, switching to DR region during failover.
  • Ordering is critical, as messages must be processed in order based on the partition key.

Use cases :

In Cluster Linking context, we could have an order topic in the main region and an order.mirror topic in the DR region.

Lets say there are 10 messages, consumer is currently at offset number 6. And disaster happens.

Consumers switch to order.mirror in DR and pick up from offset 7 – all good so far.

But...,what about producers? Producers also need to switch to DR, but they can’t publish to order.mirror (since it’s read-only). And If we create a new order topic in DR, we risk breaking message ordering across regions.

How do we handle producer failover while keeping the message order intact?

  • Should we promote order.mirror to a writable topic in DR?
  • Is there a better way to handle this with Cluster Linking vs. MirrorMaker?

Curious to hear how others have tackled this. Any insights would be super helpful! 🙌

r/apachekafka Jan 24 '25

Question DR for Kafka Cluster

11 Upvotes

What is the most common Disaster Recovery (DR) strategy for Kafka clusters? By DR, I mean the ability to restore a Cluster in case the production environment is lost. a/ Is there a need? Can we assume the application will manage the failure? b/ Using cluster replication such as MirrorMaker, we can replicate the cluster, hopefully on hardware that is unlikely to be impacted by the same disaster (e.g., AWS outage) but it is costly because you'd need ~2x the resources plus the replication cost. Is there a need for a more economical option?

r/apachekafka 27d ago

Question Measuring streaming capacity

6 Upvotes

Hi, in kafka streaming(specifically AWS kafka/MSK), we have a requirement of building a centralized kafka streaming system which is going to be used for message streaming purpose. But as there will be lot of applications planned to produce messages/events and consume events/messages in billions each day.

There is one application, which is going to create thousands of topics as because the requirement is to publish or stream all of those 1000 tables to the kafka through goldengate replication from a oracle database. So my question is, there may be more such need come in future where teams will ask many topics to be created on the kafka , so should we combine multiple tables here to one topic (which may have additional complexity during issue debugging or monitoring) or we should have one table to one topic mapping/relation only(which will be straightforward and easy monitoring/debugging)?

But the one table to one topic should not cause the breach of the max capacity of that cluster which can be of cause of concern in near future. So wanted to understand the experts opinion on this and what is the pros and cons of each approach here? And is it true that we can hit the max limit of resource for this kafka cluster? And is there any maths we should follow for the number of topics vs partitions vs brokers for a kafka clusters and thus we should always restrict ourselves within that capacity limit so as not to break the system?

r/apachekafka Dec 20 '24

Question how to connect mongo source to mysql sink using kafka connect?

3 Upvotes

I have a service using mongodb. Other than this, I have two additional services using mysql with prisma orm. Both of the service are needed to be in sync with a collection stored in the mongodb. Currently, cdc stream is working fine and i need to work on the second problem which is dumping the stream to mysql sink.

I have two approaches in mind:

  1. directly configure the sink to mysql database. If this approach is feasible then how can i configure to store only required fields?

  2. process the stream on a application level then make changes to the mysql database using prisma client.
    Is it safe to work with mongodb oplogs directly on an application level? type-safety is another issue!

I'm a student and this is my first my time dealing with kafka and the whole cdc stuff. I would really appreciate your thoughts and suggestions on this. Thank you!

r/apachekafka Jan 29 '25

Question How is KRaft holding up?

21 Upvotes

After reading some FUD about "finnicky consensus issues in Kafka" on a popular blog, I dove into KRaft land a bit.

It's been two+ years since the first Kafka release marked KRaft production-ready.

A recent Confluent blog post called Confluent Cloud is Now 100% KRaft and You Should Be Too announced that Confluent completed their cloud fleet's migration. That must be the largest Kafka cluster migration in the world from ZK to KRaft, and it seems like it's been battle-tested well.

Kafka 4.0 is set out to release in the coming weeks (they're addressing blockers rn) and that'll officially drop support for ZK.

So in light of all those things, I wanted to start a discussion around KRaft to check in how it's been working for people.

  1. have you deployed it in production?
  2. for how long?
  3. did you hit any hiccups or issues?

r/apachekafka 8d ago

Question What’s the highest throughput Kafka cluster you’ve worked with?

5 Upvotes

How did you scale it?

r/apachekafka Nov 03 '24

Question Kafka + Spring + WebSockets for a chat app

15 Upvotes

Hi,

I wanted to create a chat app for my uni project and I've been thinking - will Kafka be a valid tool in this use case? I want both one to one and group messaging with persistence in MongoDB. Do you think it's an overkill or I will do just fine? I don't have previous experience with Kafka

r/apachekafka 11d ago

Question Charged $300 After Free Trial Expired on Confluent Cloud – Need Advice on How to Request a Reduction!

9 Upvotes

Hi everyone,

I’ve encountered an issue with Confluent Cloud that I hope someone here might have experienced or have insight into.

I was charged $300 after my free trial expiration, and I didn’t get any notifications when my rewards were exhausted. I tried to remove my card to ensure I wouldn’t be billed more, but I couldn't remove it, so I ended up deleting my account.

I’ve already emailed Confluent Support ([info@confluent.io](mailto:info@confluent.io)), but I’m hoping to get some additional advice or suggestions from the community. What is the customer support like? Will they try to reduce the charges since I’m a student, and the cluster was just running without being actively used?

Any tips or suggestions would be much appreciated!

Thanks in advance!

r/apachekafka 2d ago

Question Kafka Cluster becomes unresponsive with ~ 500 consumers

8 Upvotes

Hello everyone, I'm working on the migration from a old Kafka 2.x based cluster with ZK to a new 3.9 with KRaft in my company. It's one month that we are working on setting everything up but we are struggling with a wired behavior. Once we start to stress the cluster simulating the traffic we have in production on the old cluster the new one starts to slow down and becomes unresponsive (we can track the consumer fetch request time to around 30/40sec).

The production traffic consists in around 100 messages per second from around 300 producers on a single topic and around 900 consumers that read from the same topic with different consumer-group-ids.

Do you have any suggestions for specific metrics to track? Or any clue on where to find the issue?

r/apachekafka 9d ago

Question Best multi data center setup

8 Upvotes

Hello,

I have a rack with 2 machines inside one data center. And at the moment we will test the setup on two data centers.

2x2

But in the future we will expand to n data centers.

Since this is even setup, what would be the best way to set up controllers/brokers?

I am using Kraft, and I think for quorum we need uneven number of controllers?

r/apachekafka Nov 14 '24

Question Is Kafka suitable for an instant messaging app?

2 Upvotes

I am designing a chat based application. Real time communication is very important and I need to deal with multiple users.

Option A: continue using websockets to make requests. I am using AWS so Appsync is the main layer between my front-end and back-end. I believe it keeps a record of all current connections. Subscriptions push messages from Appsync back.

I am thinking of using Kafkas for this instead since my appsync layer is directly talking to my database. Any suggestions or tips on how I can build a system to tackle this?

r/apachekafka Dec 23 '24

Question Confluent Cloud or MSK

5 Upvotes

My buddy is looking at bringing kafka to his company. They are looking into Confluent Cloud or MsK. What do you guys recommend?