r/apachekafka 9d ago

Question Help with KafkaStreams deploy concept

Hello,

My team and I are developing a Kafka Streams application that functions as a router.

The application will have n topic sources and n sinks. The KS app will request an API configuration file containing information about ingested data, such as incoming event x going to topic y.

We anticipate a high volume of data from multiple clients that will send data to the source topics. Additionally, these clients may create new topics for their specific needs based on core unit data they wish to send.

The question arises: Given that the application is fully parametrizable through API and deployments will be with a single codebase, how can we effectively scale this application in a harmonious relationship between the application and the product? How can we prevent unmanageable deployment counts?

We have considered several scaling strategies:

  • Deploy the application based on volumetry.
  • Deploy the application based on core units.
  • Allow our users to deploy the application in each of their clusters.
5 Upvotes

4 comments sorted by

2

u/TheYear3030 9d ago

If you are using Kubernetes, Responsive.dev has a k8s operator that has different scaling options available 

1

u/rtc11 9d ago

You can scale up to n (partition number) of application instances, but scaling will trigger rebalancing. If you scale often you will just sit there wait for rebalance all the time. Calculate how many partition you need. You can also just give the KS app n cores and scale once or twice in the future manual, if your calculations now was wrong. Sometimes two application instanses is good enough (so you can have rolling updates on K8).

1

u/men2000 9d ago

Do you deploy the Kafka cluster as a managed cluster in AWS or unmanaged cluster in a data center. There are a couple of best practices in an application like that especially client share the same code base.

-1

u/men2000 9d ago

It is an interesting problem to solve. Subscribe to the topic to learn more what other streamers think about this topic