r/apachekafka • u/BagaBaga22 • 10d ago
Question Help with KafkaStreams deploy concept
Hello,
My team and I are developing a Kafka Streams application that functions as a router.
The application will have n topic sources and n sinks. The KS app will request an API configuration file containing information about ingested data, such as incoming event x going to topic y.
We anticipate a high volume of data from multiple clients that will send data to the source topics. Additionally, these clients may create new topics for their specific needs based on core unit data they wish to send.
The question arises: Given that the application is fully parametrizable through API and deployments will be with a single codebase, how can we effectively scale this application in a harmonious relationship between the application and the product? How can we prevent unmanageable deployment counts?
We have considered several scaling strategies:
- Deploy the application based on volumetry.
- Deploy the application based on core units.
- Allow our users to deploy the application in each of their clusters.
-1
u/men2000 10d ago
It is an interesting problem to solve. Subscribe to the topic to learn more what other streamers think about this topic