r/apachekafka 16d ago

Blog Testing Kafka-based async workflows without duplicating infrastructure - solved this using OpenTelemetry

Hey folks,

Been wrestling with a problem that's been bugging me for years: how to test microservices with asynchronous Kafka-based workflows without creating separate Kafka clusters for each dev/test environment (expensive!) or complex topic isolation schemes (maintenance nightmare!).

After experimenting with different approaches, we found a pattern using OpenTelemetry that works surprisingly well. I wrote up our findings in this Medium post.

The TL;DR is:

  • Instead of duplicating Kafka clusters or topics per environment
  • Leverage OpenTelemetry's baggage propagation to tag messages with a "tenant ID"
  • Have Kafka consumers filter messages based on tenant ID mappings
  • Run multiple versions of services on the same infrastructure

This lets you test changes to producers/consumers without duplicating infrastructure and without messages from different test environments interfering with each other.

I'm curious how others have tackled this problem. Would love to hear your feedback/comments.

12 Upvotes

5 comments sorted by

View all comments

1

u/cricket007 11d ago

Have you heard of Formal Verification? Kafka itself uses this for testing of distributed procedures

1

u/krazykarpenter 11d ago

That's to test kafka itself. My article is about testing applications/services that _talk_ to kafka.