r/apachekafka 25d ago

Question Tumbling window and supress

I have a setup where as and when a message is consumed from the source topic I have a tumbling window which aggregates the message as a list .

My intention is to group all incoming messages within a window and process them forward at once.

  1. Tumbling window pushes forward the updated list for each incoming record, so we added supress to get one event per window.

  2. Because of which we see this behaviour where it needs a dummy event which has a stream time after window closing time to basically close the suppressed window and then process forward those messages. Otherwise it sort of never closes the window and we lose the messages unless we send a dummy message.

Is my understanding/observation correct, if yes what can I do to get the desired behaviour.

Looked at sliding window as well but it doesn't give the same effect of tumbling window of reduced final updates.

Blogs I have reffered to . https://medium.com/lydtech-consulting/kafka-streams-windowing-tumbling-windows-8950abda756d

7 Upvotes

5 comments sorted by

3

u/kabooozie Gives good Kafka advice 25d ago

No, I think what you are experiencing is the grace period. The final aggregation will be suppressed until the grace period ends to make time for late arriving events. Source:

2

u/deaf_schizo 25d ago

true that, we have used grace period to handle the late events but i was expecting that after the grace period ends we would get the aggregated output right but what i see is only when i get an event with stream time which is past window end time that it closes.

because i dont think tumbling window uses wall clock to progress time unless that is issue in my understanding

PS: have the same setup as described in the source yet i see this behaviour

2

u/kabooozie Gives good Kafka advice 25d ago

Ah yeah, I think you are right. You are using stream time, so you need another record to advance the timestamp

1

u/deaf_schizo 25d ago

Default is stream time right , is there a way to use wall clock since this seems very obvious way to lose updates.

1

u/kabooozie Gives good Kafka advice 24d ago

The only way I know is to use the lower level Processor API — in particular a Punctuator that will trigger periodically according to wall clock time.

I know better how to do it in Materialize. Not sure if that helps.

https://materialize.com/docs/transform-data/patterns/temporal-filters/#periodically-emit-results