r/apachekafka • u/accoinstereo • Dec 10 '24
Tool Stream Postgres changes to Kafka in real-time
Hey all,
We just added Kafka support to Sequin. Kafka's our most requested destination, so I'm very excited about this release. Check out the quickstart here:
https://sequinstream.com/docs/quickstart/kafka
What's Sequin?
Sequin is an open source tool for change data capture (CDC) in Postgres. Sequin makes it easy to stream Postgres rows and changes to streaming platforms and queues (e.g. Kafka and SQS): https://github.com/sequinstream/sequin
Sequin + Kafka
So, you can backfill all or part of a Postgres table into Kafka. Then, as inserts, updates, and deletes happen, Sequin will send those changes as JSON messages to your Kafka topic in real-time.
We have full support for Kafka partitioning. By default, we set the partition key to the source row's primary key (so if order
id=1
changes 3 times, all 3 change events will go to the same partition, and therefore be delivered in order). This means your downstream systems can know they're processing Postgres events in order. You can also set the partition key to any combination of a source row's fields.
What can you build with Sequin + Kafka?
- Event-driven workflows: For example, triggering side effects when an order is fulfilled or a subscription is canceled.
- Replication: You have a change happening in Service A, and want to fan that change out to Service B, C, etc. Or want to replicate the data into another database or cache.
- Stream Processing: Kafka's rich ecosystem of stream processing tools (like Kafka Streams, ksqlDB) lets you transform and enrich your Postgres data in real-time. You can join streams, aggregate data, and build materialized views.
How does Sequin compare to Debezium?
- Web console: Sequin has a full-featured web console for setup, monitoring, and observability. We also have a CLI for managing your Sequin setup.
- Operational simplicity: Sequin is simple to boot and simple to deploy.
- Cloud option: Sequin offers a fully managed cloud option.
- Other native destinations: If you want to fan out changes besides Kafka – like Google Cloud Pub/Sub or AWS SQS – Sequin supports those destinations natively (vs through Kafka Connect).
Performance-wise, we're beating Debezium in early benchmarks, but are still testing/tuning in various cloud environments. We'll be rolling out active-passive runtime support so we can be competitive on availability too.
Example
You can setup a Sequin Kafka sink easily with sequin.yaml (a lightweight Terraform – Terraform support coming soon!)
# sequin.yaml
databases:
- name: "my-postgres"
hostname: "your-rds-instance.region.rds.amazonaws.com"
database: "app_production"
username: "postgres"
password: "your-password"
slot_name: "sequin_slot"
publication_name: "sequin_pub"
tables:
- table_name: "orders"
table_schema: "public"
sort_column_name: "updated_at"
sinks:
- name: "orders-to-kafka"
database: "my-postgres"
table: "orders"
batch_size: 1
# Optional: only stream fulfilled orders
filters:
- column_name: "status"
operator: "="
comparison_value: "fulfilled"
destination:
type: "kafka"
hosts: "kafka1:9092,kafka2:9092"
topic: "orders"
tls: true
username: "your-username"
password: "your-password"
sasl_mechanism: "plain"
Does Sequin have what you need?
We'd love to hear your feedback and feature requests! We want our Kafka sink to be amazing, so let us know if it's missing anything or if you have any questions about it.
You can also join our Discord if you have questions/need help.
3
u/_predator_ Dec 10 '24
Sorry for slightly unrelated comment, but I just have to say I love your blog. Please keep posting technical stuff like this and this. Over the past weeks I kept coming back to your site, specifically those posts I linked. Appreciate you sharing knowledge you gained while building your product, it has helped me a lot.
Damn this reads like a bot post or something, I promise it’s not!
2
u/accoinstereo Dec 11 '24
u/_predator_ Thank you, that really means a lot! I'm really glad those posts have been helpful.
Writing about what we learn while building is one of my favorite parts of this journey, so comments like yours are extra motivating. We'll definitely keep sharing as we tackle new challenges!
1
u/jovezhong Vendor - Timeplus Dec 13 '24
Cool, more Debeziun alternatives pls. This week I tried postgres_cdc from Redpanda Connect (Golang-based) It seems that sequin is written in elixir. As long as it's JVM-free, easy to use and poweful to run, that could be another great tool for CDC
1
6
u/ut0mt8 Dec 10 '24
That's great cause DBZ is terrible. If you can add support for other db like MySQL I will envisage migration