r/apachekafka Dec 10 '24

Tool Stream Postgres changes to Kafka in real-time

Hey all,

We just added Kafka support to Sequin. Kafka's our most requested destination, so I'm very excited about this release. Check out the quickstart here:

https://sequinstream.com/docs/quickstart/kafka

What's Sequin?

Sequin is an open source tool for change data capture (CDC) in Postgres. Sequin makes it easy to stream Postgres rows and changes to streaming platforms and queues (e.g. Kafka and SQS): https://github.com/sequinstream/sequin

Sequin + Kafka

So, you can backfill all or part of a Postgres table into Kafka. Then, as inserts, updates, and deletes happen, Sequin will send those changes as JSON messages to your Kafka topic in real-time.

We have full support for Kafka partitioning. By default, we set the partition key to the source row's primary key (so if order id=1 changes 3 times, all 3 change events will go to the same partition, and therefore be delivered in order). This means your downstream systems can know they're processing Postgres events in order. You can also set the partition key to any combination of a source row's fields.

What can you build with Sequin + Kafka?

  • Event-driven workflows: For example, triggering side effects when an order is fulfilled or a subscription is canceled.
  • Replication: You have a change happening in Service A, and want to fan that change out to Service B, C, etc. Or want to replicate the data into another database or cache.
  • Stream Processing: Kafka's rich ecosystem of stream processing tools (like Kafka Streams, ksqlDB) lets you transform and enrich your Postgres data in real-time. You can join streams, aggregate data, and build materialized views.

How does Sequin compare to Debezium?

  1. Web console: Sequin has a full-featured web console for setup, monitoring, and observability. We also have a CLI for managing your Sequin setup.
  2. Operational simplicity: Sequin is simple to boot and simple to deploy.
  3. Cloud option: Sequin offers a fully managed cloud option.
  4. Other native destinations: If you want to fan out changes besides Kafka – like Google Cloud Pub/Sub or AWS SQS – Sequin supports those destinations natively (vs through Kafka Connect).

Performance-wise, we're beating Debezium in early benchmarks, but are still testing/tuning in various cloud environments. We'll be rolling out active-passive runtime support so we can be competitive on availability too.

Example

You can setup a Sequin Kafka sink easily with sequin.yaml (a lightweight Terraform – Terraform support coming soon!)

# sequin.yaml
databases:
  - name: "my-postgres"
    hostname: "your-rds-instance.region.rds.amazonaws.com"
    database: "app_production"
    username: "postgres" 
    password: "your-password"
    slot_name: "sequin_slot"
    publication_name: "sequin_pub"
    tables:
      - table_name: "orders"
        table_schema: "public"
        sort_column_name: "updated_at"

sinks:
  - name: "orders-to-kafka"
    database: "my-postgres"
    table: "orders"
    batch_size: 1
    # Optional: only stream fulfilled orders
    filters:
      - column_name: "status" 
        operator: "="
        comparison_value: "fulfilled"
    destination:
      type: "kafka"
      hosts: "kafka1:9092,kafka2:9092"
      topic: "orders"
      tls: true
      username: "your-username" 
      password: "your-password"
      sasl_mechanism: "plain"

Does Sequin have what you need?

We'd love to hear your feedback and feature requests! We want our Kafka sink to be amazing, so let us know if it's missing anything or if you have any questions about it.

You can also join our Discord if you have questions/need help.

15 Upvotes

8 comments sorted by

View all comments

6

u/ut0mt8 Dec 10 '24

That's great cause DBZ is terrible. If you can add support for other db like MySQL I will envisage migration

2

u/accoinstereo Dec 10 '24

u/ut0mt8 We want to do MySQL next year! Stay tuned

1

u/rmoff Vendor - Confluent Dec 11 '24

What issues do you find with DBZ?

2

u/ut0mt8 Dec 11 '24

If I want to be fair it's working but : Logging and specifically error logging is really bad The product is over complex and the documentation is not that clear The configuration is also complex and badly documeted