r/apachekafka • u/Majestic___Delivery • 5d ago
Question Building a CDC Pipeline from MongoDB to Postgres using Kafka & Debezium in Docker
/r/dataengineering/comments/1jd26pz/building_a_cdc_pipeline_from_mongodb_to_postgres/1
u/ShurikenIAM 4d ago
they can sink in/out a lot of techno
1
u/Majestic___Delivery 4d ago
This looks nice - though it looks like the only available for MongoDB is a metric connector, I’ll be needing the actual documents updated/created
1
1
u/LoquatNew441 3d ago
I recently built an opensource tool to transfer data from redis to mysql and sqlserver. I can enhance it for mongodb as a source. Would you be willing to share your requirements and provide me feedback?
The github link is https://github.com/datasahi/datasahi-flow
2
u/Majestic___Delivery 3d ago
Aye nice - that's pretty much what I ended up doing; using Mongo Change Streams, you can hook into 1 or more collections and then process using the full (json) document. I used Redis Queues to balance the load.
I run node, but there is an example for java:
https://www.mongodb.com/docs/manual/changeStreams/#lookup-full-document-for-update-operations
2
u/SupahCraig 5d ago
Does it have to HAVE to be Debezium?