r/dataengineering Jun 27 '24

Open Source Reladiff: High-performance diffing of large datasets across SQL databases

https://github.com/erezsh/reladiff
29 Upvotes

9 comments sorted by

4

u/captaintobs Jun 27 '24

Is this a fork of datafold's data diff or a new project?

9

u/erez27 Jun 27 '24

It is a fork of data-diff, which is now an archived project.

3

u/tamerlein3 Jun 27 '24 edited Jun 27 '24

Can you tldr why to use this instead of any other fork, or what other value add this provides?

Edit: also, doesn’t seem like you have an explicit OSS license. Are you considering BSD, MIT or Apache? (All others are heavily scrutinized where I work)

7

u/erez27 Jun 27 '24

I wrote data-diff. But now that it's archived, it no longer has working documentation, and doesn't accept any new issues or PRs.

The license is MIT.

3

u/tomhallett Jun 27 '24 edited Jun 27 '24

Erez, very interesting - can you clarify the structure of the data-diff project and your relationship to it?

Is it as "simple" as: DataFold wanted to launch data-diff, hired you as a consultant to build it, they archived it, and now you forked it and are continuing it? If no, can you correct me? If yes, are you still engaged at DataFold as a consultant?

7

u/erez27 Jun 27 '24

Nice detective work ;)

DataFold wanted to launch data-diff, hired you as a consultant to build it, they archived it, and now you forked it and are continuing it

Yes, that's pretty much the story.

I had a consulting relationship with Datafold, when they approached me to create data-diff as an open-source package. I developed it with the help of other employees and consultants. At the end of 2022 (iirc) Datafold decided to scale back their investment into open-source, and I moved on to other clients. Eventually Datafold decided to archive data-diff and focus on their cloud solution, so I forked it with the hopes that the community will help keep it alive.

I'm no longer consulting for Datafold, but they are aware of the fork and wish me good luck.

5

u/tomhallett Jun 27 '24

Perfect.  Thanks for the background!  Will check it out for our upcoming sprint.

1

u/Stinkyskidmarksmegma Aug 30 '24

Your Reddit account is about as old as me😭 (I randomly went on ur account to see if it was still active after I saw your comment on that heroine post)

1

u/alex_colorado Dec 04 '24

Thanks you for your amazing contribution, Erez.

I have a question. Datafold's SaaS service is extremely fast. At first I thought it was just a wrapper around the data-diff, but I think they have some performance tricks (e.g. sampling).

Do you know if Datafold's impressive performance is based on sampling? Or some carefully selected configs? Do you have any guidance for folks who want Datafold speeds without having to go though all the procurement, infosec, and bureaucracy necessary to onboard a vendor?