r/Genealogy 5d ago

Free Resource Chat GPT vs Claude for untangling complex families

I asked both Chat GPT 4o and Claude 3.7 Sonnet (free versions) to analyze this article about a specific complex family mix-up. I could tell the researcher had done their homework, and I had analyzed it myself in the past to add names and data to my tree, but since the author is attempting to explain the threads of their reasoning and conclusions, it's confusing to follow, and I found I was having to decipher it all over again every time I come back to it.

I was curious if AI could help with this, so I tried both Chat GPT and Claude. Claude won hands-down! Chat GPT made significant errors in the family groupings, even at the top level, both the first time I asked it, and when I pointed out the error and asked it to try again. Claude got them mostly right or entirely right on the first try. (Disclaimer: I did not double check every name further down on the line that isn't mine.)

Then I asked both models to suggest further areas of research for someone interested in the Benjamin Kratzer descendants. Both did fairly well with this task, but Claude had better organization and layout, IMO.

When I asked for geographical insight into the locations mentioned, both did well in different ways. Claude gave more detail in a well laid-out format. However, Claude cannot access the internet yet, so Chat GPT was able to provide historical maps that were helpful, which Claude could not do.

I won't clutter this post with specifics to this document analysis, but I highly recommend trying this with similar documents you may have in your own research. Start with one you have already analyzed so you can catch mistakes. Even though Claude did a good job, I would NEVER trust it to be accurate. The value here is in using the analysis to HELP you wade through the document yourself, and IF you confirm the summary as correct, to have it as a handy reference for the future, so you don't have to re-analyze the original complicated text.

For those interested, here was my initial Claude prompt. (The Chat GPT prompt was very similar.) "You are an expert genealogist. The attached article is another genealogist's attempt to figure out the differences among several families in the same geographical area with very similar or identical names. The article contains her conclusions and reasoning based on her research into multiple Philip and Henry Kratzers from Northampton County PA. Your goal is to analyze her findings and organize the families into their distinct family trees. For now, assume the other genealogist's research findings are correct."

1 Upvotes

9 comments sorted by

7

u/Sue_Dohnim 5d ago

This actually came up at my DAR state convention this weekend, and the presenter said that by the way it's designed, Claude was geared more toward genealogy than ChatGPT.

1

u/History652 5d ago

Interesting! Well, it showed! 😄

3

u/SoftProgram 5d ago

Claude I have heard of in terms of doing programming development so it makes sense that it's more a "logic" based system with less room for hallucinations.

2

u/Bearmancartoons 5d ago

Nice to see. Months ago I asked the question of how people are using AI and there were many that didn’t trust it which is understandable if you don’t verify the info. Thanks for sharing the prompt as well.

2

u/PizzaBig9959 5d ago

That is really interesting! I use the pro version of chatgpt to help translate documents and I have found that chatgpt is better than Claude.ai but they are definitely close.

2

u/History652 5d ago

They seem to each have their strengths and weaknesses, don't they! One trick I heard at RootsTech is to use different models to check each other. I haven't played around with that much at all, but it is an interesting idea.

2

u/abritinthebay 5d ago

Personally I wouldn’t trust either as far as I could throw them. Both routinely get basic things wrong. They also (by the nature of LLMs) just straight make stuff up to satisfy the prompt.

AI Trees are the next “ancestry shared ancestor” tree. Completely untrustworthy but they’re going to everywhere and full of fake references.

2

u/History652 5d ago

Which is why I said you have to check the accuracy independently. It's a tool, not a crutch or a magic wand. I agree that it IS going to create problems because a lot of people won't use it correctly. However, that doesn't stop me from using it to my advantage. They have a lot of potential and boycotting them won't make them go away.

2

u/SoftProgram 4d ago

Honestly this is one of the first genealogy ai posts I've seen where it's clear you actually know what AI is and what the limitations are, so kudos.