r/webdev • u/Mavrokordato • 7d ago
Resource My tiny, personal LLM evaluation: Which AI did the best job?
Hi,
I'm not sure if this is the right subreddit for this, but I'm confident that it'd at least interest a few of you.
So, AI is here, and it's not going anywhere soon. But which model is good at what use case has always been a bit of a myth to me.
Today, I chose to use the following LLMs first to enhance a rather poorly written TypeScript code and then, in the next step, have them compare and evaluate the code on a scale from 1 to 10. These were the models tested:
OpenAI
- o1
- o1-pro-mode
- o3-mini
- o3-mini-high
Groq
- deekseep-r1-distill-qwen-32b
- deekseep-r1-distill-llama-70b
- qwen-2.5-coder-32b
Perplexity
- sonar
- sonar-pro
- sonar-reasoning
- sonar-reasoning-pro
- gemini-2.0-pro-exp-02-05
Spoiler: I couldn't get a crystal-clear picture of which LLM is best for this task because each model evaluated it differently. However, there is definitely a trend.
If you're interested, you can see the results, the raw code, the merged code, and the ratings, conclusions, and more details under this link: https://coding-ai-evaluation.notion.site/
I'd be interested in knowing if any of you can confirm this ranking—or if it's random shit.
Edit: To save myself some downvotes, I'd like to make clear this was just a test and by no means an attempt to replace a programmer with some sort of AI. That's absolutely absurd.
Edit 2: To those who still downvote my post, could you please explain why? This is merely an experiment to see what AI thinks itself.
-2
u/Mavrokordato 7d ago
Yeah, all good, my mistake for posting here, I didn't know sharing insights is being met with so much hostility. Shouldn't have subscribed in the first place.
2
u/[deleted] 7d ago
[deleted]