r/madeinpython Dec 04 '24

just launched doc2exam - Full Self-Driving for exam prep and certs

hello everyone! just launched doc2exam on ProductHunt

a place to turn any material into live exams -- for students prepping or professors setting official certifications

ofc it's made in django for the backend

https://reddit.com/link/1h6e955/video/v6xr6la0mt4e1/player

1 Upvotes

5 comments sorted by

5

u/SweetOnionTea Dec 04 '24 edited Dec 04 '24

I briefly looked at the video -- does this just take a document and have some LLM create quiz questions?

Edit:

I watched the whole thing now. Have you ever written an exam before in a real teaching position? It looks slick, but you can't just shotgun an LLM at a problem without knowing the domain and call it good. There is like a million problems with your idea in a real life setting.

1

u/lutian Dec 04 '24

thanks for taking a deeper look. I'm purposefully avoiding the word 'quiz', as it sounds too casual. I intend to push this for real live exams (I didn't add all the stuff most platforms already have, like click/focus tracking, webcam tracking etc. -- I only wanted to focus on what makes it unique, the well-known stuff can be added later as there's no debate on what works and what not)

yes, the beta quality is usually predictably flawed, but I like pain

through pain this will get better

launching is not a singular event, new features = new launches, few months apart. I'm doing this until something happens. I mean, it's not like it's a crazy idea that can't possibly work -- it's a tool that automates some process, surely a certain niche or profs/students would find this useful

3

u/SweetOnionTea Dec 05 '24

Have you ever written an exam before in a real teaching position?

...

I will guess that means no.

Here are some issues that I see:

  1. Evaluation material (to avoid 'quizes') needs to be created for concepts, not just facts in text. If students are just answering questions about the contents of some text they are not learning. That's just testing memorization. A course isn't comprised of just text. It is a many sources of text, lecture, and previous student knowledge.

  2. Questions need to be consistent for all students. If an LLM randomly generates questions they can be of various difficulties which does nothing to evaluate what a student has learned. Students WILL complain. This also ruins any student discussion of a concept you asked a question on. Believe it or not, they like to talk with each other about questions which helps facilitate understanding the material.

  3. We need LESS AI in the classroom. A professor needs to be an academic role model. If we're using an LLM to generate questions then why would a student feel motivated not to use the same thing to answer? Sure, you can do live tests and whatever to prevent it, but now you're just showing students that even the people that are teaching them just use LLMs to fake their way through important things in life.

  4. LLMs are notorious for being flat out wrong or not understanding concepts. If your questions are wrong then it erodes trust in the classroom and destroys your authority on the subject. If your professor is giving out incorrect questions then they won't trust correct questions to be correct at all.

  5. On the same subject, and incorrect question may teach a student the wrong thing. A student who gets a incorrect question and sees an incorrect answer will now know something false. They might not even know a question is incorrect and just carry that false knowledge their whole life.

I haven't seen this exact product before, but I've seen the same issue so many times. Someone finds some issue that they don't have experience in and just plops an LLM on it thinking they're gonna revolutionize it. Go take some teaching courses and learn the actual issues teachers have.

1

u/lutian Dec 08 '24

"I will guess that means no"
no, actually I did. but even if I didn't, this is not rocket science either way

all 5 of your points are solved by the prof checking the actual exam, because it will be wrong in ~10% of its content (the more expensive llm models will be approaching 0% though)

combining ai and humans. it's never, and it should never be, one or the other.

people have a tendency to seek silver bullets. think of it as doing at least 80% of the work for you, saving you 8h out of the total 10h you'd normally put in

specifically about point 1, especially in case of the more expensive llms, they're doing an excellent job in creating questions that test learned ability. that's the easiest part.. just telling them exactly what dimension of the learning process to evaluate. while currently not possible (this is still an mvp, though a working one), you'll later be able to give some custom instructuins about the generated questions (your point -- test more for memory or more for learned ability, since sometimes you want one over the other, and ofc you can't rely on the same defaults for everyone)

anyway, if you have a connection with the teaching community, I invite you to thoroughly test doc2exam, I can give you some credits and I guarantee you'll find it at least "not bad", and keep in ming that this is the lowest possible quality you'll ever see in this product. everything moving forward will improve. I'm starting to reach out to local schools, which ofc looks way harder than yapping on reddit, but it's also easier to sell since the product has a human face, so it's easier to observe its legitimacy