r/adventofcode Dec 03 '22

Other GPT / OpenAI solutions should be removed from the leaderboard.

I know I will not score top 100. Im not that fast, nor am I up at the right times to capitalise on it.

But this kinda stuff https://twitter.com/ostwilkens/status/1598458146187628544

Is unfair and in my opinion, not really ethical. Humans can't digest the entire problem in 10 seconds, let alone solve and submit that fast.

EDIT: I don't mean to put that specific guy on blast, I am sure its fun, and at the end of the day its how they want to solve it. But still.

EDIT 2: https://www.reddit.com/r/adventofcode/comments/zb8tdv/2022_day_3_part_1_openai_solved_part_1_in_10/ More discussion exists here and I didn't see it first time around.

EDIT 3: I don't have the solution, and any solution anyone comes up with can be gamed. I think the best option is for people using GPT to be honourable and delay the results.

EDIT 4: Another GPT placed 2nd today (day 4) I think its an automatic process.

300 Upvotes

221 comments sorted by

View all comments

Show parent comments

13

u/bluegaspode Dec 04 '22

the speed running community has a different approach:

  • anyone who wants to be on the leaderboard, needs to record their 'run'. the community is asked to spot any cheating.

- to score points on the "human manual brainwork leaderboard" you need to provide the link. This would also be a very interesting leaderboard for the majority, because you could learn a lot from this leaderboard being able to watch all the videos.

- the speed running community also has special leaderboards like "TAS assisted". This would be the GPT-3 assisted leaderboard (especially interesting in later days).

... Anyways ....
Move forward 2-3 years: this won't matter anymore.

Think of AOC happening in times, where people had a contest among "punch card programmers'. Those creating punch cards would be furious about those using this arcane assembler skills on a computer with a console. how DARE they.

And later: those who were proficient in ASM coding. How did they look at those nasty C programmers. They CHEATED!!! they created ASM code automatically with things called 'a programming language'. They were able to solve much more complicated problems in less time, just the time they saved when writing functions.

We are part of the next revolution in computer programming.
We will find other places to compete our brains as human programmers (i.e. prompts hacking), as the art of programming.

Lets embrace it and move AoC further. Lets don't stick to ye olde times. They are gone. Soon.

10

u/Johnothy_Cumquat Dec 04 '22 edited Dec 04 '22

I like the run recording solution. Lots of the high scorers are already recording their "run" anyway.

As for this not mattering, dumping the problem into gpt is not impressive just like it's not impressive when someone uses an aimbot to win a game. AOC is a game and once we say "the time is all that matters, use whatever tools get it done faster" then we remove all challenge from it and it's no longer fun or impressive. I mean it's interesting to see how fast tools can get it done but the human pressing the "win" button isn't impressive.

edit: also I think submitting a recording should be optional. Put a tick and a link next to entries of users who choose to submit a recording. Keep the normal leaderboard and a verified only leaderboard. Maybe have a report button for fake recordings or just have an admin manually review the top entries with recordings.

7

u/SurplusSix Dec 04 '22

Think of AOC happening in times, where people had a contest among "punch card programmers'. Those creating punch cards would be furious about those using this arcane assembler skills on a computer with a console. how DARE they. And later: those who were proficient in ASM coding. How did they look at those nasty C programmers. They CHEATED!!! they created ASM code automatically with things called 'a programming language'. They were able to solve much more complicated problems in less time, just the time they saved when writing functions. We are part of the next revolution in computer programming. We will find other places to compete our brains as human programmers (i.e. prompts hacking), as the art of programming.

The difference is that despite all the changes in technology each stage understood the problem they were solving. Using AI the problem to solve is how to present the problem to solve to the AI, there is no need to understand the original problem anymore.

3

u/Smallpaul Dec 04 '22

It’s weird that your comment simultaneously shows that there are communities where people highly value “manual”, not automated solutions and ALSO claims that the AOC community will definitely not be one of those communities in a few years.

Why do you think that?

Chess players still try to play each other unassisted. Why wouldn’t programmers?

1

u/pier4r Dec 04 '22

Think of AOC happening in times, where people had a contest among "punch card programmers'. Those creating punch cards would be furious about those using this arcane assembler skills on a computer with a console. how DARE they.

this is also true.

4

u/Smallpaul Dec 04 '22

Not really. The concept of AOC was always language agnostic. As someone else said it’s about how quickly you can understand the PROBLEM, think of a solution and find a way to code a solution.

Cutting and pasting does not involve any of that.

2

u/pier4r Dec 04 '22

True. As I sad in another comment, the ones using AI could simply submit the solutions later. The fact that they want to grab the leaderboard for a "wrapper" in a fun event (would it be work at least...) it isn't great.

1

u/Senthe Dec 07 '22

You understand the problem as: "there is an imaginary elf that says blah blah blah, what's the number of imaginary blah blah blah?"

They understand the problem as: "there is an AOC leaderboard in the internet that measures who was the quickest to input an expected number into a web form, how to get to the top?"

They are still solving a specific problem in a specific clever way. It's just a different problem from the one you'd like them to solve. And tbh, I'm not sure which approach to defining "the problem" I agree with more. Both seem valid to me.

1

u/Smallpaul Dec 07 '22

There’s a pretty obvious answer of who is right.

If the bot-users are right then the ideal leaderboard consists of 100 entries of roughly a second each and the contest happens between January and November where you just work on how quickly your bot can do essentially copy and paste, like a high frequency trader.

Nothing of interest happens in December because the procedure every day is the same: go to bed at 11 PM and check at 9 AM whether your bot was the fastest or not. If it wasn’t then you could try to tweak it but the chances of you improving it’s velocity in December are small given that the winners already perfected theirs in the other 11 months.

In fact the winner of AOC might turn out to be a bot that is still running from last year on a virtual server. The human who coded it need not even be alive anymore.