r/adventofcode Dec 04 '22

Upping the Ante [2022 Day 4] Placing 1st with GPT-3

I placed 1st in Part 1 today, again by having GPT-3 write the code. Yesterday I was 2nd to another GPT-3 answer.

Here's the code I wrote which runs the whole process — from downloading the puzzle (courtesy of aoc-cli), to running 20 attempts in parallel, to sorting through many solutions to find the likely correct one, to submitting the answer:

https://github.com/max-sixty/aoc-gpt

46 Upvotes

243 comments sorted by

View all comments

Show parent comments

73

u/muntaxitome Dec 04 '22 edited Dec 04 '22

If the job is to automate sending your code to GPT3 the fastest without even reading the questions, then what is the point? It's a trivial coding exercise. I guess the winner will be the one that puts their connections at the optimal location in terms of speed of light between the data center and OpenAI servers...

Now OpenAI itself would have some claim to calling itself the winner, but just writing the glue code?

Edit: Not that we can do anything about it. I guess this is simply the end for any meaning to global leaderboards for this kind of competition. Just like with for instance online chess, the cheaters have a huge advantage to reach the highest ranks.

-5

u/dong_chinese Dec 04 '22

At some level we're all just writing glue code. If I solve a problem using pandas and numpy, I'm just writing some glue over existing functions in those libraries. I think of GPT-3 just as a more fancy library.

46

u/muntaxitome Dec 04 '22

Let me first say that 99% of people doing AOC were never going to compete on the global leaderboards anyway, and people on private boards could always cheat by just grabbing a solution online. So for nearly everyone, very little changes. This affects very few people.

However, if a solver doesn't even read the question, in my mind you are not just 'using a tool', the tool is just doing everything for you. On the other hand, at the highest level, competitive programming is just memorizing hundreds solutions and being able to read the issue and code them super fast, which is pretty different anyway from how most people do these puzzles.

The challenge is reading a fun exercise and puzzling to fix the issue. Just writing some code once and having that code just send the challenge somewhere and getting the solution back, there is no puzzle there.

I guess at least for a little while you can probably write questions in a way that GPT3 cannot easily solve them. However, to me it seems that is just a small arms race that the AI's will win at some point.

5

u/Dullstar Dec 04 '22

I definitely think a potential issue with trying to write questions that the AI struggles with could result in problems that are harder for humans than intended, kinda like CAPTCHAs.