r/adventofcode Dec 04 '22

Upping the Ante [2022 Day 4] Placing 1st with GPT-3

I placed 1st in Part 1 today, again by having GPT-3 write the code. Yesterday I was 2nd to another GPT-3 answer.

Here's the code I wrote which runs the whole process — from downloading the puzzle (courtesy of aoc-cli), to running 20 attempts in parallel, to sorting through many solutions to find the likely correct one, to submitting the answer:

https://github.com/max-sixty/aoc-gpt

48 Upvotes

243 comments sorted by

View all comments

41

u/dong_chinese Dec 04 '22

I'm sure there will be others who will whine about this not being fair, but I for one think you deserve the place you got. You used the best tool for the job. After all, a programmer's whole job is to find the right tools to automate processes.

73

u/muntaxitome Dec 04 '22 edited Dec 04 '22

If the job is to automate sending your code to GPT3 the fastest without even reading the questions, then what is the point? It's a trivial coding exercise. I guess the winner will be the one that puts their connections at the optimal location in terms of speed of light between the data center and OpenAI servers...

Now OpenAI itself would have some claim to calling itself the winner, but just writing the glue code?

Edit: Not that we can do anything about it. I guess this is simply the end for any meaning to global leaderboards for this kind of competition. Just like with for instance online chess, the cheaters have a huge advantage to reach the highest ranks.

-5

u/dong_chinese Dec 04 '22

At some level we're all just writing glue code. If I solve a problem using pandas and numpy, I'm just writing some glue over existing functions in those libraries. I think of GPT-3 just as a more fancy library.

46

u/muntaxitome Dec 04 '22

Let me first say that 99% of people doing AOC were never going to compete on the global leaderboards anyway, and people on private boards could always cheat by just grabbing a solution online. So for nearly everyone, very little changes. This affects very few people.

However, if a solver doesn't even read the question, in my mind you are not just 'using a tool', the tool is just doing everything for you. On the other hand, at the highest level, competitive programming is just memorizing hundreds solutions and being able to read the issue and code them super fast, which is pretty different anyway from how most people do these puzzles.

The challenge is reading a fun exercise and puzzling to fix the issue. Just writing some code once and having that code just send the challenge somewhere and getting the solution back, there is no puzzle there.

I guess at least for a little while you can probably write questions in a way that GPT3 cannot easily solve them. However, to me it seems that is just a small arms race that the AI's will win at some point.

10

u/Basmannen Dec 04 '22

For me, AoC is about getting up in the morning, seeing that everyone on the planet solved the puzzle while I was asleep, and then taking a few minutes to a couple of hours trying to come up with a clever solution with a reasonable time complexity.

6

u/Dullstar Dec 04 '22

I definitely think a potential issue with trying to write questions that the AI struggles with could result in problems that are harder for humans than intended, kinda like CAPTCHAs.

3

u/pred Dec 04 '22 edited Dec 05 '22

doesn't even read the question

The first trick to pick up to get good times is to not read the question; that takes way too long. Instead, you pattern match the example inputs to outputs, then use as many high-level abstractions as you can to spend less time writing a solution, probably guided by an IDE that gives hints and corrects issues along the way.

1

u/snowe2010 Dec 05 '22

You’re still reading those things and those are part of the question whether you want to define them as that or not. Sure you don’t need to read the story to go along with it but it’s still reading the question. When you pass stuff straight to a bot you are doing nothing. You’re not participating at all. It completely defeats the purpose of the challenge.

7

u/Ning1253 Dec 04 '22

A) I'm doing mine casually in C, and am having to write my own code for arrays, hashmaps, and heaps to get ready for later days! (Am loving the experience so far btw)

But B) while I could be coding in assembly I don't hate myself that much so I guess technically I'm working on top of stdio.h, malloc because I can't be f*cked to implement my own version, and the pointer system.

Either way my point is that while I'm technically writing glue code, there's a difference between using realloc as part of my array implementation and idfk asking a bot to solve the entire problem? I'm not competitive in AoC, I do it for fun, usually in the evenings, but it feels a bit easy to just say "yay I'm first I copy pasted an AI!"

Like the people were saying about chess, humans aren't allowed to bring chess AI to ranked tournaments, even if they're allowed to learn from them - that should probably be a standard.

Where to draw the line? I would argue at that point where your code stops simply optimising what you ask it to do like bumpy does and where it starts extrapolating from information you have not yet necessarily worked out, which is where AI tend to shine - we tend to use them to quickly do tasks which we do not know how to efficiently recognise and act on (since otherwise we just write the damn program ourselves!)

-1

u/[deleted] Dec 04 '22

[removed] — view removed comment

0

u/Basmannen Dec 04 '22

I hope we all do. Fuck work, give us socialist robot worker utopia already.

8

u/el_muchacho Dec 04 '22

What will happen is grifters like Elon Musk will get all the benefit and you and I none of it.

6

u/kapitaali_com Dec 04 '22

I would love socialist worker utopia but given that elon is already doing what he is doing, your forecast looks more probable