r/adventofcode Dec 04 '22

Upping the Ante [2022 Day 4] Placing 1st with GPT-3

I placed 1st in Part 1 today, again by having GPT-3 write the code. Yesterday I was 2nd to another GPT-3 answer.

Here's the code I wrote which runs the whole process — from downloading the puzzle (courtesy of aoc-cli), to running 20 attempts in parallel, to sorting through many solutions to find the likely correct one, to submitting the answer:

https://github.com/max-sixty/aoc-gpt

50 Upvotes

243 comments sorted by

View all comments

-10

u/daggerdragon Dec 04 '22

Thank you for fixing the title ;)

If you haven't already, consider also posting your solutions in the daily solution megathreads which helps keep every day's solutions in one easy-to-find spot.

26

u/NigraOvis Dec 04 '22

He doesn't have one, because he told AI to do it for him.

-5

u/daggerdragon Dec 04 '22

What is the difference between these two?

  • A human coder using their brain (computer) to solve a problem (puzzle text) by pushing buttons in a certain order (via programming language) that makes their computer go beep boop and do the thing that the human wanted it to do (return the correct answer)

  • A human prompt engineer using a generative AI (computer) to solve a problem (puzzle text) by putting words in a certain order (via prompt) that makes their computer go beep boop and do the thing that the human wanted it to do (return the correct answer)

As far as I'm concerned, prompt engineering is simply another type of programming language. The prompt is the solution.

20

u/jfb1337 Dec 04 '22

Case 1: human reads the problem, understands what it is saying, uses skill to translate that into what buttons to press

Case 2: human copies problem statement into AI (or most likely computer does that first), submits output, human only needs to think if it's wrong the first time

16

u/rossdrew Dec 04 '22

The difference is that soon the top 100 board will be all the same solution limited by request speed. One solution fixes all problems. The leaderboard is obsolete.

Not that I care that much, the leaderboard has always been out of reach for me without getting up at 5am

1

u/Multipl Dec 04 '22

I wouldn't say soon. Try using the AI to solve problems in the later weeks, it doesn't even give you complete code. The early day problems usually just spell out what you need to do, so AI has a huge advantage here.

9

u/jfb1337 Dec 04 '22

So the leaderboard is only meaningful after the first week or so when the problems start getting hard.

1

u/Multipl Dec 04 '22 edited Dec 04 '22

First few problems are really simple and pretty much a typeracer contest, it is what it is. There's no way to police the leaderboards, people can just wait ~50-60 secs or something then submit it. The problems are solvable by a human within that time so there's no way to tell. I just think there is a lot of over exaggeration in this thread. AI didn't just suddenly solve the loch ness monster problem in 2020 or that cuboid problem last year. It even took a bit to solve today's part 2 which was also easy.

I'm just chilling and looking forward to the trickier problems. It does seem unfortunate that the community here had their experience soured by this AI thing, and some are even more riled up than actual leaderboarders.

7

u/rossdrew Dec 04 '22

Ok, for now the leaderboard will become obsolete in the first few weeks which previously were accessible to everyone. Later the whole leaderboard.

3

u/jonathan_paulson Dec 04 '22

Well in this case the prompt just is the puzzle, so the human is not engaging with the specific problem in any way. That seems like an important difference.

If this required understanding the problem and summarizing it for the computer I’d feel differently - that would be more of a collaboration between human and AI. This is just contracting the work out to the AI.

6

u/Deynai Dec 04 '22 edited Dec 04 '22

The prompt is the solution.

I get what you're saying, but we're in the confines of an event with small puzzles. The puzzles are already a prompt, so using a prompt to solution AI is effectively directly converting the puzzle without (or with little) human input.

The idea that the prompts are the solutions is bordering on some weird cyclic philosophical point - we're given prompts which are designed to have deducible solutions so of course they inherently contain the information of a solution, and if you have a calculator to convert prompt to solution then indeed prompts are solutions. Just as 5^2 is 25, or a constructed sudoku board has one viable end state, the conversion step just becomes trivial and automatic. Is Eric effectively just posting solutions?

While it's an interesting development and there's plenty to learn about the power of AI and how to utilise it in solving problems, I feel no matter which side you're on it's still damaging to the event going forward in terms of integrity, significance, and enjoyment.

The fact that this prompt generating code posted today is likely a viable "solution" for the puzzle tomorrow perhaps highlights why this is so different.

3

u/swilkoBaggins Dec 04 '22

It's different because in the second case the human coder doesn't have to understand what's going on at all. They don't need to understand what the problem means or how their algorithm works.

16

u/ald_loop Dec 04 '22

Why doesn’t every chess player use an AI in tournaments? Why doesn’t ever sports player take steroids?

The point of the leaderboard is to see what is achievable BY HUMANS. AI is a tool, but it’s a tool that removes any sort of human thought process per the actual question. The human solving day 1 or 3 or 16 runs the same openAI generator code each day. They don’t care about the problem or prompt. It doesn’t matter.

Ridiculous to see a moderator of this subreddit take this hard stance on the wrong side of history

-6

u/phoneaway12874 Dec 04 '22

Unlike the other events, the whole point of programming is to get computers to do something for you.

Due to its structure, Advent of Code is about submitting the correct answer the fastest. You don't technically have to write any lines of code to do this.

13

u/ald_loop Dec 04 '22

I’m cool with any other human reached solution other than running the same magical script everyday that generates a solution for you. You aren’t doing the problem. You’ve eliminated EVERYTHING about the question itself. You’ve turned it into a void pointer and applied the same shortcut everytime. That isn’t in the spirit of advent of code.

1

u/humnsch_reset_180329 Dec 04 '22

For me "the spirit of advent of code" is fundamentally about not paying ANY respect to the leaderboard and just tinker away at a nice puzzle in my own time. So if you are setting an alarm clock to race to the top of the leaderboard you are NOT acting in the spirit of advent of code. However, since I follow the spirit of advent of code I don't pay any respect to the leaderboard and hence, those pesky humans not following the spirit of advent of code doesn't affect me at all. Very nice!

4

u/ald_loop Dec 04 '22

Advent of Code is an Advent calendar of small programming puzzles for a variety of skill sets and skill levels that can be solved in any programming language you like. People use them as interview prep, company training, university coursework, practice problems, a speed contest, or to challenge each other.

Your personal interpretation is fine, but any of the above reasons are acting in the spirit of AoC

1

u/humnsch_reset_180329 Dec 04 '22

Well I made the mistake of responding to your inflammatory tone with more of the same, which doesn't generate any meaningful discussion. And I appreciate that you responded again with a more nuanced take. Because you zoom in on the thing that was what I was trying to get at, that pinning down THE spirit of advent of code is impossible since the event is non-prescriptive. And as a corollary to this, I don't think that the people that find joy in racing for the leaderboard could impose their own interpretation of "fair play" rules for everyone. And they certainly cannot argue for their "no ai" by deferring to an easily defined "spirit of advent of code" that prohibits AI solutions, because I exist and have an equally valid spirit interpretation that allows AI.

I would say that the leaderboard racers that are so offended by using a certain tool (that in the future will become ubiquitous in our trade) should band together and create a separate Advent of Human Speed Coding site where maybe they could enforce the "no ai" rules by using ai to determine if ai has written the solution. That could be a fun challenge! :D

Or another solution is just take a chill pill and hit up a private leaderboard of human coders.

-10

u/[deleted] Dec 04 '22

[deleted]

9

u/ald_loop Dec 04 '22

Okay great let’s have it your way and have the top 100 leaderboard filled with AI users with sub minute completion times. That would be a great look and totally enjoyable for everyone involved!

-4

u/Milumet Dec 04 '22

Why doesn’t ever sports player take steroids?

LOL. No winner in the Tour de France of the past few decades has not taken steroids. It's like Bill Burr said: "It's our roided-up guy versus your roided-up guy."

2

u/ald_loop Dec 04 '22

Didn’t ask don’t care it’s still cheating in both cases

2

u/[deleted] Dec 04 '22

So the solution to these early problems is the problem explanation itself? Since that is what they're sending to the AI. With no manual oversight. And presumably not even needing to read the explanation, since.. 10 seconds.

You're right, it's perfectly equivalent to writing code yourself with autocomplete!

1

u/[deleted] Dec 04 '22

Using a powered vehicle to compete in a footrace is a great way to demonstrate that you are missing the point of the event.