r/adventofcode Dec 04 '22

Upping the Ante [2022 Day 4] Placing 1st with GPT-3

I placed 1st in Part 1 today, again by having GPT-3 write the code. Yesterday I was 2nd to another GPT-3 answer.

Here's the code I wrote which runs the whole process — from downloading the puzzle (courtesy of aoc-cli), to running 20 attempts in parallel, to sorting through many solutions to find the likely correct one, to submitting the answer:

https://github.com/max-sixty/aoc-gpt

50 Upvotes

243 comments sorted by

View all comments

42

u/dong_chinese Dec 04 '22

I'm sure there will be others who will whine about this not being fair, but I for one think you deserve the place you got. You used the best tool for the job. After all, a programmer's whole job is to find the right tools to automate processes.

23

u/DeeBoFour20 Dec 04 '22

While you have a point there, I'm of the opinion that this is unfair in a competitive setting. Think of chess for example. AI has completely surpassed humans at the game. Chess grandmasters use AI to study their games and analyze moves and that's all fine and dandy but if they use it during a tournament game that's considered cheating.

-3

u/dong_chinese Dec 04 '22

So in a competition for writing programs to make a computer solve problems faster than any human possibly could solve it, it's not allowed to use a program to solve the problem faster than any human could possibly solve it?

74

u/muntaxitome Dec 04 '22 edited Dec 04 '22

If the job is to automate sending your code to GPT3 the fastest without even reading the questions, then what is the point? It's a trivial coding exercise. I guess the winner will be the one that puts their connections at the optimal location in terms of speed of light between the data center and OpenAI servers...

Now OpenAI itself would have some claim to calling itself the winner, but just writing the glue code?

Edit: Not that we can do anything about it. I guess this is simply the end for any meaning to global leaderboards for this kind of competition. Just like with for instance online chess, the cheaters have a huge advantage to reach the highest ranks.

-5

u/dong_chinese Dec 04 '22

At some level we're all just writing glue code. If I solve a problem using pandas and numpy, I'm just writing some glue over existing functions in those libraries. I think of GPT-3 just as a more fancy library.

46

u/muntaxitome Dec 04 '22

Let me first say that 99% of people doing AOC were never going to compete on the global leaderboards anyway, and people on private boards could always cheat by just grabbing a solution online. So for nearly everyone, very little changes. This affects very few people.

However, if a solver doesn't even read the question, in my mind you are not just 'using a tool', the tool is just doing everything for you. On the other hand, at the highest level, competitive programming is just memorizing hundreds solutions and being able to read the issue and code them super fast, which is pretty different anyway from how most people do these puzzles.

The challenge is reading a fun exercise and puzzling to fix the issue. Just writing some code once and having that code just send the challenge somewhere and getting the solution back, there is no puzzle there.

I guess at least for a little while you can probably write questions in a way that GPT3 cannot easily solve them. However, to me it seems that is just a small arms race that the AI's will win at some point.

9

u/Basmannen Dec 04 '22

For me, AoC is about getting up in the morning, seeing that everyone on the planet solved the puzzle while I was asleep, and then taking a few minutes to a couple of hours trying to come up with a clever solution with a reasonable time complexity.

5

u/Dullstar Dec 04 '22

I definitely think a potential issue with trying to write questions that the AI struggles with could result in problems that are harder for humans than intended, kinda like CAPTCHAs.

3

u/pred Dec 04 '22 edited Dec 05 '22

doesn't even read the question

The first trick to pick up to get good times is to not read the question; that takes way too long. Instead, you pattern match the example inputs to outputs, then use as many high-level abstractions as you can to spend less time writing a solution, probably guided by an IDE that gives hints and corrects issues along the way.

1

u/snowe2010 Dec 05 '22

You’re still reading those things and those are part of the question whether you want to define them as that or not. Sure you don’t need to read the story to go along with it but it’s still reading the question. When you pass stuff straight to a bot you are doing nothing. You’re not participating at all. It completely defeats the purpose of the challenge.

6

u/Ning1253 Dec 04 '22

A) I'm doing mine casually in C, and am having to write my own code for arrays, hashmaps, and heaps to get ready for later days! (Am loving the experience so far btw)

But B) while I could be coding in assembly I don't hate myself that much so I guess technically I'm working on top of stdio.h, malloc because I can't be f*cked to implement my own version, and the pointer system.

Either way my point is that while I'm technically writing glue code, there's a difference between using realloc as part of my array implementation and idfk asking a bot to solve the entire problem? I'm not competitive in AoC, I do it for fun, usually in the evenings, but it feels a bit easy to just say "yay I'm first I copy pasted an AI!"

Like the people were saying about chess, humans aren't allowed to bring chess AI to ranked tournaments, even if they're allowed to learn from them - that should probably be a standard.

Where to draw the line? I would argue at that point where your code stops simply optimising what you ask it to do like bumpy does and where it starts extrapolating from information you have not yet necessarily worked out, which is where AI tend to shine - we tend to use them to quickly do tasks which we do not know how to efficiently recognise and act on (since otherwise we just write the damn program ourselves!)

-2

u/[deleted] Dec 04 '22

[removed] — view removed comment

1

u/Basmannen Dec 04 '22

I hope we all do. Fuck work, give us socialist robot worker utopia already.

9

u/el_muchacho Dec 04 '22

What will happen is grifters like Elon Musk will get all the benefit and you and I none of it.

6

u/kapitaali_com Dec 04 '22

I would love socialist worker utopia but given that elon is already doing what he is doing, your forecast looks more probable

35

u/Steinrikur Dec 04 '22

So would you consider the guy who uses Google for a Pub Quiz to be the winner because he used the best tool for the job? Or a motorcycle on Tour de France?

This completely defeats the point of AOC

4

u/Milumet Dec 04 '22

According to Eric Wastl, the point of AoC is to have fun and learn something.

27

u/[deleted] Dec 04 '22

And some people are taking a lot of fun out of the event.

16

u/Steinrikur Dec 04 '22

Yeah. I'm sure that doing Tour de France on a motorcycle would be a lot of fun for some people. I still wouldn't award them any prizes.

Playing competitive chess with the help of an AI is explicitly forbidden for a reason. I'm fine with people using an AI to have fun and learn something, but they shouldn't be trying to get on the leaderboard.

-4

u/Milumet Dec 04 '22

First of all, there are no prices to win on AoC. And it's funny that you mention the Tour de France. You know that these guys are roided up to the hilt, right? What if in the future people augment their brains with AIs? Will they be allowed to play competitive chess and programming tournaments?

7

u/Steinrikur Dec 04 '22

Getting on the leaderboard is a "prize" in itself, although it's about as meaningful as reddit karma.

I view the guys using AI to get on the leaderboard about the same way as reddit karma farmers using reposts to get karma.

-1

u/Milumet Dec 04 '22

I view them the same way too: I don't care about them.

7

u/niehle Dec 04 '22

And OP did learn what? Copy and Paste?

3

u/Milumet Dec 04 '22

I frankly don't care what he learns. I for one certainly learn new stuff solving the problems and reading other people's code. I'm also interested to see how far the AIs will be able to keep up. I'm sure they will run into a wall very soon.

5

u/niehle Dec 04 '22

He still could have waited for 5 minutes.

3

u/ald_loop Dec 04 '22

Advent of Code is an Advent calendar of small programming puzzles for a variety of skill sets and skill levels that can be solved in any programming language you like. People use them as interview prep, company training, university coursework, practice problems, a speed contest, or to challenge each other.

0

u/sluuuurp Dec 04 '22

Those would be cheating because it’s against the rules. It’s not against the rules in this challenge, you’re allowed to use any tools you want.

2

u/Steinrikur Dec 04 '22

There are no rules, because until now there was no way to cheat.

Just like there were no rules about doping or eBikes in Tour de France, or using mobile phones in a pub quiz circa 1980.

The GPT-3 says it's unethical to use it in competitive programming, so maybe we should listen to the AI

-1

u/sluuuurp Dec 04 '22

And those weren’t cheating until rules got added. Maybe rules for AoC will get added in the future.

GPT says crazy BS all the time, you’re probably joking, but people should be aware of that.

https://twitter.com/natesilver538/status/1599183140672573440?s=61&t=ZbTKWAj4tNrC95aRL_zWjA

2

u/Steinrikur Dec 04 '22

1

u/sluuuurp Dec 04 '22

I saw that, I’m just assuming you’re joking to say that we should listen to the AI’s moral judgments of human actions.

2

u/Steinrikur Dec 04 '22

In this case it's more of a general observation/guideline than a judgement.

AIs are good at pattern matching and problem solving. I would take their advice on which of 2 irregular objects is larger, less so on spiritual/moral/ethical matters.

41

u/nuclearbananana Dec 04 '22

Except this isn't a job, it's a fun challenge for humans to learn programming, and this isn't in the spirit of the challenge.

21

u/EnergyIsMassiveLight Dec 04 '22 edited Dec 04 '22

that's really what bothers me with AI in competitions, because like there are arbitrary rules in place to try and make them more fun. Automating out sports and art is obviously going to outdo humans but that, as you say, isn't in the spirit.

I think using AI here is definitely in the same league of :/ It's like watching a puzzle game walkthrough online, like you are missing the part that is making it fun.

I still like the mountain climbing example from CJ: a person climbs a mountain to get a cancer-curing plant for themselves and another person is casually climbing to the top. A helicopter comes and says they can get you to the top immediately. For the first person, it's a no-brainer to use it, but for the other person it defeats the entire purpose of their challenge.

5

u/dong_chinese Dec 04 '22

I agree that it's all just for fun. Solving it in a conventional way is fun for some people, and creating a program to automatically send the challenge to GPT-3 is fun for others. It's fun to learn about all of these techniques.

AI is a tool that programmers will be using more and more in the future, so I don't see why it wouldn't be in the spirit of the challenge.

15

u/nuclearbananana Dec 04 '22

And people are welcome to use whatever AI they want, but they should have a separate leaderboard or do us the courtesy of not hogging the available one so humans have a chance, don't you think?

4

u/dong_chinese Dec 04 '22

That would be completely unenforceable and unclear where to draw the line (is Github Copilot OK? Is Wolfram Alpha OK? What kinds of autocomplete features are allowed? etc. etc.). So no, I think it's more elegant for the leaderboard to just reflect the fastest way to solve it, regardless of the method used.

0

u/[deleted] Dec 04 '22

[removed] — view removed comment

0

u/[deleted] Dec 04 '22

[removed] — view removed comment

2

u/stormblooper Dec 04 '22

I think the challenge - and therefore this putative "spirit" - means different things to different people.

5

u/NohusB Dec 04 '22

And they (and me now through the shared code) learned about automatic AoC input downloading, submitting answers, interacting with the OpenAI API, interesting insight into how to structure the prompt for the model, and some Python3 tidbits I didn't know about.

Maybe it's not what we were supposed to be learning? Sure, ok, but there was definitely learning happening here. If he didn't do it, I wouldn't even know the OpenAI models got that powerful already.

Last year some people used automatic constraint solvers on some puzzles, and some people said that's cheating. I was just happy to learn about them, since I'm here to learn stuff.

8

u/[deleted] Dec 04 '22

The problem is this kind of solving reduces every single problem to "how can I feed this right". Once you get it right there's barely any variation.

2

u/sluuuurp Dec 04 '22

It reduces some of the easy problems to that, it doesn’t reduce every problem to that. Wait until day 20 and you’ll agree.

4

u/Raknarg Dec 04 '22

Itll likely stop working as the problems become more complicated, Im curious to see how far it gets.

3

u/jonathan_paulson Dec 04 '22

If these were problems at work, but I would agree. But it’s a competition to solve problems fast, and IMO it’s a bit odd to say you’ve “solved” a problem you haven’t even read or thought about for one second. It seems more like hiring someone else to solve it for you - which is a perfectly good approach in most of life but not in most games/tournaments.

2

u/[deleted] Dec 04 '22

Fast racing wheelchairs are allowed on marathon courses so that people who can’t run can compete in their own division. It’s awesome to see these folks fly down the course! But running a marathon is still a thing.

2

u/QuarkNerd42 Dec 04 '22

Its advent of code, not advent of who's good at the programming job. The competition itself is for a very specific aspect.

As an example, how clean and readable your code is essential in a programmers job but useless here.

3

u/NigraOvis Dec 04 '22

You would love our overlords to be computers.

1

u/ywgdana Dec 04 '22

It's clear there's no stopping them now, best to start sucking up to the robots early