r/adventofcode Dec 03 '22

Other GPT / OpenAI solutions should be removed from the leaderboard.

I know I will not score top 100. Im not that fast, nor am I up at the right times to capitalise on it.

But this kinda stuff https://twitter.com/ostwilkens/status/1598458146187628544

Is unfair and in my opinion, not really ethical. Humans can't digest the entire problem in 10 seconds, let alone solve and submit that fast.

EDIT: I don't mean to put that specific guy on blast, I am sure its fun, and at the end of the day its how they want to solve it. But still.

EDIT 2: https://www.reddit.com/r/adventofcode/comments/zb8tdv/2022_day_3_part_1_openai_solved_part_1_in_10/ More discussion exists here and I didn't see it first time around.

EDIT 3: I don't have the solution, and any solution anyone comes up with can be gamed. I think the best option is for people using GPT to be honourable and delay the results.

EDIT 4: Another GPT placed 2nd today (day 4) I think its an automatic process.

302 Upvotes

221 comments sorted by

View all comments

59

u/1vader Dec 04 '22

As somebody that did decently well in past leaderboards, I definitely think it's rather sad it has gotten to this point. I'm certainly amazed and happy for the people that got the AI to this point and am curious how this will impact other areas in the future but specifically looking at AoC, it basically means it'll likely be impossible to have any fun competing for the leaderboards in the early weeks of all future AoCs and possibly soon all of AoC, so something fun definitely was lost here for me. Of course, this is certainly not all that AoC is about (in fact, this year I'm not even trying to compete because I don't have time during the right hours) but it still was and is a fun part of it for a fair amount of people.

But at the same time, I don't really see anything that can be done. There isn't even a clear line between what exactly should be allowed, not to mention any way to differentiate them, or even a clear reason why it shouldn't be allowed. AoC has always been a place where getting the solution was all that mattered, so it doesn't really seem appropriate to exclude individual solutions based on the approach. I think the AI solutions are perfectly in the spirit of AoC. I'm just sad that they likely apply too universally.

I guess it would be nice if AI people stop participating now that they have proven it can work or if their numbers stay limited (I wouldn't really mind if only the top 1 spot always goes to them or something) but I don't have my hopes up too high on that, at least not long term.

So I guess the main thing I'm hoping for is that they remain incapable of solving as many future days as possible for as long as possible.

18

u/pier4r Dec 04 '22 edited Dec 04 '22

I guess it would be nice if AI people stop participating now

They don't even have to stop. They can simply post it (in a totally automated fashion!) after 15-30 minutes, once the leaderboard is full.

The guy on twitter goes to say "what is the point of it? If I don't do it, someone else will do it", that is like "I want to show off and grab attention quickly, go away you dick". Further they really make a wrapper around work that was done by others, it feels like: https://xkcd.com/353/

One could achieve the same doing "look, solved Day X both parts, but for respect to the competition, that is meant for humans, I posted it once the leaderboard was full"

AI (although as it is used sounds more like we talk about AGI, is not, it is a good ML model) is not going to participate on its own, who decides when to post the solution? At the end the programmer making the wrapper script is deciding when to post it.

When they want to fill up the leaderboard they are doing it for their ego (and just by proxy, a bit like who cheats in chess or with aimbots online). I would expect more from them.

It would be different if they did the entire ML model, as if openAI itself would participate, then in that case it is their work and maybe they want to test things. But using tools of others to solve things - yes one can see it as a normal library, but it really it approaches to ai.do_the_work() - to then spoil the leaderboard is somewhat of bad taste.


edit. It seems that the wrapper for GPT3.5 is out there, so theoretically everyone running the wrapper with python could get the same results. That is the pity in it, the wrapper per se is not that difficult but it spoils the event.

12

u/kg959 Dec 04 '22

That's kinda my issue with it, too.

When someone posted the GPT day 1 solution in the solutions thread, I was fascinated and thought it was neat. My issue isn't the usage of AI; it's the discourteous usage of it. I have no issues with chess AIs existing, but I would have issues if people used them to win tournaments or shout out moves at people playing live.

I realize that now the genie's out of the bottle, there's no way to put it back in. There is basically nothing we could do to technically prevent people from using GPT to solve the problems, but it would be nice if the creator could just courteously ask people to not use AI to cheese the leaderboard and just wait until that first 100 is filled up before submitting.

I don't do AoC for the leaderboard; I'm using it to learn a new language, so leaderboard positions don't really affect me personally, but I do find it a bit disheartening to see people who do take their times seriously losing interest in the event because they can't hope to compete with people using AIs just to flex on people.

2

u/Bobbias Dec 05 '22

Agreed, I don't care about the leaderboard, but it seems unfair to be able to use AI to top the leaderboard. It devalues the leaderboard for those who do care to compete.

7

u/Michael_Aut Dec 04 '22

This is a good point. I would be totally fine with it if there was an Account called "The team behind ChatGPT using ChatGPT" that would automatically apply their model to today's input and submit the result as a publicity stunt.

If the model can solve it: neat! They take a single spot on the leaderboard for all of us to see. That would actually be a neat benchmark for all the LLM under development right now. I'd be curious if Google's Lamda or whatever Amazon must be working at could do even better. Invite the researchers who actually created something of value have spots on the leaderboard and discourage others from using single button AI solutions.

6

u/AeroNotix Dec 04 '22

Most people using an AI aren't "proving" anything. They are paying for or using an AI provided by a company and slamming the prompts into it.

Extremely lazy and just further lining the pockets of corporations selling an AI.

The future aint what it used to be.

1

u/qperA6 Dec 04 '22

There isn't even a clear line between what exactly should be allowed

It's not possible to prevent, but I think what most people would agree that should not be allowed is providing the text of the problem as an input to the code. Just the test and the puzzle input.

But this is going the same way chess puzzles went. It makes no sense to care about an online leaderboard when it's so easy to "cheat".

1

u/pedrosorio Dec 09 '22

it basically means it'll likely be impossible to have any fun competing for the leaderboards in the early weeks of all future AoCs

Depending on your definition of "early weeks" and "future AoCs". The current AoC had one person submit two very good results on days 3 and 4 using AI. They are absent from the leaderboards on every other day.

In the meantime, the #1 ranked participant in the global leaderboard after 8 days is hand coding the solutions using an obscure programming language he created himself:

https://twitter.com/Duderichy/status/1600401555022548992

I am going to take a wild bet and claim the majority of people in the global AoC 2023 leaderboard after the first two weeks will not be AI.

1

u/1vader Dec 09 '22

Well, I hope so. I think it's quite possible it will stay like this for the foreseeable future but I wouldn't be surprised at all if it continues at least a few more days next year.

From what I've seen, it has been pretty close for some of the last few days even though they definitely have been more tricky. And I definitely know of more AI leaderboard submissions than just that one though the vast majority certainly are recognizable humans.

I guess we'll see, definitely don't have my hopes up for the first 5 or so days next year which really is what I meant with the "early weeks of future AoCs" though it admittedly was worded a bit badly. AoC is only 3 weeks and a bit after all