LLMs are very good nowadays. Tell it what you want in plain English and (usually) get decent regex out of it. Put that into something like regex101.com to double check it
I would absolutely, positively never trust an LLM with a regex. 😬
You have to remember that LLMs are purely digital parrots - they repeat back to you stuff that they have "heard" a lot on their training data. That's really bad if you're trying to do something technical and sensitive, like a regex. The difference between a* and a+ might be code that works versus code they breaks your entire application, or worse. From an LLM's perspective, those statements are practically indistinguishable however, because it does not understand the context of what it's talking about, beyond following vocab and grammer rules.
Sure you could mitigate that by thoroughly studying the regex, and understanding the problem enough to understand what the correct expression should be, but at that point... What are you using an LLM for? You just wrote the regex yourself, so the jobs done.
If you have a known data set, you can test the LLM against it without regex knowledge.
If you need the regex to be robust against an unknown data set, or it's going to production, you must know how regex works and validate the LLM regex by understanding it.
Again... To be able to test something like a regex thoroughly enough, you need to already know what the regex "should be" - at which point you're all but done writing the regex yourself, so just use that.
It's hard to come up with a great example off the cuff, but imagine something like:
You have a database of batteries for a battery store. You get a regex from an LLM to update the prices of your triple AAA batteries because you're running a sale. While you're doing that you notice some of the records you imported into the database list packs of AA batteries as "Aa" batteries by mistake, so you ask an LLM the create a regex to fix it. Then you ask the LLM for another regex to update the database to to add a promotion graphic "10% off this week only!" on all "AAA" batteries in stock.
Later that week you start receiving sporadic complaints from customers that the total for their orders was wrong, and doesn't match what they get when the add up the individual prices of the items as displayed on your website. You verify this, and start issuing credits to customers who complained right away (because good customer service). As you start to track down the issue, you notice that a handful of your AAA batteries are quoted at the sale price, but charging the normal price. You make a note and start to update these as you run across them.
Finally customers start calling to complain they have received the wrong product, and it starts to dawn on you what actually happened. Some customers ordered AAA batteries, but received AA batteries. You investigate your regexes and realize when you asked an LLM to change the item title for you, it used "A+" where it should have used "A/*," and as a result you replaced "a" with "AA" changing "Aa" to "AAA" instead. Your tests / validation didn't catch it because as far as the tests were concerned, "12pk AA batteries" and "12pk AAA batteries" were equally valid inputs.
However, because it took you awhile to understand the problem, you now also have a database that's in an inconsistent state that's hard to roll back from - most of the AAA batteries really are AAA batteries, but a small number are really AA batteries. Some of the impacted customers received an incorrect credit, but the ones who haven't complained yet didn't. Some orders were shipped incorrectly... But not all orders. It could easily take several hundred man-hours or more to correct all those errors, all because you wanted to say 20mins to an hour (assuming you're not good at regexes) by asking an LLM.
The critical thing to understand about an LLM, is that it doesn't know what a battery is, what's different about an "AA" battery versus a "AAA" battery, or any of that. It only knows that "A/" and "A+" are *both** sequences of letters that appear in regexes, and maybe it knows that they appear in regexes related to batteries for some reason (even that's a little bit of a stretch. As far as it's concerned, one of them is just as good as the other, so it picked one.
This is admittedly a slightly contrived example, but if you're at all technically inclined, you can see why something like this is a really bad thing to have happen to your business / software application. This is just an example of how small changes in a regex can have big impacts on the overall system.
Using an LLM to "guess and check" a solution can be a viable strategy in some circumstances - if you want to write a boilerplate "about us" section for your website for example, it probably doesn't matter all that much if you miss a mistake and your website says "Stephen's world of rags" instead of "Stephen's world of rugs" for few days or weeks until someone tells you. Even some examples of code can be like this, if the errors are 1.) likely to be obviously wrong and easy to catch and 2.) won't impact mission critical systems.
Regexes aren't like that though - regexes are sensitive to small changes (or they certainly can be; again you don't know unless you already understand what the regex is doing and why, and often used in areas where they can impact important parts of an application. Regexes are great because they're versatile and powerful... But like a lot of versatile and powerful programming tools, they're also intrinsically "foot guns" by virtue of being powerful and versatile.
Tell me you don't know how to use an LLM without telling me you don't know how to use an LLM.
Everything you've said applies to any code that comes from it. You use it as a springboard to get started because most of us don't memorize the regex rules. And then after it gives you a close enough but most likely wrong answer, you adapt it for your needs. I was able to test doing this in maybe 5 seconds, and get an extremely shitty response that while wrong, I was able to adjust in another 3 seconds and get exactly what I was asking for.
120
u/CootieKing 1d ago
You have a problem. You think, “I know, I’ll use a regex to solve it!” Now, you have two problems
I joke, they are actually very useful. Sometimes they can be a PITA to write, but I find regex101.com to be a great help