Why are regular expressions so controversial?

I don't think people object to regular expressions because they're slow, but rather because they're hard to read and write, as well as tricky to get right. While there are some situations where regular expressions provide an effective, compact solution to the problem, they are sometimes shoehorned into situations where it's better to use an easy-to-read, maintainable section of code instead.

Regexes are a great tool, but people think "Hey, what a great tool, I will use it to do X!" where X is something that a different tool is better for (usually a parser). It is the standard using a hammer where you need a screwdriver problem.

Almost everyone I know who uses regular expressions regularly (pun intended) comes from a Unix-ish background where they use tools that treat REs as first-class programming constructs, such as grep, sed, awk, and Perl. Since there's almost no syntactic overhead to use a regular expression, their productivity goes way up when they do.

In contrast, programmers who use languages in which REs are an external library tend not to consider what regular expressions can bring to the table. The programmer "time-cost" is so high that either a) REs never appeared as part of their training, or b) they don't "think" in terms of REs and prefer to fall back on more familiar patterns.

Regular expressions allow you to write a custom finite-state machine (FSM) in a compact way, to process a string of input. There are at least two reasons why using regular expressions is hard:

• Old-school software development involves a lot of planning, paper models, and careful thought. Regular expressions fit into this model very well, because to write an effective expression properly involves a lot of staring at it, visualizing the paths of the FSM.

  Modern software developers would much rather hammer out code, and use a debugger to step through execution, to see if the code is correct. Regular expressions do not support this working style very well. One "run" of a regular expression is effectively an atomic operation. It's hard to observe stepwise execution in a debugger.

• It's too easy to write a regular expression that accidentally accepts more input than you intend. The value of a regular expression isn't really to match valid input, it's to fail to match invalid input. Techniques to do "negative tests" for regular expressions are not very advanced, or at least not widely used.

  This goes to the point of regular expressions being hard to read. Just by looking at a regular expression, it takes a lot of concentration to visualize all possible inputs that should be rejected, but are mistakenly accepted. Ever try to debug someone else's regular expression code?

If there's a resistance to using regular expressions among software developers today, I think it's chiefly due to these two factors.

People tend to think regular expressions are hard; but that's because they're using them wrong. Writing complex one-liners without any comments, indenting or named captures. (You don't cram your complex SQL expression in one line, without comments, indenting or aliases, do you?). So yes, for a lot of people, they don't make sense.

However, if your job has anything to do with parsing text (roughly any web-application out there...) and you don't know regular expression, you suck at your job and you're wasting your own time and that of your employer. There are excellent resources out there to teach you everything about them that you'll ever need to know, and more.

Because they lack the most popular learning tool in the commonly accepted IDEs: There's no Regex Wizard. Not even Autocompletion. You have to code the whole thing all by yourself.

"Regular Expressions: Now You Have Two Problems" is a great article from Jeff Atwood on the matter. Basically, regular expressions are "hard"! They can create new problems. They are effective, however.

I don't think they're that controversial.

I also think you've sort of answered your own question, because you point out how silly it would be to use them everywhere (Not everything is a regular language 2) or to avoid using them at all. You, the programmer, have to make an intelligent decision about when regular expressions will help the code or hurt it. When faced with such a decision, two important things to keep in mind are maintainability (which implies readability) and extensibility.

For those that are particularly averse to them, my guess is that they've never learned to use them properly. I think most people who spend just a few hours with a decent tutorial will figure them out and become fluent very quickly. Here's my suggestion for where to get started:

http://docs.python.org/howto/regex

Although that page talks about regular expressions in the context of Python, I've found the information is very applicable elsewhere. There are a few things that are Python-specific, but I believe they are clearly noted, and easy to remember.

Regular expressions are to strings what arithmetic operators are to numbers, and I wouldn't consider them controversial. I think that even a fairly millitant OO activist like myself (who would tend to choose other objects over strings) would be hard pressed to reject them.

The problem is that regexes are potentially so powerful that you can do things with them that you should use something different for.

A good programmer should know where to use them, and where not. The typical example is parsing non-regular languages (see Deciding whether a language is regular).

I think that you can't go wrong if you at first restrict yourself to real regular expressions (no extensions). Some extensions can make your life a bit easier, but if you find something hard to express as a real regex, this may well be an indication that a regex is not the right tool.

You almost may as well be asking about why goto's are controversial.

Basically, when you get so much "obvious" power, people are apt to abuse them for situations they aren't the best option for. The number of people asking to parse CSVs or XML or HTML in regexes, for example, astounds me. It's the wrong tool for the job. But some users insist on using regexes anyway.

Personally, I try to find that happy medium - use regexes for what they're good for, and avoid them when they're less than optimal.

Note that regexes can still be used to parse CSVs, XML, HTML, etc. But usually not in a single regex.

I don't think "controversial" is the right word.

But I've seen tons of examples where people say "what's the regular expression I need to do such-and-such a string manipulation?" which are X-Y problems.

In other words, they've started from the assumption that a regex is what they need, but they'd be better off with a split(), a translation like perl's tr/// where characters are substituted one for the other, or just an index().

Regular expressions are a serious mystery to a lot of people, including myself. It works great but it's like looking at a math equation. I'm happy to report though that somebody has finally created a consolidated location of various regular expression functions at http://regexlib.com/. Now if Microsoft would only create a regular expression class that would automatically do much of the common stuff like eliminating letters, or filtering dates.

I find regular expressions invaluable at times. When I need to do some "fuzzy" searches, and maybe replaces. When data may vary and have a certain randomness. However, when I need to do a simple search and replace, or check for a string, I do not use regular expressions. Although I know many people who do, they use it for everything. That is the controversy.

If you want to put a tack in the wall, don't use a hammer. Yes, it will work, but by the time you get the hammer, I could put 20 tacks in the wall.

Regular expressions should be used for what they were designed for, and nothing less.

While I think regexes are an essential tool, the most annoying thing about them is that there are different implementations. Slight differences in syntax, modifiers, and -especially- "greed" can make things really chaotic, requiring trial-and-error and sometimes generating puzzling bugs.

In some cases I think you HAVE to use them. For instance to build a lexer.

In my opinion, this is a point of view of people who can write regexp and people who don't (or hardly). I personnaly thing this is a good think for example to valid the input of a form, be it in javascript to warn the user, or in server-side language.

The best valid and normal usage for regex is for email address format validation.

That's a good application of it.

I have used regular expressions countless times as one-offs in TextPad to massage flat files, create csv files, create SQL insert statements and that sort of thing.

Well written regular expressions shouldn't be too slow. Usually the alternatives, like tons of calls to Replace are far slower options. Might as well do it in one pass.

Many situations call for exactly regular expressions and nothing else.

Replacing special non-printing characters with innocuous characters is another good usage.

I can of course imagine that there are some codebases that overuse regular expressions to the detriment of maintainability. I have never seen that myself. I have actually been eschewed by code reviewers for not using regular expressions enough.