MYSQL error : #3699 - Timeout exceeded in regular expression match

Viewed 2817

Running a MYSQL query with REGEX and I get a timeout error.

I have a Bitnami NGINX Wordpress stack on AWS with Ubuntu 16.

Using MYSQL 8.0.16

What settings in MYSQL (or whereever else) do I need to change to stop this error?

help appreciated..

tried playing with the my.cnf file and restarting MYSQL but I dont know which timeout setting, there seems to be several

this is the SQL query

REPLACE INTO wp_term_relationships (term_taxonomy_id, object_id) SELECT '272',  ID
FROM  `wp_posts` 
   WHERE (
   post_title REGEXP 'German|Deutsch|Berlin|Hamburg|Munich|München|Cologne|Köln|Frankfurt|Stuttgart|Düsseldorf|Dortmund|Essen|Leipzig|Bremen|Dresden|Hanover|Hannover|Nuremberg|Nürnberg|Duisburg|Bochum|Wuppertal|Bielefeld|Bonn|Münster|Karlsruhe|Mannheim|Augsburg|Wiesbaden|Gelsenkirchen|Mönchengladbach|Braunschweig|Chemnitz|Kiel|Aachen|Halle|Saale|Magdeburg|Freiburg|Krefeld|Lübeck|Oberhausen|Erfurt|Mainz|Rostock|Kassel|Hagen|Hamm|Saarbrücken|Mülheim|Potsdam|Ludwigshafen|Oldenburg|Leverkusen|Osnabrück|Solingen|Heidelberg|Herne|Neuss|Darmstadt|Paderborn|Regensburg|Ingolstadt|Würzburg|Fürth|Wolfsburg|Offenbach|Ulm|Heilbronn|Pforzheim|Göttingen|Bottrop|Trier|Recklinghausen|Reutlingen|Bremerhaven|Koblenz|Bergisch Gladbach|Jena|Remscheid|Erlangen|Moers|Siegen|Hildesheim|Salzgitter|Baden-Württemberg|Bavaria|Bayern|Berlin|Brandenburg|Bremen|Hesse|Hessen|Lower Saxony|Niedersachsen|Mecklenburg-Vorpommern|North Rhine-|Westphalia|Nordrhein-Westfalen|Rhineland-Palatinate|Rheinland-Pfalz|Saarland|Saxony|Sachsen|Saxony-Anhalt|Sachsen-Anhalt|Schleswig-Holstein|Thuringia|Thüringen' 
   OR post_content REGEXP 'German|Deutsch|Berlin|Hamburg|Munich|München|Cologne|Köln|Frankfurt|Stuttgart|Düsseldorf|Dortmund|Essen|Leipzig|Bremen|Dresden|Hanover|Hannover|Nuremberg|Nürnberg|Duisburg|Bochum|Wuppertal|Bielefeld|Bonn|Münster|Karlsruhe|Mannheim|Augsburg|Wiesbaden|Gelsenkirchen|Mönchengladbach|Braunschweig|Chemnitz|Kiel|Aachen|Halle|Saale|Magdeburg|Freiburg|Krefeld|Lübeck|Oberhausen|Erfurt|Mainz|Rostock|Kassel|Hagen|Hamm|Saarbrücken|Mülheim|Potsdam|Ludwigshafen|Oldenburg|Leverkusen|Osnabrück|Solingen|Heidelberg|Herne|Neuss|Darmstadt|Paderborn|Regensburg|Ingolstadt|Würzburg|Fürth|Wolfsburg|Offenbach|Ulm|Heilbronn|Pforzheim|Göttingen|Bottrop|Trier|Recklinghausen|Reutlingen|Bremerhaven|Koblenz|Bergisch Gladbach|Jena|Remscheid|Erlangen|Moers|Siegen|Hildesheim|Salzgitter|Baden-Württemberg|Bavaria|Bayern|Berlin|Brandenburg|Bremen|Hesse|Hessen|Lower Saxony|Niedersachsen|Mecklenburg-Vorpommern|North Rhine-|Westphalia|Nordrhein-Westfalen|Rhineland-Palatinate|Rheinland-Pfalz|Saarland|Saxony|Sachsen|Saxony-Anhalt|Sachsen-Anhalt|Schleswig-Holstein|Thuringia|Thüringen'
   )
   AND post_type = 'post'

#3699 - Timeout exceeded in regular expression match.

*update - not sure how to add an image to comments, so will add here.

variable settings, cheers @Rick James on how list them

enter image description here

3 Answers
SET GLOBAL regexp_time_limit=1024;

(see here for the documentation)

You should make the regex into a trie.
This is less work for the engine, it only has to do at most 16 steps to fail
instead of the current 110 steps it's doing now.

Try

A(?:achen|ugsburg)|B(?:a(?:den-Württemberg|varia|yern)|er(?:gisch[ ]Gladbach|lin)|ielefeld|o(?:chum|nn|ttrop)|r(?:a(?:ndenbur|unschwei)g|eme(?:rhave)?n))|C(?:hemnitz|ologne)|D(?:armstadt|eutsch|ortmund|resden|uisburg|üsseldorf)|E(?:r(?:furt|langen)|ssen)|F(?:r(?:ankfurt|eiburg)|ürth)|G(?:e(?:lsenkirche|rma)n|öttingen)|H(?:a(?:gen|lle|m(?:burg|m)|n(?:n)?over)|e(?:i(?:delberg|lbronn)|rne|ssen?)|ildesheim)|Ingolstadt|Jena|K(?:a(?:rlsruhe|ssel)|iel|oblenz|refeld|öln)|L(?:e(?:ipzig|verkusen)|ower[ ]Saxony|udwigshafen|übeck)|M(?:a(?:gdeburg|inz|nnheim)|ecklenburg-Vorpommern|oers|unich|önchengladbach|ü(?:lheim|n(?:chen|ster)))|N(?:euss|iedersachsen|or(?:drhein-Westfalen|th[ ]Rhine-)|uremberg|ürnberg)|O(?:berhausen|ffenbach|ldenburg|snabrück)|P(?:aderborn|forzheim|otsdam)|R(?:e(?:cklinghausen|gensburg|mscheid|utlingen)|h(?:einland-Pfalz|ineland-Palatinate)|ostock)|S(?:a(?:a(?:le|r(?:brücken|land))|chsen(?:-Anhalt)?|lzgitter|xony(?:-Anhalt)?)|chleswig-Holstein|iegen|olingen|tuttgart)|T(?:h(?:uringia|üringen)|rier)|Ulm|W(?:estphalia|iesbaden|olfsburg|uppertal|ürzburg)

Change the (?: to ( if your engine doesn't support non-capture syntax.
Also, you can change [ ] to just space if you want to on the compressed
regex.

https://regex101.com/r/iEzw33/1

Expanded

    A
    (?: achen | ugsburg )
 |  B
    (?:
         a
         (?: den-Württemberg | varia | yern )
      |  er
         (?: gisch [ ] Gladbach | lin )
      |  ielefeld
      |  o
         (?: chum | nn | ttrop )
      |  r
         (?:
              a
              (?: ndenbur | unschwei )
              g
           |  eme
              (?: rhave )?
              n
         )
    )
 |  C
    (?: hemnitz | ologne )
 |  D
    (?:
         armstadt
      |  eutsch
      |  ortmund
      |  resden
      |  uisburg
      |  üsseldorf
    )
 |  E
    (?:
         r
         (?: furt | langen )
      |  ssen
    )
 |  F
    (?:
         r
         (?: ankfurt | eiburg )
      |  ürth
    )
 |  G
    (?:
         e
         (?: lsenkirche | rma )
         n
      |  öttingen
    )
 |  H
    (?:
         a
         (?:
              gen
           |  lle
           |  m
              (?: burg | m )
           |  n
              (?: n )?
              over
         )
      |  e
         (?:
              i
              (?: delberg | lbronn )
           |  rne
           |  ssen?
         )
      |  ildesheim
    )
 |  Ingolstadt
 |  Jena
 |  K
    (?:
         a
         (?: rlsruhe | ssel )
      |  iel
      |  oblenz
      |  refeld
      |  öln
    )
 |  L
    (?:
         e
         (?: ipzig | verkusen )
      |  ower [ ] Saxony
      |  udwigshafen
      |  übeck
    )
 |  M
    (?:
         a
         (?: gdeburg | inz | nnheim )
      |  ecklenburg-Vorpommern
      |  oers
      |  unich
      |  önchengladbach
      |  ü
         (?:
              lheim
           |  n
              (?: chen | ster )
         )
    )
 |  N
    (?:
         euss
      |  iedersachsen
      |  or
         (?: drhein-Westfalen | th [ ] Rhine- )
      |  uremberg
      |  ürnberg
    )
 |  O
    (?:
         berhausen
      |  ffenbach
      |  ldenburg
      |  snabrück 
    )
 |  P
    (?: aderborn | forzheim | otsdam )
 |  R
    (?:
         e
         (?:
              cklinghausen
           |  gensburg
           |  mscheid
           |  utlingen 
         )
      |  h
         (?: einland-Pfalz | ineland-Palatinate )
      |  ostock
    )
 |  S
    (?:
         a
         (?:
              a
              (?:
                   le
                |  r
                   (?: brücken | land )
              )
           |  chsen
              (?: -Anhalt )?
           |  lzgitter
           |  xony
              (?: -Anhalt )?
         )
      |  chleswig-Holstein
      |  iegen
      |  olingen
      |  tuttgart
    )
 |  T
    (?:
         h
         (?: uringia | üringen )
      |  rier
    )
 |  Ulm
 |  W
    (?:
         estphalia
      |  iesbaden
      |  olfsburg
      |  uppertal
      |  ürzburg
    )

Lets check the speed:

Regex1:   A(?:achen|ugsburg)|B(?:a(?:den-Württemberg|varia|yern)|er(?:gisch[ ]Gladbach|lin)|ielefeld|o(?:chum|nn|ttrop)|r(?:a(?:ndenbur|unschwei)g|eme(?:rhave)?n))|C(?:hemnitz|ologne)|D(?:armstadt|eutsch|ortmund|resden|uisburg|üsseldorf)|E(?:r(?:furt|langen)|ssen)|F(?:r(?:ankfurt|eiburg)|ürth)|G(?:e(?:lsenkirche|rma)n|öttingen)|H(?:a(?:gen|lle|m(?:burg|m)|n(?:n)?over)|e(?:i(?:delberg|lbronn)|rne|ssen?)|ildesheim)|Ingolstadt|Jena|K(?:a(?:rlsruhe|ssel)|iel|oblenz|refeld|öln)|L(?:e(?:ipzig|verkusen)|ower[ ]Saxony|udwigshafen|übeck)|M(?:a(?:gdeburg|inz|nnheim)|ecklenburg-Vorpommern|oers|unich|önchengladbach|ü(?:lheim|n(?:chen|ster)))|N(?:euss|iedersachsen|or(?:drhein-Westfalen|th[ ]Rhine-)|uremberg|ürnberg)|O(?:berhausen|ffenbach|ldenburg|snabrück)|P(?:aderborn|forzheim|otsdam)|R(?:e(?:cklinghausen|gensburg|mscheid|utlingen)|h(?:einland-Pfalz|ineland-Palatinate)|ostock)|S(?:a(?:a(?:le|r(?:brücken|land))|chsen(?:-Anhalt)?|lzgitter|xony(?:-Anhalt)?)|chleswig-Holstein|iegen|olingen|tuttgart)|T(?:h(?:uringia|üringen)|rier)|Ulm|W(?:estphalia|iesbaden|olfsburg|uppertal|ürzburg)
Completed iterations:   50  /  50     ( x 1000 )
Matches found per iteration:   110
Elapsed Time:    5.74 s,   5736.58 ms,   5736578 µs
Matches per sec:   958,759

Wow, a million words matched per second !

"Timeout" is presumably due to a slow query against a large table. So, let's address the speed of the query.

  • REPLACE is DELETE + INSERT.
  • REPLACE is slow due to theDELETE. And it will loseterm_order` if you have already set it.
  • Consider using INSERT IGNORE instead.
  • The WHERE clause, due to the OR and the REGEXP, will need to scan the entire table. A FULLTEXT index could avoid that; see below.
  • Refactoring the REGEXP won't help much; changing to LIKE is likely to be slower; etc.
  • If the timeout is truely inside the REGEXP engine, then you may need to run the query a few cities at a time.

FULLTEXT would greatly improve the speed. But, if you don't want to go that direction, the next best is changing REPLACE to INSERT IGNORE.

The FULLTEXT approach is:

ALTER TABLE wp_posts ADD FULLTEXT(post_title, post_content);  -- one-time task

INSERT IGNORE INTO ... (...)
    SELECT 272, ID
        FROM wp_posts
        WHERE MATCH(post_title, post_content)
              AGAINST('German Deutsch Berlin Hamburg Munich München ...' IN BOOLEAN MODE)

(There may be an issue with handling of accents, depending on the version of MySQL and which ENGINE is used.)

Related