Showing posts with label Mysql REGEXP. Show all posts
Showing posts with label Mysql REGEXP. Show all posts

Monday, 25 November 2019

MySQL queries with REGEXP

Official MySQL documentation provides information that using regular expressions is "powerful way of specifying a pattern for a complex search". Is it really such a powerful way of filtering and should be used, or is it a solution that should be avoided? As it usually happens in real life, there are many opinions and no universal answer. Unfortunately, it often turns out that the truth lies somewhere in the middle.
One of my clients asked me yesterday for a little help with a query badly hitting performance of their production server. They were complaining about performance of REGEXP powered query and asked for advice on how to make it efficient.
Original query used in customer's application:
 SELECT id FROM list WHERE user_name REGEXP '^bulba[0-9]+$';
Of course `list` table was properly indexed.
Explain:
 mysql> explain SELECT SQL_NO_CACHE id FROM list WHERE user_name REGEXP '^bulba[0-9]+$'\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: list
         type: index
possible_keys: NULL
          key: id7
      key_len: 24
          ref: NULL
         rows: 4175201
        Extra: Using where; Using index
1 row in set (0.00 sec)
Execution plan pasted above shows that MySQL was using index scan (type: index), which is faster way to get your data than table scan, but still, it's relatively slow and overall performance strongly depends on amount of data it has to parse. Customer case is quite a good example of it. Even assuming that InnoDB buffer pool is big enough to cover all index pages and almost no IO will be needed, it's still over four million rows to examine.
Can we deal with it somehow?
I was playing with it a bit trying different alternatives and here is what I found as much more efficient replacement to original query:
SELECT id FROM list WHERE user_name LIKE 'bulba%' and user_name REGEXP 'bulba[0-9]';
Execution plan again:
 mysql> explain SELECT SQL_NO_CACHE id FROM list WHERE user_name LIKE 'bulba%' AND user_name REGEXP 'bulba[0-9]'\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: list
         type: range
possible_keys: id7
          key: id7
      key_len: 19
          ref: NULL
         rows: 31
        Extra: Using where; Using index
1 row in set (0.00 sec)
Explain looks better. Now, instead of index scan MySQL will use 'id7' to find rows matching criteria with minimum overhead. This is because MySQL can't really use index when REGEXP is the only filtering used. New one, in fact, is doing primary filtering with LIKE, REGEXP just extends it.
And finally, optimization benefit (timings for both queries):
Original query by customer:
19 rows in set (4.96 sec)
Fixed one:
19 rows in set (0.01 sec)
It seems that we were able to achieve the goal.
Of course it does not mean that all the queries using the REGEXP are "bad". That is only a reminder that special care should be taken when writing SQL code with the use of this very nice functionality. You should aware of queries with REGEXP as the only filtering role.

Tuesday, 30 July 2019

How NOT REGEXP Works in MySQL

In MySQL, NOT REGEXP is a negation of the REGEXP operator.
In other words, if the string matches the regular expression provided, the result is 0, otherwise it’s 1. This is the opposite result to what the REGEXP would return (when it isn’t prefixed with NOT).

Syntax

The syntax goes like this:
expr NOT REGEXP pat
Where expr is the input string and pat is the regular expression for which you’re testing the string against.
It’s the equivalent of doing the following:
NOT (expr REGEXP pat)

Example 1 – Basic Usage

Here’s an example of how to use this in a SELECT statement:
SELECT 'Bread' NOT REGEXP '^Br' AS 'Result';
Result:
+--------+
| Result |
+--------+
|      0 |
+--------+
Here, the pattern is matched if the input string starts with Br. It does, but because we use NOT REGEXP, we get a negative result (0).
The above statement is the equivalent of doing this:
SELECT NOT ('Bread' REGEXP '^Br') AS 'Result';
Result:
+--------+
| Result |
+--------+
|      0 |
+--------+

Example 2 – Compared to REGEXP

To make it clear, here we compare the results from REGEXP and NOT REGEXP:
SELECT 
  'Bread' REGEXP '^Br' AS 'REGEXP',
  'Bread' NOT REGEXP '^Br' AS 'NOT REGEXP';
Result:
+--------+------------+
| REGEXP | NOT REGEXP |
+--------+------------+
|      1 |          0 |
+--------+------------+

Example 3 – A Positive Result

The previous examples resulted in 0 for NOT REGEXP, because the string did actually match the pattern. Here’s an example where we get a 1, which indicates that the string doesn’t match:
SELECT 
  'Sofa' REGEXP '^Br' AS 'REGEXP',
  'Sofa' NOT REGEXP '^Br' AS 'NOT REGEXP';
Result:
+--------+------------+
| REGEXP | NOT REGEXP |
+--------+------------+
|      0 |          1 |
+--------+------------+

How NOT REGEXP Works in MySQL

In MySQL, NOT REGEXP is a negation of the REGEXP operator.
In other words, if the string matches the regular expression provided, the result is 0, otherwise it’s 1. This is the opposite result to what the REGEXP would return (when it isn’t prefixed with NOT).

Syntax

The syntax goes like this:
expr NOT REGEXP pat
Where expr is the input string and pat is the regular expression for which you’re testing the string against.
It’s the equivalent of doing the following:
NOT (expr REGEXP pat)

Example 1 – Basic Usage

Here’s an example of how to use this in a SELECT statement:
SELECT 'Bread' NOT REGEXP '^Br' AS 'Result';
Result:
+--------+
| Result |
+--------+
|      0 |
+--------+
Here, the pattern is matched if the input string starts with Br. It does, but because we use NOT REGEXP, we get a negative result (0).
The above statement is the equivalent of doing this:
SELECT NOT ('Bread' REGEXP '^Br') AS 'Result';
Result:
+--------+
| Result |
+--------+
|      0 |
+--------+

Example 2 – Compared to REGEXP

To make it clear, here we compare the results from REGEXP and NOT REGEXP:
SELECT 
  'Bread' REGEXP '^Br' AS 'REGEXP',
  'Bread' NOT REGEXP '^Br' AS 'NOT REGEXP';
Result:
+--------+------------+
| REGEXP | NOT REGEXP |
+--------+------------+
|      1 |          0 |
+--------+------------+

Example 3 – A Positive Result

The previous examples resulted in 0 for NOT REGEXP, because the string did actually match the pattern. Here’s an example where we get a 1, which indicates that the string doesn’t match:
SELECT 
  'Sofa' REGEXP '^Br' AS 'REGEXP',
  'Sofa' NOT REGEXP '^Br' AS 'NOT REGEXP';
Result:
+--------+------------+
| REGEXP | NOT REGEXP |
+--------+------------+
|      0 |          1 |
+--------+------------+

MySQL REGEXP Examples

In MySQL, the REGEXP operator is used to determine whether or not a string matches a regular expression. It’s a synonym for REGEXP_LIKE().
If the string matches the regular expression provided, the result is 1, otherwise it’s 0.

Syntax

The syntax goes like this:
expr REGEXP pat
Where expr is the input string and pat is the regular expression for which you’re testing the string against.

Example 1 – Basic Regular Expression

The most basic regular expression we can use is one that has no special characters in it. Here, we just use a string. If any part of the input string matches that string, it returns a match.
SELECT 
  'Corn' REGEXP 'Corn' AS 'Corn',
  'Acorn' REGEXP 'Corn' AS 'Acorn',
  'Corner' REGEXP 'Corn' AS 'Corner',
  'Cheese' REGEXP 'Corn' AS 'Cheese';
Result:
+------+-------+--------+--------+
| Corn | Acorn | Corner | Cheese |
+------+-------+--------+--------+
|    1 |     1 |      1 |      0 |
+------+-------+--------+--------+

Example 2 – Match the Beginning of a String

In this example, the regular expression specifies that the string must begin with Co.
SELECT 
  'Corn' REGEXP '^Co' AS 'Corn',
  'Acorn' REGEXP '^Co' AS 'Acorn',
  'Cheese' REGEXP '^Co' AS 'Cheese';
Result:
+------+-------+--------+
| Corn | Acorn | Cheese |
+------+-------+--------+
|    1 |     0 |      0 |
+------+-------+--------+

Example 3 – Match the End of a String

In this example, the regular expression specifies that the string must end with rn.
SELECT 
  'Corn' REGEXP 'rn$' AS 'Corn',
  'Acorn' REGEXP 'rn$' AS 'Acorn',
  'Cheese' REGEXP 'rn$' AS 'Cheese';
Result:
+------+-------+--------+
| Corn | Acorn | Cheese |
+------+-------+--------+
|    1 |     1 |      0 |
+------+-------+--------+

Example 4 – Match Any Character

The . character matches any character.
SELECT 
  'Corn' REGEXP '.' AS 'Corn',
  'Cheese' REGEXP '.' AS 'Cheese',
  '' REGEXP '.' AS '';
Result:
+------+--------+---+
| Corn | Cheese |   |
+------+--------+---+
|    1 |      1 | 0 |
+------+--------+---+
This character is typically used in conjunction with other characters to specify further criteria. For example:
SELECT 
  'Corn' REGEXP '^C.rn$' AS 'Corn', 
  'Crn' REGEXP '^C.rn$' AS 'Crn';
Result:
+------+-----+
| Corn | Crn |
+------+-----+
|    1 |   0 |
+------+-----+
Here we specify that the string must start with C, that it must be followed by a character (any character), and that it must end with rn.
Note that this character specifies a single instance of the character. If you want to specify multiple instances (for example ee instead of just e), you’ll need to add more . characters.
SELECT 
  'Tweet' REGEXP '^Tw..t$' AS 'Tweet', 
  'Tweat' REGEXP '^Tw..t$' AS 'Tweat', 
  'Tweet' REGEXP '^Tw.t$' AS 'Tweet', 
  'Twit' REGEXP '^Tw..t$' AS 'Twit';
Result:
+-------+-------+-------+------+
| Tweet | Tweat | Tweet | Twit |
+-------+-------+-------+------+
|     1 |     1 |     0 |    0 |
+-------+-------+-------+------+
Another way to do this is to specify the number of occurrences within curly brackets:
SELECT 
  'Tweet' REGEXP '^Tw.{2}t$' AS 'Tweet', 
  'Tweat' REGEXP '^Tw.{2}t$' AS 'Tweat', 
  'Tweet' REGEXP '^Tw.{1}t$' AS 'Tweet', 
  'Twit' REGEXP '^Tw.{2}t$' AS 'Twit';
Result:
+-------+-------+-------+------+
| Tweet | Tweat | Tweet | Twit |
+-------+-------+-------+------+
|     1 |     1 |     0 |    0 |
+-------+-------+-------+------+
However, if you know the character that you’re looking for, you can specify that character (instead of the . character), as demonstrated in the following example.

Example 5 – Match Zero or More Instances of a Specific Character

We can do the following to specify zero or more instances of the e character:
SELECT 
  'Twet' REGEXP '^Twe*t$' AS 'Twet',
  'Tweet' REGEXP '^Twe*t$' AS 'Tweet',
  'Tweeet' REGEXP '^Twe*t$' AS 'Tweeet',
  'Twt' REGEXP '^Twe*t$' AS 'Twt',
  'Twit' REGEXP '^Twe*t$' AS 'Twit',
  'Twiet' REGEXP '^Twe*t$' AS 'Twiet',
  'Tweit' REGEXP '^Twe*t$' AS 'Tweit';
Result:
+------+-------+--------+-----+------+-------+-------+
| Twet | Tweet | Tweeet | Twt | Twit | Twiet | Tweit |
+------+-------+--------+-----+------+-------+-------+
|    1 |     1 |      1 |   1 |    0 |     0 |     0 |
+------+-------+--------+-----+------+-------+-------+
The first four match but the last three don’t.

Example 6 – Match One or More Instances of a Specific Character

We can modify the previous example so that we only get a match if one or more characters is found (the previous example returned a match if zero or more were found). To do this, we simply use + instead of *:
SELECT 
  'Twet' REGEXP '^Twe+t$' AS 'Twet',
  'Tweet' REGEXP '^Twe+t$' AS 'Tweet',
  'Tweeet' REGEXP '^Twe+t$' AS 'Tweeet',
  'Twt' REGEXP '^Twe+t$' AS 'Twt',
  'Twit' REGEXP '^Twe+t$' AS 'Twit',
  'Twiet' REGEXP '^Twe+t$' AS 'Twiet',
  'Tweit' REGEXP '^Twe+t$' AS 'Tweit';
Result:
+------+-------+--------+-----+------+-------+-------+
| Twet | Tweet | Tweeet | Twt | Twit | Twiet | Tweit |
+------+-------+--------+-----+------+-------+-------+
|    1 |     1 |      1 |   0 |    0 |     0 |     0 |
+------+-------+--------+-----+------+-------+-------+
In this case, the fourth word returns a different result to the previous example.

Example 7 – Match Zero or One Instance of a Specific Character

We can modify the previous example so that we only get a match on zero or one of the desired characters. To do this, we use ?:
SELECT 
  'Twet' REGEXP '^Twe?t$' AS 'Twet',
  'Tweet' REGEXP '^Twe?t$' AS 'Tweet',
  'Tweeet' REGEXP '^Twe?t$' AS 'Tweeet',
  'Twt' REGEXP '^Twe?t$' AS 'Twt',
  'Twit' REGEXP '^Twe?t$' AS 'Twit',
  'Twiet' REGEXP '^Twe?t$' AS 'Twiet',
  'Tweit' REGEXP '^Twe?t$' AS 'Tweit';
Result:
+------+-------+--------+-----+------+-------+-------+
| Twet | Tweet | Tweeet | Twt | Twit | Twiet | Tweit |
+------+-------+--------+-----+------+-------+-------+
|    1 |     0 |      0 |   1 |    0 |     0 |     0 |
+------+-------+--------+-----+------+-------+-------+

Example 8 – Alternation

You can use the | character to match one or another sequence of characters:
SELECT 
  'Tweet' REGEXP 'Tw|et' AS 'Tweet',
  'For Let' REGEXP 'Tw|et' AS 'For Let',
  'Banana' REGEXP 'Tw|et' AS 'Banana';
Result:
+-------+---------+--------+
| Tweet | For Let | Banana |
+-------+---------+--------+
|     1 |       1 |      0 |
+-------+---------+--------+
Here’s another example where I search for whole words:
SELECT 
  'Cat' REGEXP 'Cat|Dog' AS 'Cat',
  'Dog' REGEXP 'Cat|Dog' AS 'Dog',
  'Doggone' REGEXP 'Cat|Dog' AS 'Doggone',
  'Banana' REGEXP 'Cat|Dog' AS 'Banana';
Result:
+-----+-----+---------+--------+
| Cat | Dog | Doggone | Banana |
+-----+-----+---------+--------+
|   1 |   1 |       1 |      0 |
+-----+-----+---------+--------+
We still get a match even when our regular expression matches only part of the the string.

Example 9 – Match Zero or More Instances of a Sequence

You can use brackets along with the asterisk ()* to specify zero or more instances of a sequence:
SELECT 
  'Banana' REGEXP '(an)*' AS 'Banana',
  'Land' REGEXP '(an)*' AS 'Land',
  'Cheese' REGEXP '(an)*' AS 'Cheese';
Result:
+--------+------+--------+
| Banana | Land | Cheese |
+--------+------+--------+
|      1 |    1 |      1 |
+--------+------+--------+
Another example:
SELECT 
  'Banana' REGEXP '^B(an)*d$' AS 'Banana',
  'Band' REGEXP '^B(an)*d$' AS 'Band',
  'Bald' REGEXP '^B(an)*d$' AS 'Bald',
  'Bad' REGEXP '^B(an)*d$' AS 'Bad';
Result:
+--------+------+------+-----+
| Banana | Band | Bald | Bad |
+--------+------+------+-----+
|      0 |    1 |    0 |   0 |
+--------+------+------+-----+

Example 10 – Repetition

As seen in a previous example, you can use curly brackets to specify repetition. This notation provides a more general way of writing regular expressions than some of the previous examples:
SELECT 
  'Tweeet' REGEXP 'e{3}' AS 'Tweeet',
  'Tweet' REGEXP 'e{3}' AS 'Tweet';
Result:
+--------+-------+
| Tweeet | Tweet |
+--------+-------+
|      1 |     0 |
+--------+-------+

Example 11 – Range

You can use the hyphen character to specify a range. Here’s an example that specifies a range of numbers:
SELECT 
  'Tweet 123' REGEXP '[0-9]' AS 'Tweet 123',
  'Tweet ABC' REGEXP '[0-9]' AS 'Tweet ABC';
Result:
+--------+-------+
| Tweeet | Tweet |
+--------+-------+
|      1 |     0 |
+--------+-------+
And the following example specifies a range of letters:
SELECT 
  'Tweet 123' REGEXP '[A-Z]' AS 'Tweet 123',
  'ABC' REGEXP '[A-Z]' AS 'ABC',
  '123' REGEXP '[A-Z]' AS '123';
Result:
+--------+-------+
| Tweeet | Tweet |
+--------+-------+
|      1 |     0 |
+--------+-------+
Here’s what happens if we limit the range of numbers:
SELECT 
  '123' REGEXP '[1-3]' AS '123',
  '012' REGEXP '[1-3]' AS '012',
  '045' REGEXP '[1-3]' AS '045';
Result:
+-----+-----+-----+
| 123 | 012 | 045 |
+-----+-----+-----+
|   1 |   1 |   0 |
+-----+-----+-----+

Example 12 – Not Within a Range

We can use the ^ character to modify the previous example so that the specified range of characters are excluded:
SELECT 
  '123' REGEXP '[^1-3]' AS '123',
  '012' REGEXP '[^1-3]' AS '012',
  '045' REGEXP '[^1-3]' AS '045';
Result:
+-----+-----+-----+
| 123 | 012 | 045 |
+-----+-----+-----+
|   0 |   1 |   1 |
+-----+-----+-----+
So we get the opposite result to the previous example.

3 Ways to Detect if a String Matches a Regular Expression in MySQL

MySQL has a number of functions and operators that allow us to perform operations using regular expressions (regex). This article presents two operators and one function that enable us to find out if a string matches a regular expression specified by a given pattern.
These regex functions and operators are:
These are all basically equivalent, as the operators (the second two) are both synonyms of the function (the first one). In any case, you can see examples of all three in action below.

The REGEXP_LIKE() Function

First, let’s look at the function. Here’s an example of running a regex search using the REGEXP_LIKE() function:
SELECT 
  REGEXP_LIKE('Car', '^C') AS 'Match',
  REGEXP_LIKE('Bar', '^C') AS 'No Match';
Result:
+-------+----------+
| Match | No Match |
+-------+----------+
|     1 |        0 |
+-------+----------+
The first string matches (because it starts with C) so the result is 1. The second string doesn’t match and so the result is 0.
However, this function can be much more useful than simply returning a 1 or 0. For example, it can be added to the WHERE clause when querying a database. In this case, we can get a list of rows that contain a match for the pattern.
Here’s an example:
SELECT AlbumId, AlbumName
FROM Albums
WHERE REGEXP_LIKE(AlbumName, '^Power');
Result:
+---------+------------+
| AlbumId | AlbumName  |
+---------+------------+
|       1 | Powerslave |
|       2 | Powerage   |
+---------+------------+
Here’s the full table:
SELECT AlbumId, AlbumName
FROM Albums;
Result:
+---------+--------------------------+
| AlbumId | AlbumName                |
+---------+--------------------------+
|       1 | Powerslave               |
|       2 | Powerage                 |
|       3 | Singing Down the Lane    |
|       4 | Ziltoid the Omniscient   |
|       5 | Casualties of Cool       |
|       6 | Epicloud                 |
|       7 | Somewhere in Time        |
|       8 | Piece of Mind            |
|       9 | Killers                  |
|      10 | No Prayer for the Dying  |
|      11 | No Sound Without Silence |
|      12 | Big Swing Face           |
|      13 | Blue Night               |
|      14 | Eternity                 |
|      15 | Scandinavia              |
|      16 | Long Lost Suitcase       |
|      17 | Praise and Blame         |
|      18 | Along Came Jones         |
|      19 | All Night Wrong          |
|      20 | The Sixteen Men of Tain  |
+---------+--------------------------+

The REGEXP Operator

Here’s how to do the same thing using the REGEXP operator:
SELECT 
  'Car' REGEXP '^C' AS 'Match',
  'Bar' REGEXP '^C' AS 'No Match';
Result:
+-------+----------+
| Match | No Match |
+-------+----------+
|     1 |        0 |
+-------+----------+
And the database example:
SELECT AlbumId, AlbumName
FROM Albums
WHERE AlbumName REGEXP '^Power';
Result:
+---------+------------+
| AlbumId | AlbumName  |
+---------+------------+
|       1 | Powerslave |
|       2 | Powerage   |
+---------+------------+
You can also use NOT REGEXP to return the opposite result.

The RLIKE Operator

And here it is using RLIKE:
SELECT 
  'Car' RLIKE '^C' AS 'Match',
  'Bar' RLIKE '^C' AS 'No Match';
Result:
+-------+----------+
| Match | No Match |
+-------+----------+
|     1 |        0 |
+-------+----------+
And the database example:
SELECT AlbumId, AlbumName
FROM Albums
WHERE AlbumName RLIKE '^Power';
Result:
+---------+------------+
| AlbumId | AlbumName  |
+---------+------------+
|       1 | Powerslave |
|       2 | Powerage   |
+---------+------------+
In this case I simply swapped REGEXP for RLIKE and left the rest of the code alone.
You can also use NOT RLIKE to return the opposite result.

More REGEX Functions

MySQL also includes a few other regex functions and operators.  Three of these are listed below. Technically, you could also use the first two to “detect” whether a string matches a regex pattern (in which case, maybe this article should be titled “5 Ways to Detect if a String Matches a Regular Expression in MySQL” instead of just “3 ways…”).
Anyway, here are three more regex functions:
  • You can use the REGEXP_INSTR() function to return the starting index of a substring that matches the regular expression pattern.
  • The REGEXP_SUBSTR() function returns the substring that matches the given regular expression pattern.
  • And the REGEXP_REPLACE() function replaces occurrences of the substring within a string that matches the given regular expression pattern.