Removing non-letter Characters from String leaves UTF-16 High Surrogates

user_9203198 0
Viewed 103

I am using the regex [^\\p{L}] and java.util.regex.Matcher#replaceAll(String) to match and remove all non-letter characters from a string. I noticed that for characters containing UTF-16 surrogates, replaceAll() creates a structurally invalid string (OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1).

First a working example:

import java.util.regex.Matcher;
import java.util.regex.Pattern;

class Main {  
  public static void main(String args[]) { 
    Pattern p = Pattern.compile("[^\\p{L}]");
    System.out.println(p.matcher("abcဍ*").replaceAll(""));
  } 
}

The above program prints abcဍ as expected (ဍ is MYANMAR LETTER DDA).

Now let's test the character "" (\uD835\uDD0D, MATHEMATICAL FRAKTUR CAPITAL J, Category: Letter, Uppercase [Lu]), which contains high surrogates:

import java.util.regex.Matcher;
import java.util.regex.Pattern;

class Main {  
  public static void main(String args[]) {
    String original = ""; // "\uD835\uDD0D"
    
    Pattern p1 = Pattern.compile("[\\p{L}]");   // try regex without negation first
    Matcher m1 = p1.matcher(original);
    String r1 = m1.replaceAll("");
    System.out.println("r1: " + r1);
  
    Pattern p2 = Pattern.compile("[^\\p{L}]");  // now try regex with negation
    Matcher m2 = p2.matcher(original);
    String r2 = m2.replaceAll("");
    System.out.println("r2: " + r2);
    System.out.println("r2 length: " + r2.length());
    System.out.println("r2 char(0): " + (int) r2.charAt(0));

    System.out.println("original: " + original);
  } 
}

Output:

r1:                     // r1 = empty string as expected
r2: ?                   // r2 = broken string
r2 length: 1            
r2 char(0): 55349       // 0xD835 (high surrogate)
original:

Other examples I tested that produce structurally invalid strings:

  • (\uD801\uDCD8), OSAGE SMALL LETTER A, Category: Letter, Lowercase [Ll]
  • (\uD806\uDECA), PAU CIN HAU LETTER KHA, Category: Letter, Other [Lo]

Is my regex broken, or is this a bug in the Java Class Library? If this is a bug, is there a workaround?

regex-negation regex java string
2 Answers

In your current Java version, the problem is that \p{L} is a placeholder, mnemonic for a more complex pattern that includes alternatives with multicharacter sequences.

That means, the \p{L} pattern cannot be used inside a character class, as character classes are not meant to match char sequences, only individual chars.

Thus, you need to make sure you use it outside of a character class. Here, you can simply use the reverse Unicode category class, \P{L}, to match any char but a Unicode letter. The following:

Pattern p2 = Pattern.compile("\\P{L}");

returns what you need.

In case you cannot replace with the reverse class, you can use an alterenation. For example, if you have [^\d\s\p{L}], you can use (?:\P{L}|[^\d\s]).

Note that your code looks working in Java 17+ (confirmed by user16320675).

user_38329700

For cases like this I wrote my own utility that allows to convert any String to unicode sequence and vice-versa. Here is a sample:

result = "Hello World";
result = StringUnicodeEncoderDecoder.encodeStringToUnicodeSequence(result);
System.out.println(result);
result = StringUnicodeEncoderDecoder.decodeUnicodeSequenceToString(result);
System.out.println(result);

The output of this code is:

\u0048\u0065\u006c\u006c\u006f\u0020\u0057\u006f\u0072\u006c\u0064
Hello World

The library can be found at Maven Central or at Github It comes as maven artifact and with sources and javadoc

Here is javadoc for the class StringUnicodeEncoderDecoder. This helps me a lot when I need to diagnose problems like yours, but also in extreme scenarios you can convert String into unicodes and modify it as codes and convert it back. It might be a workaround for you until you find better solution

user_58024170
Related