An allowed character is being percent encoded

user_5606659 0
Viewed 135

Documentation for the string method addingPercentEncoding(withAllowedCharacters:):

Returns a new string made from the receiver by replacing all characters not in the specified set with percent-encoded characters.

The predefined set CharacterSet.alphanumerics says:

Returns a character set containing the characters in Unicode General Categories L*, M*, and N*.

The L (Letter) category consists of 5 subcategories: Ll,Lm,Lt,Lu,Lo. So I assume L* means all of L's subcategories.

I'll choose to look at the Ll subcategory (https://www.compart.com/en/unicode/category/Ll#UNC_SCRIPTS), and pick the character "æ" (U+00E6). I can then see that the alphanumerics character set indeed contains this character. But when I add percent encoding to a string containing this character, it gets percent encoded.

"\u{E6}" // "æ"
CharacterSet.alphanumerics.contains("\u{E6}") // true
"æ".addingPercentEncoding(withAllowedCharacters: .alphanumerics) // "%C3%A6" 

// Let's try with "a"
"\u{61}" // "a"
CharacterSet.alphanumerics.contains("\u{61}") // true
"a".addingPercentEncoding(withAllowedCharacters: .alphanumerics) // "a"

Why does this happen? It's in the allowed character set that I passed in, so it shouldn't be replaced, right?

I feel like it has something to do with the fact that "a" (U+0061) is also 0x61 in UTF-8 but "æ" (U+00E6) is [0xC3, 0xA6]; not 0xE6. Or that it takes up more than 1 byte?

String(data: Data([0x61]), encoding: .utf8)! // "a"
String(data: Data([0xC3, 0xA6]), encoding: .utf8)! // "æ"
String(data: Data([0xE6]), encoding: .utf8)! // crashes

Update

Is it because the percent encoding algorithm converts the string to Data and goes through 1 byte at a time? so it'll look at 0xC3 which isn't an allowed character, so that gets percent encoded. Then it'll look at 0xA6 which also isn't an allowed character, so that gets percent encoded too. So allowed characters technically have to be a single byte?

swift
1 Answers

A truly allowed character has to be in the allowed character set, and be an ASCII character. Thanks @alobaili for pointing that out.

If you're curious, the predefined set CharacterSet.alphanumerics contains 129172 characters in total, but only 62 are truly allowed when this set is passed to a string's addingPercentEncoding(allowedSet:) method.

A quick way to inspect all the truly allowed characters in a particular CharacterSet can be done like so:

func inspect(charSet: CharacterSet) {
    var characters: [String] = []
    for char: UInt8 in 0..<128 { // ASCII range
        let u = UnicodeScalar(char)
        if charSet.contains(u) {
            characters.append(String(u))
        }
    }
    print("Characters:", characters.count)
    print(characters)
}

inspect(charSet: .alphanumerics) // [a-z, A-Z, 0-9]

This is handy as you can't simply iterate through a CharacterSet. It can be useful to know what those allowed elements are. For example, the predefined CharacterSet.urlQueryAllowed only says:

Returns the character set for characters allowed in a query URL component.

We can know what those allowed characters are:

inspect(charSet: .urlQueryAllowed)

// Characters: 81
// ["!", "$", "&", "\'", "(", ")", "*", "+", ",", "-", ".", "/", "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", ":", ";", "=", "?", "@", "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z", "_", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "~"]

Just For Fun

There's another (long, sure-fire) way, which looks at all the characters in the set (not just the ASCII ones), and compares a string of the character itself, with the string after adding percent encoding with only that character in the allowed set. When those two are equal then you know it really is an allowed character. Code adapted from this helpful article.

func inspect(charSet: CharacterSet) {
    var characters: [String] = []
    var allowed: [String] = []
    var asciiCount = 0
    for plane: UInt8 in 0..<17 {
        if charSet.hasMember(inPlane: plane) {
            let planeStart = UInt32(plane) << 16
            let nextPlaneStart = (UInt32(plane) + 1) << 16
            for char: UTF32Char in planeStart..<nextPlaneStart {
                if let u = UnicodeScalar(char), charSet.contains(u) {
                    let s = String(u)
                    characters.append(s)
                    
                    if s.addingPercentEncoding(withAllowedCharacters: CharacterSet([u])) == s {
                        allowed.append(s)
                    }
                    
                    if u.isASCII {
                        asciiCount += 1
                    }
                }
            }
        }
    }
    print("Characters:", characters.count)
    print("Allowed:", allowed.count)
    print("ASCII:", asciiCount)
}

inspect(charSet: .alphanumerics)

// Characters: 129172
// Allowed: 62
// ASCII: 62
user_56066590
Related