Remove from a string the characters where bytesize is greater than 2 with Ruby

user_10347572 0
Viewed 208

I have a problem with mysql and certain characters. If a user enters "hello ●", I obtain this error:

Mysql2::Error: Incorrect string value: '\\xE2\\x97\\x8F he...' for column 'subject'

I would like to exclude all characters whose bytesize is greater than two, i.e., keep French characters like é, à, ç, and remove emojis or characters like ●.

Given string = "hèllö>●!", I would like to obtain "hèllö>!". In order to do so, I wrote this:

def bytesize(var)
   var.each_char do |char|
      puts char.bytesize
   end
end

bytesize(string)
1
2
1
1
2
1
3
1
# => "hèllö>●!"

which is not what I expected. What is the best way to remove from all characters whose the bytesize is greater than two from a string?

I don't do that in the model because I can manage this with a gem, but my problem appears when a job wants to put the string in the logs of Amazon SES.

ruby-on-rails ruby regex
2 Answers

I suspect that you are getting that error message because you have the wrong column text encoding. If you are using Unicode in your system, and this day and age you should be, your column type should be utf8mb4. See this on how to change your column types.

Taking your comment into account the following will remove any characters outside the BMP

sentence.gsub(/[\u{10000}-\u{10FFFF}]/,'')
user_42438400

Elaborating on OP's efforts, not using regular expressions:

string = "hèllö>●!"

cleaned = string.each_char.with_object("") do |char, str|
   str << char unless char.bytesize > 2
end

p cleaned
user_2903940
Related