How to get rid of non-ascii characters in ruby

Admin 1
Viewed 57685

I have a Ruby CGI (not rails) that picks photos and captions from a web form. My users are very keen on using smart quotes and ligatures, they are pasting from other sources. My web app does not deal well with these non-ASCII characters, is there a quick Ruby string manipulation routine that can get rid of non-ASCII chars?

cgi ascii unicode ruby
8 Answers

class String
 def remove_non_ascii(replacement="") 
   self.gsub(/[\u0080-\u00ff]/, replacement)
 end
end
user_1102130

If you have active support you can use I18n.transliterate

I18n.transliterate("áëëçüñżλφθΩ")
"aee?cunz?????"

Or if you don't want the question marks...

I18n.transliterate("áëëçüñżλφθΩ", replacement: "")
"aeecunz"

Note that this doesn't remove invalid byte sequences it just replaces non ascii characters. For my use case this was what I wanted though and was simple.

user_26661200

Quick GS revealed this discussion which suggests the following method:

class String
  def remove_nonascii(replacement)
    n=self.split("")
    self.slice!(0..self.size)
    n.each { |b|
     if b[0].to_i< 33 || b[0].to_i>127 then
       self.concat(replacement)
     else
       self.concat(b)
     end
    }
    self.to_s
  end
end
user_900420

No there isn't short of removing all characters beside the basic ones (which is recommended above). The best slution would be handling these names properly (since most filesystems today do not have any problems with Unicode names). If your users paste in ligatures they sure as hell will want to get them back too. If filesystem is your problem, abstract it away and set the filename to some md5 (this also allows you to easily shard uploads into buckets which scan very quickly since they never have too many entries).

user_1538860
Related