Unable to delete whitespace from string with tr, sed

user_10376320 0
Viewed 234

I have a file that contains a whitespace character that I'm not able to successfully remove with command-line tools such as tr or sed. Here's the input:

2,  78  ,, 1
6,    74, ,1

and I want the output to look like:

2,78,,1
6,74,,1

Attempts

If I try tr -d "[[:space:]] the result is 2, 78,,16,74,,1 which leaves a space character and removes the newline.

If I try sed 's/[[:space:]]//g' the result is

2, 78,,1
6,74,,1

which still leaves the space. I converted the string to hex, and it seems the offending character is a0, but even then the results are not what I'd expect: sed 's/\xa0//g' yields

2, �78  ,, 1
6,    74, ,1

Question

What is that whitespace character that is not getting caught by the [[:space:]] character class? How can I delete it?

removing-whitespace tr sed utf-8 character-encoding
2 Answers

The offending character is a UTF-8-encoded non-breaking space, with hex representation \xc2\xa0. You can remove all spaces, including non-breaking spaces, with

sed -E 's/[[:space:]]|\xc2\xa0//g'

Explanation

  • -E turns on extended regex to allow the | to represent logical OR
  • 's/pattern/replacement/' substitutes pattern matches with the replacement text (in this case, an empty string), with /g repeating the pattern substitution multiple times per line
  • [[:space:]] matches most whitespace characters, including spaces and tabs
  • \xc2\xa0 is the hex code for the UTF-8 non-breaking space
user_103763200

The characters you want to remove are the non-printable ones (i.e the ones not in the [:print:] character class) rather than the ones just the ones in the [:space:] character class:

$ printf 'foo\xc2\xa0bar\n' > file
$ cat file
foo bar
$ tr -dc '[:print:]' < file
foobar$

but I notice the equivalent doesn't work in GNU sed or GNU awk and idk why.

user_17450010
Related