Question mark not showing up in string

user_2678657 0
Viewed 367

I copied a piece of text from a website. This piece of text contains a space. I later try to manipulate this string in C#, but my code doesn't recognize the space. I started digging deeper, so I tried the following Powershell command to convert the string to hexadecimal to see what's going on:

"2+1 53" | Format-Hex

(see screenshot here: Powershell code)

As you can see in the image, it shows that the result is:

32 2B 31 3F 35 33

which converted back to normal text is

2+1?53

Notice that the question mark wasn't present in my original string. What is going on? How can a question mark be present but not show up? Or where did it come from if it was not present in my original string?

Update: Perhaps I should stress that I need to figure out what that "space" character is, so that I can later get rid of it using "replace" method.

encoding powershell string
4 Answers

Most likely, there's another character in that text, which is not a space. You can check it by putting the text into a file and then using

Get-Content C:\temp\file.txt | Format-Hex

To reproduce, I used that text:

Get-Service –Name BITS
#           ^ it's not a normal dash, check it at http://asciivalue.com/index.php

This happens if I paste into console window:

31.88 ms | C:\> "Get-Service -Name BITS" | Format-Hex


           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   47 65 74 2D 53 65 72 76 69 63 65 20 3F 4E 61 6D  Get-Service ?Nam
00000010   65 20 42 49 54 53                                e BITS

And that when I get it from the script:

60.02 ms | C:\> Get-Content C:\temp\script.ps1 | Format-Hex


           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   47 65 74 2D 53 65 72 76 69 63 65 20 3F 3F 3F 4E  Get-Service ???N
00000010   61 6D 65 20 42 49 54 53                          ame BITS

As you can see, that character is being converted to question mark (3F in hex output) or triple question mark (3F 3F 3F) while getting content from file.

user_99025550

It depends on the version of PowerShell. I have 5.1, there the default encoding of Format-Hex is ASCII, which will replace every non-ASCII character (like your space) with a question mark.

Specify a different encoding to prevent non-ASCII characters from being replaced. Example:

PS> "⇆" | Format-Hex -Encoding Unicode


           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   C6 21                                            Æ!

Here, the code point is U+21C6. Google that and you will find out what it represents.

user_24859660

A regular space is 0x20. There are many unicode spaces. http://jkorpela.fi/chars/spaces.html How did you make the string? Here's an example of EN SPACE (nut), U+2002. You should be able to copy and paste this yourself. Hmm, in powershell 7 for windows, the special space doesn't paste.

[int[]][char[]]'foo bar' | % tostring x

66
6f
6f
2002
62
61
72


'foo bar' | Format-Hex -Encoding BigEndianUnicode


           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   00 66 00 6F 00 6F 20 02 00 62 00 61 00 72        .f.o.o ..b.a.r

Format-hex will translate it to ascii by default in powershell 5. Unicode characters will be replaced by 3F or ?. 

'foo bar' | format-hex -Encoding ascii               


Label: String (System.String) <72F012A4>

       Offset Bytes                                           Ascii
              00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
       ------ ----------------------------------------------- -----
0000000000000 66 6F 6F 3F 62 61 72                            foo?bar

UTF8 will encode in one to three bytes depending on how high the unicode number is (how many bits to encode). Three bytes in this case (U+2002), always starting with 'E' and then the first number, in this case '2'.

' ' | format-hex -Encoding utf8           
# "`u{2002}" | format-hex # powershell 7

Label: String (System.String) <03ACFE1C>

       Offset Bytes                                           Ascii
              00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
       ------ ----------------------------------------------- -----
0000000000000 E2 80 82                                        �
user_66549420

I found the answer to my question here: String Comparison, .NET and non breaking space

The "space" that was present in the string was the "non-breaking space" and getting rid of it in C# was as easy as:

string cellText = "String with non breaking spaces.";
cellText = Regex.Replace(cellText, @"\u00A0", " ");
user_26786570
Related