How do I convert a string to a byte[] in .NET (C#) without manually specifying a specific encoding?
I'm going to encrypt the string. I can encrypt it without converting, but I'd still like to know why encoding comes to play here.
Also, why should encoding even be taken into consideration? Can't I simply get what bytes the string has been stored in? Why is there a dependency on character encodings?
It depends on the encoding of your string (ASCII, UTF-8, ...).
For example:
byte[] b1 = System.Text.Encoding.UTF8.GetBytes (myString);
byte[] b2 = System.Text.Encoding.ASCII.GetBytes (myString);
A small sample why encoding matters:
string pi = "\u03a0";
byte[] ascii = System.Text.Encoding.ASCII.GetBytes (pi);
byte[] utf8 = System.Text.Encoding.UTF8.GetBytes (pi);
Console.WriteLine (ascii.Length); //Will print 1
Console.WriteLine (utf8.Length); //Will print 2
Console.WriteLine (System.Text.Encoding.ASCII.GetString (ascii)); //Will print '?'
ASCII simply isn't equipped to deal with special characters.
Internally, the .NET framework uses UTF-16 to represent strings, so if you simply want to get the exact bytes that .NET uses, use System.Text.Encoding.Unicode.GetBytes (...).
See Character Encoding in the .NET Framework (MSDN) for more information.
BinaryFormatter bf = new BinaryFormatter();
byte[] bytes;
MemoryStream ms = new MemoryStream();
string orig = "喂 Hello 谢谢 Thank You";
bf.Serialize(ms, orig);
ms.Seek(0, 0);
bytes = ms.ToArray();
MessageBox.Show("Original bytes Length: " + bytes.Length.ToString());
MessageBox.Show("Original string Length: " + orig.Length.ToString());
for (int i = 0; i < bytes.Length; ++i) bytes[i] ^= 168; // pseudo encrypt
for (int i = 0; i < bytes.Length; ++i) bytes[i] ^= 168; // pseudo decrypt
BinaryFormatter bfx = new BinaryFormatter();
MemoryStream msx = new MemoryStream();
msx.Write(bytes, 0, bytes.Length);
msx.Seek(0, 0);
string sx = (string)bfx.Deserialize(msx);
MessageBox.Show("Still intact :" + sx);
MessageBox.Show("Deserialize string Length(still intact): "
+ sx.Length.ToString());
BinaryFormatter bfy = new BinaryFormatter();
MemoryStream msy = new MemoryStream();
bfy.Serialize(msy, sx);
msy.Seek(0, 0);
byte[] bytesy = msy.ToArray();
MessageBox.Show("Deserialize bytes Length(still intact): "
+ bytesy.Length.ToString());
This is a popular question. It is important to understand what the question author is asking, and that it is different from what is likely the most common need. To discourage misuse of the code where it is not needed, I've answered the later first.
Every string has a character set and encoding. When you convert a System.String object to an array of System.Byte you still have a character set and encoding. For most usages, you'd know which character set and encoding you need and .NET makes it simple to "copy with conversion." Just choose the appropriate Encoding class.
// using System.Text;
Encoding.UTF8.GetBytes(".NET String to byte array")
The conversion may need to handle cases where the target character set or encoding doesn't support a character that's in the source. You have some choices: exception, substitution or skipping. The default policy is to substitute a '?'.
// using System.Text;
var text = Encoding.ASCII.GetString(Encoding.ASCII.GetBytes("You win €100"));
// -> "You win ?100"
Clearly, conversions are not necessarily lossless!
Note: For System.String the source character set is Unicode.
The only confusing thing is that .NET uses the name of a character set for the name of one particular encoding of that character set. Encoding.Unicode should be called Encoding.UTF16.
That's it for most usages. If that's what you need, stop reading here. See the fun Joel Spolsky article if you don't understand what an encoding is.
Now, the question author asks, "Every string is stored as an array of bytes, right? Why can't I simply have those bytes?"
He doesn't want any conversion.
From the C# spec:
Character and string processing in C# uses Unicode encoding. The char type represents a UTF-16 code unit, and the string type represents a sequence of UTF-16 code units.
So, we know that if we ask for the null conversion (i.e., from UTF-16 to UTF-16), we'll get the desired result:
Encoding.Unicode.GetBytes(".NET String to byte array")
But to avoid the mention of encodings, we must do it another way. If an intermediate data type is acceptable, there is a conceptual shortcut for this:
".NET String to byte array".ToCharArray()
That doesn't get us the desired datatype but Mehrdad's answer shows how to convert this Char array to a Byte array using BlockCopy. However, this copies the string twice! And, it too explicitly uses encoding-specific code: the datatype System.Char.
The only way to get to the actual bytes the String is stored in is to use a pointer. The fixed statement allows taking the address of values. From the C# spec:
[For] an expression of type string, ... the initializer computes the address of the first character in the string.
To do so, the compiler writes code skip over the other parts of the string object with RuntimeHelpers.OffsetToStringData. So, to get the raw bytes, just create a pointer to the string and copy the number of bytes needed.
// using System.Runtime.InteropServices
unsafe byte[] GetRawBytes(String s)
{
if (s == null) return null;
var codeunitCount = s.Length;
/* We know that String is a sequence of UTF-16 codeunits
and such codeunits are 2 bytes */
var byteCount = codeunitCount * 2;
var bytes = new byte[byteCount];
fixed(void* pRaw = s)
{
Marshal.Copy((IntPtr)pRaw, bytes, 0, byteCount);
}
return bytes;
}
As @CodesInChaos pointed out, the result depends on the endianness of the machine. But the question author is not concerned with that.
You need to take the encoding into account, because 1 character could be represented by 1 or more bytes (up to about 6), and different encodings will treat these bytes differently.
Joel has a posting on this:
The first part of your question (how to get the bytes) was already answered by others: look in the System.Text.Encoding namespace.
I will address your follow-up question: why do you need to pick an encoding? Why can't you get that from the string class itself?
The answer is in two parts.
First of all, the bytes used internally by the string class don't matter, and whenever you assume they do you're likely introducing a bug.
If your program is entirely within the .Net world then you don't need to worry about getting byte arrays for strings at all, even if you're sending data across a network. Instead, use .Net Serialization to worry about transmitting the data. You don't worry about the actual bytes any more: the Serialization formatter does it for you.
On the other hand, what if you are sending these bytes somewhere that you can't guarantee will pull in data from a .Net serialized stream? In this case you definitely do need to worry about encoding, because obviously this external system cares. So again, the internal bytes used by the string don't matter: you need to pick an encoding so you can be explicit about this encoding on the receiving end, even if it's the same encoding used internally by .Net.
I understand that in this case you might prefer to use the actual bytes stored by the string variable in memory where possible, with the idea that it might save some work creating your byte stream. However, I put it to you it's just not important compared to making sure that your output is understood at the other end, and to guarantee that you must be explicit with your encoding. Additionally, if you really want to match your internal bytes, you can already just choose the Unicode encoding, and get that performance savings.
Which brings me to the second part... picking the Unicode encoding is telling .Net to use the underlying bytes. You do need to pick this encoding, because when some new-fangled Unicode-Plus comes out the .Net runtime needs to be free to use this newer, better encoding model without breaking your program. But, for the moment (and forseeable future), just choosing the Unicode encoding gives you what you want.
It's also important to understand your string has to be re-written to wire, and that involves at least some translation of the bit-pattern even when you use a matching encoding. The computer needs to account for things like Big vs Little Endian, network byte order, packetization, session information, etc.
Also please explain why encoding should be taken into consideration. Can't I simply get what bytes the string has been stored in? Why this dependency on encoding?!!!
Because there is no such thing as "the bytes of the string".
A string (or more generically, a text) is composed of characters: letters, digits, and other symbols. That's all. Computers, however, do not know anything about characters; they can only handle bytes. Therefore, if you want to store or transmit text by using a computer, you need to transform the characters to bytes. How do you do that? Here's where encodings come to the scene.
An encoding is nothing but a convention to translate logical characters to physical bytes. The simplest and best known encoding is ASCII, and it is all you need if you write in English. For other languages you will need more complete encodings, being any of the Unicode flavours the safest choice nowadays.
So, in short, trying to "get the bytes of a string without using encodings" is as impossible as "writing a text without using any language".
By the way, I strongly recommend you (and anyone, for that matter) to read this small piece of wisdom: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
byte[] strToByteArray(string str)
{
System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
return enc.GetBytes(str);
}
With the advent of Span<T> released with C# 7.2, the canonical technique to capture the underlying memory representation of a string into a managed byte array is:
byte[] bytes = "rubbish_\u9999_string".AsSpan().AsBytes().ToArray();
Converting it back should be a non-starter because that means you are in fact interpreting the data somehow, but for the sake of completeness:
string s;
unsafe
{
fixed (char* f = &bytes.AsSpan().NonPortableCast<byte, char>().DangerousGetPinnableReference())
{
s = new string(f);
}
}
The names NonPortableCast and DangerousGetPinnableReference should further the argument that you probably shouldn't be doing this.
Note that working with Span<T> requires installing the System.Memory NuGet package.
Regardless, the actual original question and follow-up comments imply that the underlying memory is not being "interpreted" (which I assume means is not modified or read beyond the need to write it as-is), indicating that some implementation of the Stream class should be used instead of reasoning about the data as strings at all.
I'm not sure, but I think the string stores its info as an array of Chars, which is inefficient with bytes. Specifically, the definition of a Char is "Represents a Unicode character".
take this example sample:
String str = "asdf éß";
String str2 = "asdf gh";
EncodingInfo[] info = Encoding.GetEncodings();
foreach (EncodingInfo enc in info)
{
System.Console.WriteLine(enc.Name + " - "
+ enc.GetEncoding().GetByteCount(str)
+ enc.GetEncoding().GetByteCount(str2));
}
Take note that the Unicode answer is 14 bytes in both instances, whereas the UTF-8 answer is only 9 bytes for the first, and only 7 for the second.
So if you just want the bytes used by the string, simply use Encoding.Unicode, but it will be inefficient with storage space.
The key issue is that a glyph in a string takes 32 bits (16 bits for a character code) but a byte only has 8 bits to spare. A one-to-one mapping doesn't exist unless you restrict yourself to strings that only contain ASCII characters. System.Text.Encoding has lots of ways to map a string to byte[], you need to pick one that avoids loss of information and that is easy to use by your client when she needs to map the byte[] back to a string.
Utf8 is a popular encoding, it is compact and not lossy.
Fastest way
public static byte[] GetBytes(string text)
{
return System.Text.ASCIIEncoding.UTF8.GetBytes(text);
}
EDIT as Makotosan commented this is now the best way:
Encoding.UTF8.GetBytes(text)
How do I convert a string to a byte[] in .NET (C#) without manually specifying a specific encoding?
A string in .NET represents text as a sequence of UTF-16 code units, so the bytes are encoded in memory in UTF-16 already.
Mehrdad's Answer
You can use Mehrdad's answer, but it does actually use an encoding because chars are UTF-16. It calls ToCharArray which looking at the source creates a char[] and copies the memory to it directly. Then it copies the data to a byte array that is also allocated. So under the hood it is copying the underlying bytes twice and allocating a char array that is not used after the call.
Tom Blodget's Answer
Tom Blodget's answer is 20-30% faster than Mehrdad since it skips the intermediate step of allocating a char array and copying the bytes to it, but it requires you compile with the /unsafe option. If you absolutely do not want to use encoding, I think this is the way to go. If you put your encryption login inside the fixed block, you don't even need to allocate a separate byte array and copy the bytes to it.
Also, why should encoding be taken into consideration? Can't I simply get what bytes the string has been stored in? Why is there a dependency on character encodings?
Because that is the proper way to do it. string is an abstraction.
Using an encoding could give you trouble if you have 'strings' with invalid characters, but that shouldn't happen. If you are getting data into your string with invalid characters you are doing it wrong. You should probably be using a byte array or a Base64 encoding to start with.
If you use System.Text.Encoding.Unicode, your code will be more resilient. You don't have to worry about the endianness of the system your code will be running on. You don't need to worry if the next version of the CLR will use a different internal character encoding.
I think the question isn't why you want to worry about the encoding, but why you want to ignore it and use something else. Encoding is meant to represent the abstraction of a string in a sequence of bytes. System.Text.Encoding.Unicode will give you a little endian byte order encoding and will perform the same on every system, now and in the future.
Upon being asked what you intend to do with the bytes, you responded:
I'm going to encrypt it. I can encrypt it without converting but I'd still like to know why encoding comes to play here. Just give me the bytes is what I say.
Regardless of whether you intend to send this encrypted data over the network, load it back into memory later, or stream it to another process, you are clearly intending to decrypt it at some point. In that case, the answer is that you're defining a communication protocol. A communication protocol should not be defined in terms of implementation details of your programming language and its associated runtime. There are several reasons for this:
For communicating (either with a completely disparate process or with the same program in the future), you need to define your protocol strictly to minimize the difficulty of working with it or accidentally creating bugs. Depending on .NET's internal representation is not a strict, clear, or even guaranteed to be consistent definition. A standard encoding is a strict definition that will not fail you in the future.
In other words, you can't satisfy your requirement for consistency without specifying an encoding.
You may certainly choose to use UTF-16 directly if you find that your process performs significantly better since .NET uses it internally or for any other reason, but you need to choose that encoding explicitly and perform those conversions explicitly in your code rather than depending on .NET's internal implementation.
So choose an encoding and use it:
using System.Text;
// ...
Encoding.Unicode.GetBytes("abc"); # UTF-16 little endian
Encoding.UTF8.GetBytes("abc")
As you can see, it's also actually less code to just use the built in encoding objects than to implement your own reader/writer methods.
This is because, as Tyler so aptly said, "Strings aren't pure data. They also have information." In this case, the information is an encoding that was assumed when the string was created.
This is based off of OP's comment on his own question, and is the correct question if I understand OP's hints at the use-case.
Storing binary data in strings is probably the wrong approach because of the assumed encoding mentioned above! Whatever program or library stored that binary data in a string (instead of a byte[] array which would have been more appropriate) has already lost the battle before it has begun. If they are sending the bytes to you in a REST request/response or anything that must transmit strings, Base64 would be the right approach.
Everybody else answered this incorrect question incorrectly.
If the string looks good as-is, just pick an encoding (preferably one starting with UTF), use the corresponding System.Text.Encoding.???.GetBytes() function, and tell whoever you give the bytes to which encoding you picked.
If you are using .NET Core or System.Memory for .NET Framework, there is a very efficient marshaling mechanism available via Span<T> and Memory<T> that can effectively reinterpret string memory as a span of bytes. Once you have a span of bytes you are free to marshal back to another type, or copy the span to an array for serialization.
To summarize what others have said:
public static class MarshalExtensions
{
public static ReadOnlySpan<byte> AsBytes(this string value) => MemoryMarshal.AsBytes(value.AsSpan());
public static string AsString(this ReadOnlySpan<byte> value) => new string(MemoryMarshal.Cast<byte, char>(value));
}
static void Main(string[] args)
{
string str1 = "你好，世界";
ReadOnlySpan<byte> span = str1.AsBytes();
string str2 = span.AsString();
byte[] bytes = span.ToArray();
Debug.Assert(bytes.Length > 0);
Debug.Assert(str1 == str2);
}
In C++ this is roughly equivalent to reinterpret_cast, and C this is roughly equivalent to a cast to the system's word type (char).
In recent versions of the .NET Core Runtime (CoreCLR), operations on spans effectively invoke compiler intrinsics and various optimizations that can sometimes eliminate bounds checking, leading to exceptional performance while preserving memory safety, assuming that your memory was allocated by the CLR and the spans are not derived from pointers from an unmanaged memory allocator.
This uses a mechanism supported by the CLR that returns ReadOnlySpan<char> from a string; Additionally, this span does not necessarily encompass the complete internal string layout. ReadOnlySpan<T> implies that you must create a copy if you need to perform mutation, as strings are immutable.
Two ways:
public static byte[] StrToByteArray(this string s)
{
List<byte> value = new List<byte>();
foreach (char c in s.ToCharArray())
value.Add(c.ToByte());
return value.ToArray();
}
And,
public static byte[] StrToByteArray(this string s)
{
s = s.Replace(" ", string.Empty);
byte[] buffer = new byte[s.Length / 2];
for (int i = 0; i < s.Length; i += 2)
buffer[i / 2] = (byte)Convert.ToByte(s.Substring(i, 2), 16);
return buffer;
}
I tend to use the bottom one more often than the top, haven't benchmarked them for speed.
// C# to convert a string to a byte array.
public static byte[] StrToByteArray(string str)
{
System.Text.ASCIIEncoding encoding=new System.Text.ASCIIEncoding();
return encoding.GetBytes(str);
}
// C# to convert a byte array to a string.
byte [] dBytes = ...
string str;
System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
str = enc.GetString(dBytes);