Copy decimal into byte array without allocations

user_1582922 0
Viewed 849

The Dream

I'm looking for a way to copy a decimal value into a byte array buffer and then be able to read those bytes back into a decimal without any heap allocations. Ideally this wouldn't require an unsafe context.

My Attempts

I've used makeshift unions in C# to do some crazy things before. It's a pretty cool way to just read memory any random way you want, but you've gotta be careful. You can get into corrupted states where a variable my explicitly be, for example byte[] but the value as viewed in the debugger is int[]. I didn't even know such a thing was possible!

NOTE: Marc makes a really important point in the comments below. You can't reliably convert numbers directly into bytes using an overlapping struct concept like this due to endianness. In this case you can safely use ints because the decimal type used 4 ints internally. Here's an [example] from protobuf-net's decimal serializer.

1: struct union w/ decimal and byte[]

This first attempt tries to use the struct union concept w/ a decimal and byte[] field, both at the 0 offset, so they're taking up the exact same memory location. I could then write to one field and read from the other field.

[StructLayout(LayoutKind.Explicit)]
private readonly struct convert_decimal_to_bytes
{
    [FieldOffset(0)]
    public readonly decimal value;
    [FieldOffset(0)]
    public readonly byte[] bytes;

    public convert_decimal_to_bytes(decimal value)
    {
        bytes = default;
        this.value = value;
    }
}

This doesn't even run w/out throwing -- type fails to load with the following:

System.TypeLoadException : Could not load type 'convert_decimal_to_bytes' from assembly 'teloneum, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' because it contains an object field at offset 0 that is incorrectly aligned or overlapped by a non-object field.

Apparently the CLR doesn't like them being overlapped -- but only because one is a value type and the other is a reference type.

2: struct union w/ decimal and 16 bytes!

So I decided to remove the reference type! Now, the following works, or at least, it will work, but the ordering is all messed up. I can certainly can a big decimal number, run it through here, and then just make by field offsets such that it matches my test, but that just seems wrong -- plus, there's at least three zero bytes in every decimal I tried this for :) Maybe I just got lazy at the end... it is late after all.

[StructLayout(LayoutKind.Explicit)]
private readonly struct convert_decimal_to_bytes
{
    [FieldOffset(0)] public readonly decimal value;

    [FieldOffset( 0)] public readonly byte byte_1;
    [FieldOffset( 1)] public readonly byte byte_2;
    [FieldOffset( 2)] public readonly byte byte_3;
    [FieldOffset( 3)] public readonly byte byte_4;
    [FieldOffset( 4)] public readonly byte byte_5;
    [FieldOffset( 5)] public readonly byte byte_6;
    [FieldOffset( 6)] public readonly byte byte_7;
    [FieldOffset( 7)] public readonly byte byte_8;
    [FieldOffset( 8)] public readonly byte byte_9;
    [FieldOffset( 9)] public readonly byte byte_10;
    [FieldOffset(10)] public readonly byte byte_11;
    [FieldOffset(11)] public readonly byte byte_12;
    [FieldOffset(12)] public readonly byte byte_13;
    [FieldOffset(13)] public readonly byte byte_14;
    [FieldOffset(14)] public readonly byte byte_15;
    [FieldOffset(15)] public readonly byte byte_16;

    public convert_decimal_to_bytes(decimal value)
    {
        byte_1  = default;
        byte_2  = default;
        byte_3  = default;
        byte_4  = default;
        byte_5  = default;
        byte_6  = default;
        byte_7  = default;
        byte_8  = default;
        byte_9  = default;
        byte_10 = default;
        byte_11 = default;
        byte_12 = default;
        byte_13 = default;
        byte_14 = default;
        byte_15 = default;
        byte_16 = default;
        this.value = value;
    }

    public convert_decimal_to_bytes(int startIndex, byte[] buffer)
    {
        value = default;
        byte_1 = buffer[startIndex++];
        byte_2 = buffer[startIndex++];
        byte_3 = buffer[startIndex++];
        byte_4 = buffer[startIndex++];
        byte_5 = buffer[startIndex++];
        byte_6 = buffer[startIndex++];
        byte_7 = buffer[startIndex++];
        byte_8 = buffer[startIndex++];
        byte_9 = buffer[startIndex++];
        byte_10 = buffer[startIndex++];
        byte_11 = buffer[startIndex++];
        byte_12 = buffer[startIndex++];
        byte_13 = buffer[startIndex++];
        byte_14 = buffer[startIndex++];
        byte_15 = buffer[startIndex++];
        byte_16 = buffer[startIndex];
    }


    public static void Copy(decimal value, int startIndex, byte[] buffer)
    {
        var convert = new convert_decimal_to_bytes(value);

        buffer[startIndex++] = convert.byte_1;
        buffer[startIndex++] = convert.byte_2;
        buffer[startIndex++] = convert.byte_3;
        buffer[startIndex++] = convert.byte_4;
        buffer[startIndex++] = convert.byte_5;
        buffer[startIndex++] = convert.byte_6;
        buffer[startIndex++] = convert.byte_7;
        buffer[startIndex++] = convert.byte_8;
        buffer[startIndex++] = convert.byte_9;
        buffer[startIndex++] = convert.byte_10;
        buffer[startIndex++] = convert.byte_11;
        buffer[startIndex++] = convert.byte_12;
        buffer[startIndex++] = convert.byte_13;
        buffer[startIndex++] = convert.byte_14;
        buffer[startIndex++] = convert.byte_15;
        buffer[startIndex]   = convert.byte_16;
    }

    public static decimal Read(int startIndex, byte[] buffer)
    {
        var convert = new convert_decimal_to_bytes(startIndex, buffer);

        return convert.value;
    }
}
c#
2 Answers

Before .NET 5.0, this was awkward without some ugly hackery. From .NET 5.0, there are more methods accepting spans.

You can use the GetBits(decimal d, Span<int>) method using a stack-allocated span, and then convert the four integers into the existing byte array however you want, e.g. with BitConverter.TryWriteBytes.

In the other direction, there's a Decimal(ReadOnlySpan<int>) constructor, so again you can stackalloc a Span<int>, use BitConverter.ToInt32(ReadOnlySpan<byte>) repeatedly to populate that span from the byte array, and pass it to the constructor.

As an aside, instead of accepting byte arrays and start indexes, you might want to embrace spans more widely through your code base.

Here's some sample code which does all of the above - it may well be possible to do it slightly more efficiently, but hopefully this gets the idea across, and this does avoid allocation:

using System;

class Program
{
    public static void Copy(decimal value, int startIndex, byte[] buffer)
    {
        Span<int> int32s = stackalloc int[4];
        decimal.GetBits(value, int32s);

        var bufferSpan = buffer.AsSpan();
        for (int i = 0; i < 4; i++)
        {
            // These slices are bigger than we need, but this is the simplest approach.
            var slice = bufferSpan.Slice(startIndex + i * 4);
            if (!BitConverter.TryWriteBytes(slice, int32s[i]))
            {
                throw new ArgumentException("Not enough space in span");
            }
        }
    }

    public static decimal Read(int startIndex, byte[] buffer)
    {
        Span<int> int32s = stackalloc int[4];
        ReadOnlySpan<byte> bufferSpan = buffer.AsSpan();
        for (int i = 0; i < 4; i++)
        {
            var slice = bufferSpan.Slice(startIndex + i * 4);
            int32s[i] = BitConverter.ToInt32(slice);
        }
        return new decimal(int32s);
    }

    static void Main()
    {
        byte[] bytes = new byte[16];
        decimal original = 1234.567m;
        Copy(original, 0, bytes);
        decimal restored = Read(0, bytes);
        Console.WriteLine(restored);
    }
}

or the same thing using MemoryMarshal:

public static void Copy(decimal value, int startIndex, byte[] buffer)
    => decimal.GetBits(value, MemoryMarshal.Cast<byte, int>(buffer.AsSpan(startIndex)));

public static decimal Read(int startIndex, byte[] buffer)
    => new decimal(MemoryMarshal.Cast<byte, int>(buffer.AsSpan(startIndex)));
user_226560

In fact, consulting the MSDN example code, I found a simple way to do exactly this using this way:

        int[] data = decimal.GetBits(h);
        bool sign = (data[3] & 0x80000000) != 0;
        byte scale = (byte)(((data[3] >> 16) & 0x7F) + (byte)(!sign ? 0x20 : 0x00));
        List<byte> bytes = new List<byte>() // For concatenation
        bytes.AddRange(BitConverter.GetBytes(data[0]));
        bytes.AddRange(BitConverter.GetBytes(data[1]));
        bytes.AddRange(BitConverter.GetBytes(data[2]));
        bytes.Add(scale);
        return bytes.ToArray();

And back:

        byte[] bytes = array.Take(13);
        byte scale = bytes[12];
        bool sign = true;
        if (scale >= 0x20)
        {
            sign = false;
            scale -= 0x20;
        }

        return new decimal(BitConverter.ToInt32(bytes,0), 
                           BitConverter.ToInt32(bytes,4), 
                           BitConverter.ToInt32(bytes,8), sign, scale);

I am using this code above to serialize/deserialize Decimals. Since MSDN documentation points that are bits that are unused in the last int (the first 16 ones and the last 3 ones, I convert the last int to an "signed scale" so I have less bytes to allocate. The original decimal size is 16-bytes long.

user_165243340
Related