How to carry number from decimal to whole byte array

Note: This is more of a logic/math problem than a specific C# problem.

I have my own class called Number - it very simply contains two separate byte arrays called Whole and Decimal. These byte arrays each represent essentially an infinitely large whole number, but, when put together the idea is that they create a whole number with a decimal part.

The bytes are stored in a little-endian format, representing a number. I'm creating a method called AddNumbers which will add two of these Numbers together.

This method relies on another method called PerformAdd, which just adds two arrays together. It simply takes in a pointer to the final byte array, a pointer to one array to add, and a pointer to the second array to add - as well as the length of each of them. The two arrays are just named "larger" and "smaller". Here is the code for this method:

    private static unsafe void PerformAdd(byte* finalPointer, byte* largerPointer, byte* smallerPointer, int largerLength, int smallerLength)
    {
        int carry = 0;

        // Go through all the items that can be added, and work them out.
        for (int i = 0; i < smallerLength; i++)
        {
            var add = *largerPointer-- + *smallerPointer-- + carry;

            // Stick the result of this addition in the "final" array.
            *finalPointer-- = (byte)(add & 0xFF);

            // Now, set a carry from this.
            carry = add >> 8;
        }

        // Now, go through all the remaining items (which don't need to be added), and add them to the "final" - still working with the carry.
        for (int i = smallerLength; i < largerLength; i++)
        {
            var wcarry = *largerPointer-- + carry;

            // Stick the result of this addition in the "final" array.
            *finalPointer-- = (byte)(wcarry & 0xFF);

            // Now, set a carry from this.
            carry = wcarry >> 8;
        }

        // Now, if we have anything still left to carry, carry it into a new byte.
        if (carry > 0)
            *finalPointer-- = (byte)carry;
    }

This method isn't where the problem lies - the problem is with how I use it. It's the AddNumbers method that uses it. The way it works is fine - it organizes the two separate byte arrays into the "larger" (larger meaning having a higher length of bytes) and "smaller". And then it creates pointers, it does this both for Whole and Decimal separately. The problem is with the decimal part.

Let's say we're adding the numbers 1251 and 2185 together, in this situation you would get 3436 - so that works perfectly!

Take another example as well: You have the numbers 4.6 and add 1.2 - once again, this works fine, and you get 5.8. The problem comes with the next example.

We have 15.673 and 1.783, you would expect 17.456, however, actually, this returns: 16.1456, and the reason for that is because it doesn't carry the "1".

So, this is my problem: How would I implement a way that knows when and how to do this? Here's the code for my AddNumbers method:

    public static unsafe Number AddNumbers(Number num1, Number num2)
    {
        // Store the final result.
        Number final = new Number(new byte[num1.Whole.Length + num2.Whole.Length], new byte[num1.Decimal.Length + num2.Decimal.Length]);

        // We're going to figure out which number (num1 or num2) has more bytes, and then we'll create pointers to smallest and largest.
        fixed (byte* num1FixedWholePointer = num1.Whole, num1FixedDecPointer = num1.Decimal, num2FixedWholePointer = num2.Whole, num2FixedDecPointer = num2.Decimal,
            finalFixedWholePointer = final.Whole, finalFixedDecimalPointer = final.Decimal)
        {
            // Create a pointer and figure out which whole number has the most bytes.
            var finalWholePointer = finalFixedWholePointer + (final.Whole.Length - 1);
            var num1WholeLarger = num1.Whole.Length > num2.Whole.Length ? true : false;

            // Store the larger/smaller whole number lengths.
            var largerLength = num1WholeLarger ? num1.Whole.Length : num2.Whole.Length;
            var smallerLength = num1WholeLarger ? num2.Whole.Length : num1.Whole.Length;

            // Create pointers to the whole numbers (the largest amount of bytes and smallest amount of bytes).
            var largerWholePointer = num1WholeLarger ? num1FixedWholePointer + (num1.Whole.Length - 1) : num2FixedWholePointer + (num2.Whole.Length - 1);
            var smallerWholePointer = num1WholeLarger ? num2FixedWholePointer + (num2.Whole.Length - 1) : num1FixedWholePointer + (num1.Whole.Length - 1);

            // Handle decimal numbers.
            if (num1.Decimal.Length > 0 || num2.Decimal.Length > 0)
            {
                // Create a pointer and figure out which decimal has the most bytes.
                var finalDecPointer = finalFixedDecimalPointer + (final.Decimal.Length - 1);
                var num1DecLarger = num1.Decimal.Length > num2.Decimal.Length ? true : false;

                // Store the larger/smaller whole number lengths.
                var largerDecLength = num1DecLarger ? num1.Decimal.Length : num2.Decimal.Length;
                var smallerDecLength = num1DecLarger ? num2.Whole.Length : num1.Decimal.Length;

                // Store pointers for decimals as well.
                var largerDecPointer = num1DecLarger ? num1FixedDecPointer + (num1.Decimal.Length - 1) : num2FixedDecPointer + (num2.Decimal.Length - 1);
                var smallerDecPointer = num1DecLarger ? num2FixedDecPointer + (num2.Decimal.Length - 1) : num1FixedDecPointer + (num1.Decimal.Length - 1);

                // Add the decimals first.
                PerformAdd(finalDecPointer, largerDecPointer, smallerDecPointer, largerDecLength, smallerDecLength);
            }

            // Add the whole number now.
            PerformAdd(finalWholePointer, largerWholePointer, smallerWholePointer, largerLength, smallerLength);
        }

        return final;
    }