Slice a list into batches based on byte size

user_480107 0
Viewed 366

Haven't found a similar question

I want to slice an array into smaller batches based on byte size limit. This is my implementation which is a little slow. In particular I want to know if there is a build-in functionality that can get me this, or maybe I can enhance my approach below.

When i accumulated the size of individual items it didn't correlate correctly to the size of the batch. probably extra metadata on the stream itself..

private static List<List<T>> SliceLogsIntoBatches<T>(List<T> data) where T : Log
{
    const long batchSizeLimitInBytes = 1048576;

    var batches = new List<List<T>>();

    while (data.Count > 0)
    {
        var batch = new List<T>();

        batch.AddRange(data.TakeWhile((log) =>
            {
                var currentBatchSizeInBytes = GetObjectSizeInBytes(batch); // this will slow down as takewhile moves on
                return (currentBatchSizeInBytes < batchSizeLimitInBytes);
            }));

        batches.Add(batch);
        data = data.Except(batch).ToList();
    }

    return batches;
}

private static long GetObjectSizeInBytes(object objectToGetSizeFor)
{
    using (var objectAsStream = ConvertObjectToMemoryStream(objectToGetSizeFor))
    {
        return objectAsStream.Length;
    }
}
list byte c#
2 Answers

You keep recalculating the size of the batch you are creating. So you are recalculating the size of some data items a lot. It would help if you would calculate the data size of each data item and simply add that to a variable to keep track of the current batch size.

Try something like this:

long batchSizeLimitInBytes = 1048576;
var batches = new List<List<T>>();
var currentBatch = new List<T>();
var currentBatchLength = 0;
for (int i = 0; i < data.Count; i++)
{
    var currentData = data[i];
    var currentDataLength = GetObjectSizeInBytes(currentData);
    if (currentBatchLength + currentDataLength > batchSizeLimitInBytes)
    {
        batches.Add(currentBatch);
        currentBatchLength = 0;
        currentBatch = new List<T>();
    }

    currentBatch.Add(currentData);
    currentBatchLength += currentDataLength;
}

As a sidenote, I would probably only want to convert the data to byte streams only once, since this is an expensive operation. You currently convert to streams just to check the length, you may want ot have this method actually return the streams batched, instead of List<List<T>>.

user_38838660

I think that your approach can be enhanced using the next idea: we can calculate an approximate size of the batch as sum of sizes of data objects; and then use this approximate batch size to form an actual batch; actual batch size is a size of list of data objects. If we use this idea we can reduce the number of invocations of the method GetObjectSizeInBytes.

Here is the code that implements this idea:

private static List<List<T>> SliceLogsIntoBatches<T>(List<T> data) where T : Log
{
    const long batchSizeLimitInBytes = 1048576;

    var batches = new List<List<T>>();
    var currentBatch = new List<T>();

    // At first, we calculate size of each data object.
    // We will use them to calculate an approximate size of the batch.
    List<long> sizes = data.Select(GetObjectSizeInBytes).ToList();

    int index = 0;
    // Approximate size of the batch.
    long dataSize = 0;

    while (index < data.Count)
    {
        dataSize += sizes[index];

        if (dataSize <= batchSizeLimitInBytes)
        {
            currentBatch.Add(data[index]);
            index++;
        }

        // If approximate size of the current batch is greater
        // than max batch size we try to form an actual batch by:
        // 1. calculating actual batch size via GetObjectSizeInBytes method;
        // and then
        // 2. excluding excess data objects if actual batch size is greater
        //    than max batch size.
        if (dataSize > batchSizeLimitInBytes || index >= data.Count)
        {
            // This loop excludes excess data objects if actual batch size
            // is greater than max batch size.
            while (GetObjectSizeInBytes(currentBatch) > batchSizeLimitInBytes)
            {
                index--;
                currentBatch.RemoveAt(currentBatch.Count - 1);
            }

            batches.Add(currentBatch);

            currentBatch = new List<T>();
            dataSize = 0;
        }
    }

    return batches;
}

Here is complete sample that demostrates this approach.

user_128332050
Related