Calling .ToArray on an Enumerable corrupts the Enumerable

user_5507624 0
Viewed 75

I'm not sure if this is specific to ML.NET, but it does happen in the context of it.

I am using ML.NET to classify some images. I realized that it poses a severe difference whether I call .ToArray() on the resulting IEnumerable or not. The former results in all the array elements becoming identical to the last one.

IEnumerable<ImageData> dataCollection = imagePaths.Select(path => new ImageData(path));
IDataView targetDataView = _mlContext.Data.LoadFromEnumerable(dataCollection);
IDataView predictionView = _transformerModel.Transform(targetDataView); 
return _mlContext.Data.CreateEnumerable<ImagePrediction>(predictionView, true).ToArray();

In the example shown above, the resulting predictions will all have their image path set to the last image path in imagePaths.

I don't believe that this is intended behaviour. What causes this and how can I safely prevent this? For the moment I decided to just not call .ToArray(), but I'd like to know more about this issue.

ml.net linq c#
1 Answers

The issue seems to be in the prediction engine where to limit the memory usage, row is reused as per reuseRowObject. Therefore when a ToList() or ToArray() method is invoked, only the last item is used to project the list/array.

public IEnumerable<TDst> RunPipe(bool reuseRowObject)
{
    var curCounter = _counter;
    using (var cursor = _cursorablePipe.GetCursor())
    {
        TDst row = null;
        while (cursor.MoveNext())
        {
            if (!reuseRowObject || row == null)
                row = new TDst();

            cursor.FillValues(row);
            yield return row;
            if (curCounter != _counter)
                throw Contracts.Except("An attempt was made to keep iterating after the pipe has been reset.");
        }
    }
}

The caller is CreateEnumerable() where you explicitly set reuseRowObject to true.

public IEnumerable<TRow> CreateEnumerable<TRow>(IDataView data, bool reuseRowObject,
    bool ignoreMissingColumns = false, SchemaDefinition schemaDefinition = null)
    where TRow : class, new()
{
    _env.CheckValue(data, nameof(data));
    _env.CheckValueOrNull(schemaDefinition);

    var engine = new PipeEngine<TRow>(_env, data, ignoreMissingColumns, schemaDefinition);
    return engine.RunPipe(reuseRowObject);
}

Setting reuseRowObject to false should solve your issue.

user_91393840
Related