I have been attempting to modify an ML.NET tutorial on Image Classification and have been running into a schema mismatch problem where the model is expecting a VarVector and it is getting passed a Vector.
The original example creates an IDataView with a column named "ImagePath" filled with image paths and then generates an "Image" column of byte data using the MLContext.Transforms.LoadRawImageBytes(). The resulting IDataView is then fed to a Multiclassification.Trainers.ImageClassification() object. It is very straightforward.
I am looking to generalize it to involve a bit more image preprocessing. I would like for the preprocessing pipeline to reduce the image resolution and then also crop the image to the correct size prior to feeding it to the ImageClassification() trainer. It follows the same flow as the example but changes the processing pipeline to use the LoadImages(), ResizeImage(), and ExtractPixels() estimators in the image processing pipeline. Below is a snippet of code that I am working with.
// Fill an IDataView with the Label and ImagePath information:
// The ImageData class is defined for this purpose and it is just:
//
// public class ImageData {
// public string ImagePath { get; set; }
// public string Label { get; set; }
// }
//
// Create a bunch of image names and their labels into a list of ImageData objects.
IEnumerable<ImageData> images = LoadImagesFromDirectory(folder: assetsRelativePath,
useFolderNameAsLabel: true);
IDataView imageData = mlContext.Data.LoadFromEnumerable(images);
// Define the preprocessing pipeline to:
// 1. Map the Label to a unique key
// 2. Load the image data using the ImagePath column
// 3. Reduce the resolution of the image data
// 4. Crop the reduced resolution image data
// 5. Transform the image data from type "Image" to a byte array.
var preprocessingPipeline = mlContext.Transforms.Conversion.MapValueToKey(
inputColumnName: "Label",
outputColumnName: "LabelAsKey")
.Append(mlContext.Transforms.LoadImages(
inputColumnName: "ImagePath",
outputColumnName: "RawImage",
imageFolder: assetsRelativePath))
.Append(mlContext.Transforms.ResizeImages(
inputColumnName: "RawImage",
outputColumnName: "ResReduxImage",
imageWidth: 512,
imageHeight: 459,
resizing: Microsoft.ML.Transforms.Image.ImageResizingEstimator.ResizingKind.Fill))
.Append(mlContext.Transforms.ResizeImages(
inputColumnName: "ResReduxImage",
outputColumnName: "CroppedImage",
imageWidth: 512,
imageHeight: 459,
resizing: Microsoft.ML.Transforms.Image.ImageResizingEstimator.ResizingKind.IsoCrop,
cropAnchor: Microsoft.ML.Transforms.Image.ImageResizingEstimator.Anchor.Center))
.Append(mlContext.Transforms.ExtractPixels(
inputColumnName: "CroppedImage",
outputColumnName: "Image",
outputAsFloatArray: false,
colorsToExtract: ImagePixelExtractingEstimator.ColorBits.Red));
// Transform the raw ImageData into the TransformedData that the Model will train on
IDataView preProcessedData = preprocessingPipeline.Fit(imageData).Transform(imageData);
// Partition the data set for training and validation
TrainTestData trainSplit = mlContext.Data.TrainTestSplit(data: preProcessedData, testFraction: 0.4);
TrainTestData validationTestSplit = mlContext.Data.TrainTestSplit(trainSplit.TestSet);
IDataView trainSet = trainSplit.TrainSet;
IDataView validationSet = validationTestSplit.TrainSet;
IDataView testSet = validationTestSplit.TestSet;
var classifierOptions = new ImageClassificationTrainer.Options()
{
FeatureColumnName = "Image",
LabelColumnName = "LabelAsKey",
ValidationSet = validationSet,
Arch = ImageClassificationTrainer.Architecture.ResnetV2101,
MetricsCallback = (metrics) => Console.WriteLine(metrics),
TestOnTrainSet = false,
ReuseTrainSetBottleneckCachedValues = true,
ReuseValidationSetBottleneckCachedValues = true,
WorkspacePath=workspaceRelativePath
};
var trainingPipeline =
mlContext.MulticlassClassification.Trainers.ImageClassification(classifierOptions)
.Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel"));
// When I call Fit() in the next line of code, the following runtime error occurs:
// System.ArgumentOutOfRangeException: 'Schema mismatch for feature column 'Image':
// expected VarVector<Byte>, got Vector<Byte> Parameter name: inputSchema'
ITransformer trainedModel = trainingPipeline.Fit(trainSet);
I am stumped about the VarVector and Vector mismatch. I have tried the following over the past two days with no success:
- to define a custom DataViewSchema to pass to the transform
- to define a custom DataView by deriving from IDataView
- to define a custom mapping using a CustomMapping<> to change the type
Can anyone provide some guidance on how to get myself out of this jam? I would really like to avoid reading in an image, preprocessing it, writing it to file, then calling the original LoadRawImageBytes() function. I don't want that extra filesystem work.