ML.NET Schema mismatch: expected VarVector<Byte> got Vector<Byte>

Viewed 869

I have been attempting to modify an ML.NET tutorial on Image Classification and have been running into a schema mismatch problem where the model is expecting a VarVector and it is getting passed a Vector.

The original example creates an IDataView with a column named "ImagePath" filled with image paths and then generates an "Image" column of byte data using the MLContext.Transforms.LoadRawImageBytes(). The resulting IDataView is then fed to a Multiclassification.Trainers.ImageClassification() object. It is very straightforward.

I am looking to generalize it to involve a bit more image preprocessing. I would like for the preprocessing pipeline to reduce the image resolution and then also crop the image to the correct size prior to feeding it to the ImageClassification() trainer. It follows the same flow as the example but changes the processing pipeline to use the LoadImages(), ResizeImage(), and ExtractPixels() estimators in the image processing pipeline. Below is a snippet of code that I am working with.

    // Fill an IDataView with the Label and ImagePath information:
    // The ImageData class is defined for this purpose and it is just:
    // 
    //  public class ImageData {
    //       public string ImagePath { get; set; }
    //       public string Label { get; set; }
    //  }
    // 
    // Create a bunch of image names and their labels into a list of ImageData objects.
    IEnumerable<ImageData> images = LoadImagesFromDirectory(folder: assetsRelativePath, 
                                                            useFolderNameAsLabel: true);
    IDataView imageData = mlContext.Data.LoadFromEnumerable(images);

    // Define the preprocessing pipeline to:
    // 1. Map the Label to a unique key
    // 2. Load the image data using the ImagePath column
    // 3. Reduce the resolution of the image data
    // 4. Crop the reduced resolution image data
    // 5. Transform the image data from type "Image" to a byte array.
    var preprocessingPipeline = mlContext.Transforms.Conversion.MapValueToKey(
                    inputColumnName: "Label",
                    outputColumnName: "LabelAsKey")
                .Append(mlContext.Transforms.LoadImages(
                    inputColumnName: "ImagePath",
                    outputColumnName: "RawImage",
                    imageFolder: assetsRelativePath))
                .Append(mlContext.Transforms.ResizeImages(
                    inputColumnName: "RawImage",
                    outputColumnName: "ResReduxImage",
                    imageWidth: 512,
                    imageHeight: 459,
                    resizing: Microsoft.ML.Transforms.Image.ImageResizingEstimator.ResizingKind.Fill))
                .Append(mlContext.Transforms.ResizeImages(
                    inputColumnName: "ResReduxImage",
                    outputColumnName: "CroppedImage",
                    imageWidth: 512,
                    imageHeight: 459,
                    resizing: Microsoft.ML.Transforms.Image.ImageResizingEstimator.ResizingKind.IsoCrop,
                    cropAnchor: Microsoft.ML.Transforms.Image.ImageResizingEstimator.Anchor.Center))
                .Append(mlContext.Transforms.ExtractPixels(
                    inputColumnName: "CroppedImage",
                    outputColumnName: "Image",
                    outputAsFloatArray: false,
                    colorsToExtract: ImagePixelExtractingEstimator.ColorBits.Red));

     // Transform the raw ImageData into the TransformedData that the Model will train on
     IDataView preProcessedData = preprocessingPipeline.Fit(imageData).Transform(imageData);

     // Partition the data set for training and validation
     TrainTestData trainSplit = mlContext.Data.TrainTestSplit(data: preProcessedData, testFraction: 0.4);
     TrainTestData validationTestSplit = mlContext.Data.TrainTestSplit(trainSplit.TestSet);

     IDataView trainSet = trainSplit.TrainSet;
     IDataView validationSet = validationTestSplit.TrainSet;
     IDataView testSet = validationTestSplit.TestSet;

     var classifierOptions = new ImageClassificationTrainer.Options()
     {
          FeatureColumnName = "Image",
          LabelColumnName = "LabelAsKey",
          ValidationSet = validationSet,
          Arch = ImageClassificationTrainer.Architecture.ResnetV2101,
          MetricsCallback = (metrics) => Console.WriteLine(metrics),
          TestOnTrainSet = false,
          ReuseTrainSetBottleneckCachedValues = true,
          ReuseValidationSetBottleneckCachedValues = true,
          WorkspacePath=workspaceRelativePath
     };

     var trainingPipeline = 
           mlContext.MulticlassClassification.Trainers.ImageClassification(classifierOptions)
           .Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel"));

     // When I call Fit() in the next line of code, the following runtime error occurs: 
     // System.ArgumentOutOfRangeException: 'Schema mismatch for feature column 'Image': 
     // expected VarVector<Byte>, got Vector<Byte> Parameter name: inputSchema'
     ITransformer trainedModel = trainingPipeline.Fit(trainSet);

I am stumped about the VarVector and Vector mismatch. I have tried the following over the past two days with no success:

  • to define a custom DataViewSchema to pass to the transform
  • to define a custom DataView by deriving from IDataView
  • to define a custom mapping using a CustomMapping<> to change the type

Can anyone provide some guidance on how to get myself out of this jam? I would really like to avoid reading in an image, preprocessing it, writing it to file, then calling the original LoadRawImageBytes() function. I don't want that extra filesystem work.

1 Answers

I figured out how to make this work. The issue here was that the underlying types were correct but the schema was wrong. I was able to solve my issue with a CustomMapping and some attribute magic.See CustomMapping documentation for how the CustomMapping works.

The corrected code snippet is below.

    // SOMEWHERE OUTSIDE OF THE FUNCTION DEFINE:
    // Define a class to represent the input column type for the custom transform
    class InputData 
    {
        [VectorType(1)] // attribute specifies vector type of known length
        public VBuffer<Byte> Image1; // the VBuffer<> type actually represents the data
    }

    // Define a class to represent the output column type for the custom transform
    class OutputData 
    {
        // THE MAGICAL FIX: attribute specifies vector type of unknown length (i.e. VarVector)
        [VectorType()]
        public VBuffer<Byte> Image; // the VBuffer<> type actually represents the data
    }

    // ------------------------------------------------------------------
    // INSIDE THE FUNCTION THAT WILL DO THE TRAINING
    // ------------------------------------------------------------------

    // Fill an IDataView with the Label and ImagePath information:
    // The ImageData class is defined for this purpose and it is just:
    // 
    //  public class ImageData {
    //       public string ImagePath { get; set; }
    //       public string Label { get; set; }
    //  }
    // 
    // Create a bunch of image names and their labels into a list of ImageData objects.
    IEnumerable<ImageData> images = LoadImagesFromDirectory(folder: assetsRelativePath, 
                                                            useFolderNameAsLabel: true);
    IDataView imageData = mlContext.Data.LoadFromEnumerable(images);

    // Define the preprocessing pipeline to:
    // 1. Map the Label to a unique key
    // 2. Load the image data using the ImagePath column
    // 3. Reduce the resolution of the image data
    // 4. Crop the reduced resolution image data
    // 5. Transform the image data from type "Image" to a byte array.
    var preprocessingPipeline = mlContext.Transforms.Conversion.MapValueToKey(
                    inputColumnName: "Label",
                    outputColumnName: "LabelAsKey")
                .Append(mlContext.Transforms.LoadImages(
                    inputColumnName: "ImagePath",
                    outputColumnName: "RawImage",
                    imageFolder: assetsRelativePath))
                .Append(mlContext.Transforms.ResizeImages(
                    inputColumnName: "RawImage",
                    outputColumnName: "ResReduxImage",
                    imageWidth: 512,
                    imageHeight: 459,
                    resizing: Microsoft.ML.Transforms.Image.ImageResizingEstimator.ResizingKind.Fill))
                .Append(mlContext.Transforms.ResizeImages(
                    inputColumnName: "ResReduxImage",
                    outputColumnName: "CroppedImage",
                    imageWidth: 512,
                    imageHeight: 459,
                    resizing: Microsoft.ML.Transforms.Image.ImageResizingEstimator.ResizingKind.IsoCrop,
                    cropAnchor: Microsoft.ML.Transforms.Image.ImageResizingEstimator.Anchor.Center))
                .Append(mlContext.Transforms.ExtractPixels(
                    inputColumnName: "CroppedImage",
                    outputColumnName: "Image1",
                    outputAsFloatArray: false,
                    colorsToExtract: ImagePixelExtractingEstimator.ColorBits.Red));

     // Transform the raw ImageData into the TransformedData that the Model will train on
     IDataView preProcessedData = 
           preprocessingPipeline.Fit(imageData).Transform(imageData);

     // Create an action representing the custom transform...
     // The data itself does not need to be changed at all, so this is just an 
     // identity transform
     Action<InputData, OutputData> convertVecType 
               = (input, output) => output.Image = input.Image1;
 
     var convertTypePipeline = mlContext.Transforms.CustomMapping(convertVecType, 
                                                                 "convertVecType");
     preProcessedData = 
              convertTypePipeline.Fit(preProcessedData).Transform(preProcessedData);

     // Partition the data set for training and validation
     TrainTestData trainSplit = mlContext.Data.TrainTestSplit(data: preProcessedData, testFraction: 0.4);
     TrainTestData validationTestSplit = mlContext.Data.TrainTestSplit(trainSplit.TestSet);

     IDataView trainSet = trainSplit.TrainSet;
     IDataView validationSet = validationTestSplit.TrainSet;
     IDataView testSet = validationTestSplit.TestSet;

     var classifierOptions = new ImageClassificationTrainer.Options()
     {
          FeatureColumnName = "Image",
          LabelColumnName = "LabelAsKey",
          ValidationSet = validationSet,
          Arch = ImageClassificationTrainer.Architecture.ResnetV2101,
          MetricsCallback = (metrics) => Console.WriteLine(metrics),
          TestOnTrainSet = false,
          ReuseTrainSetBottleneckCachedValues = true,
          ReuseValidationSetBottleneckCachedValues = true,
          WorkspacePath=workspaceRelativePath
     };

     var trainingPipeline = 
           mlContext.MulticlassClassification.Trainers.ImageClassification(classifierOptions)
           .Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel"));

     // When I call Fit() in the next line of code, the following runtime error occurs: 
     // System.ArgumentOutOfRangeException: 'Schema mismatch for feature column 'Image': 
     // expected VarVector<Byte>, got Vector<Byte> Parameter name: inputSchema'
     ITransformer trainedModel = trainingPipeline.Fit(trainSet);
Related