ML.Net Rounding Float32 Results to 0 or 1

Ok, new day. I traced the code path of the python project and made an MVP. In doing so, I have very few things to look at the difference of.

import os

import onnxruntime
import skimage.transform
import tensorflow as tf


def main():
    # disable CUDA acceleration for simplicity in running the test
    # you need drivers, an nvidia gpu, etc. for that
    os.environ['CUDA_VISIBLE_DEVICES'] = '-1'

    image_path = 'C:\\Users\\da3ds\\Pictures\\image.jpg'
    model_path = 'C:\\Users\\da3ds\\Downloads\\deepdanbooru-v3-20211112-sgd-e28\\model-resnet-custom_v3.onnx'

    # load tags
    tags_path = 'C:\\Users\\da3ds\\Downloads\\deepdanbooru-v3-20211112-sgd-e28\\tags.txt'
    with open(tags_path, 'r') as tags_stream:
        tags = [tag for tag in (tag.strip() for tag in tags_stream) if tag]

    # create inference session
    model = onnxruntime.InferenceSession(model_path, providers=['CPUExecutionProvider'])

    width = model.get_inputs()[0].shape[1]  # 512
    height = model.get_inputs()[0].shape[2]  # 512

    image_raw = tf.io.read_file(image_path)
    image = tf.io.decode_png(image_raw, channels=3)

    image = tf.image.resize(image, size=(width, height), method=tf.image.ResizeMethod.AREA, preserve_aspect_ratio=True)
    image = image.numpy()  # EagerTensor to np.array

    image_width = image.shape[0]
    image_height = image.shape[1]

    t = skimage.transform.AffineTransform(translation=(-image_width * 0.5, -image_height * 0.5))
    t += skimage.transform.AffineTransform(translation=(width * 0.5, height * 0.5))

    image = skimage.transform.warp(image, t.inverse, output_shape=(width, height), order=1, mode='edge')

    # at this point all widths and heights are probably 512
    
    # normalize the image
    image = image / 255.0

    image_shape = image.shape
    # build the input shape of Vector<1, 512, 512, 3>
    image = image.reshape((1, image_shape[0], image_shape[1], image_shape[2]))
    onnx_result = model.run(None, {'input_1:0': image})
    # onnx_result is 2 arrays deep for reason
    # 1 would make sense, as it can handle batches
    onnx_result = onnx_result[0][0]

    # print a nice result
    for i, tag in enumerate(tags):
        print(f'({onnx_result[i]:05.3f}) {tag}')


if __name__ == '__main__':
    main()

Conveniently, in doing so, I made a mistake in a default value that yielded the same result as the ML.Net results: (not) Normalizing the Image. I couldn't figure out how to do that in the ML.Net pipeline, so I made the array with Magick.Net and fed it to ML.Net directly.

Here's the final code:

using ImageMagick;
using Microsoft.ML;
using Microsoft.ML.Data;

namespace OnnxTest;

public static class Program
{
    public static void Main(string[] args)
    {
        var tags = File.ReadLines(@"C:\Users\da3ds\Downloads\deepdanbooru-v3-20211112-sgd-e28\tags.txt");
        var imageLocation = @"C:\Users\da3ds\Pictures\image.jpg";
        var modelLocation = @"C:\Users\da3ds\Downloads\deepdanbooru-v3-20211112-sgd-e28\model-resnet-custom_v3.onnx";
        MLContext mlContext = new MLContext(seed: 0);
        
        Console.WriteLine("Read model");
        Console.WriteLine($"Model location: {modelLocation}");
        Console.WriteLine(
            $"Default parameters: image size=({InputModel.Width},{InputModel.Height})");
        Console.WriteLine($"Images location: {imageLocation}");
        Console.WriteLine("");
        Console.WriteLine("=====Identify the objects in the images=====");
        Console.WriteLine("");

        // Create IDataView from empty list to obtain input data schema
        var data = new InputModel { Data = GetImage(imageLocation) };
        // Define scoring pipeline
        var predictionEngine = GetPredictionEngine(mlContext, modelLocation);
        var output = predictionEngine.Predict(data);

        var outputMapped = tags.Zip(output.Scores).Select(t => new { Tag = t.First, f = t.Second })
            .ToDictionary(a => a.Tag, a => a.f);

        var outputTags = outputMapped.Where(a => a.Value > 0.80f).Select(a => (Tag: a.Key, Score: a.Value))
            .ToList();
        foreach (var tag in outputTags)
        {
            Console.WriteLine($"({tag.Score:P1}) {tag.Tag}");
        }
    }

    private static PredictionEngine<InputModel, OutputModel> GetPredictionEngine(MLContext mlContext, string modelLocation)
    {
        var transformer = GetBasicTransformer(mlContext, modelLocation);

        // Fit scoring pipeline
        var predictionEngine = mlContext.Model.CreatePredictionEngine<InputModel, OutputModel>(transformer);
        return predictionEngine;
    }

    private static ITransformer GetBasicTransformer(MLContext mlContext, string modelLocation)
    {
        var estimator = mlContext.Transforms.ApplyOnnxModel(OutputModel.ModelOutput, InputModel.ModelInput,
                modelLocation);

        var transformer = estimator.Fit(mlContext.Data.LoadFromEnumerable(Array.Empty<InputModel>()));
        return transformer;
    }

    public static float[] GetImage(string imagePath)
    {
        using var mImage = new MagickImage(imagePath);
        mImage.Quality = 100;
        mImage.BackgroundColor = new MagickColor(0, 0, 0);
        mImage.HasAlpha = false;
        mImage.Resize(new MagickGeometry($"{InputModel.Width}>x{InputModel.Height}>"));
        mImage.Extent(InputModel.Width, InputModel.Height, Gravity.Center, new MagickColor(0,0,0));
        var pixels = mImage.GetPixels();
        var array = pixels.ToArray();
        var data = new float[InputModel.Width * InputModel.Height * InputModel.Channels];
        for (var index = 0; index < array.Length; index++)
        {
            data[index] = array[index] / 255.0f;
        }

        return data;
    }

    class InputModel
    {
        public const int Width = 512;
        public const int Height = 512;
        public const int Channels = 3;
        public const string ModelInput = "input_1:0";

        [ColumnName(ModelInput)]
        [VectorType(1, Width, Height, Channels)]
        public float[] Data { get; set; }
    }

    class OutputModel
    {
        // output tensor name
        public const string ModelOutput = "Identity:0";

        [ColumnName(ModelOutput)]
        public float[] Scores { get; set; }
    }
}

Obviously, the final...final code will be less of an MVP, but this was a test. I leave this as a trail of my efforts in case someone else hits a similar issue. At the very least, it gives my debugging steps and some sample code. Thanks for being my rubber ducks.