I managed to create a CoreML 2.0 model with flexible input/output shape sizes:
I can't figure out how to set the size in my Xcode project, however. If I set the input pixel buffer size 2048x2048, the output pixel buffer is still 1536x1536. If I set it to 768x768, the resulting pixel buffer is still 1536x1536 - but is blank outside the region of 768x768.
I examined the automatically generated Swift model class and don't see any clues there.
I can't find a single example anywhere showing how to use the "Flexibility" sizes.
In the WWDC 2018 Session 708 "What's New in Core ML", Part 1 it states:
This means that now you have to ship a single model. You don't have to have any redundant code. And if you need to switch between standard definition and high definition, you can do it much faster because we don't need to reload the model from scratch; we just need to resize it. You have two options to specify the flexibility of the model. You can define a range for its dimension, so you can define a minimal width and height and the maximum width and height. And then at inference pick any value in between. But there is also another way. You can enumerate all the shapes that you are going to use. For example, all different aspect ratios, all different resolutions, and this is better for performance. Core ML knows more about your use case earlier, so it can -- it has the opportunities of performing more optimizations.
They say "we just need to resize it". It so frustrating because they don't tell you how to just resize it! They also say "And then at inference pick any value in between" but offer no clue how to pick the value in between!
Here is how I added the flexible shape sizes:
import coremltools
from coremltools.models.neural_network import flexible_shape_utils
spec = coremltools.utils.load_spec('mymodel_fxedShape.mlmodel')
img_size_ranges = flexible_shape_utils.NeuralNetworkImageSizeRange()
img_size_ranges.add_height_range(640, 2048)
img_size_ranges.add_width_range(640, 2048)
flexible_shape_utils.update_image_size_range(spec, feature_name='inputImage', size_range=img_size_ranges)
flexible_shape_utils.update_image_size_range(spec, feature_name='outputImage', size_range=img_size_ranges)
coremltools.utils.save_spec(spec, 'myModel.mlmodel')
Here is the description of the model:
description {
input {
name: "inputImage"
shortDescription: "Image to stylize"
type {
imageType {
width: 1536
height: 1536
colorSpace: BGR
imageSizeRange {
widthRange {
lowerBound: 640
upperBound: 2048
}
heightRange {
lowerBound: 640
upperBound: 2048
}
}
}
}
}
output {
name: "outputImage"
shortDescription: "Stylized image"
type {
imageType {
width: 1536
height: 1536
colorSpace: BGR
imageSizeRange {
widthRange {
lowerBound: 640
upperBound: 2048
}
heightRange {
lowerBound: 640
upperBound: 2048
}
}
}
}
}
}
There are two layers using "outputShape":
layers {
name: "SpatialFullConvolution_63"
input: "Sequential_53"
output: "SpatialFullConvolution_63_output"
convolution {
outputChannels: 16
kernelChannels: 32
nGroups: 1
kernelSize: 3
kernelSize: 3
stride: 2
stride: 2
dilationFactor: 1
dilationFactor: 1
valid {
paddingAmounts {
borderAmounts {
}
borderAmounts {
}
}
}
isDeconvolution: true
hasBias: true
weights {
}
bias {
}
outputShape: 770
outputShape: 770
}
}
...relu layer...
layers {
name: "SpatialFullConvolution_67"
input: "ReLU_66"
output: "SpatialFullConvolution_67_output"
convolution {
outputChannels: 8
kernelChannels: 16
nGroups: 1
kernelSize: 3
kernelSize: 3
stride: 2
stride: 2
dilationFactor: 1
dilationFactor: 1
valid {
paddingAmounts {
borderAmounts {
}
borderAmounts {
}
}
}
isDeconvolution: true
hasBias: true
weights {
}
bias {
}
outputShape: 1538
outputShape: 1538
}
}
I am now trying to figure out how to remove the outputShape from those two layers.
>>> layer = spec.neuralNetwork.layers[49]
>>> layer.convolution.outputShape
[1538L, 1538L]
I tried setting it to []:
layer.convolution.outputShape = []
To a Shape:
layer.convolution.outputShape = flexible_shape_utils.Shape(())
Whatever I try, I get the error:
TypeError: Can't set composite field
Do I have to create a new layer and then link it to the layer that is outputting to it and the layer it is outputting to?
