I'm trying to optimize my PyTorch model to use it on iOS. I'm using coremltools to convert the model to CoreML and XCode 14 beta/iOS 16 beta in order to use new Performance Instruments from Apple.
I'm currently having troubles with two types of layers: BatchNorm and general_padding.
The Performance report shows that these layers are not available on the Apple Neural Engine (ANE) so the model mostly runs on GPU which makes it slow.
I downloaded several models from Apple model library (FCRN model for example) and profiled them. While the General tab shows that these models have a lot of BatchNorm layers, the Performance tab doesn't show any BatchNorm layers at all and runs the whole model on ANE.
Why can this happen, are there different types of batchnorm layers or something?
I noticed that most batchnorm layers are not supported on the ANE but several are marked as supported on ANE even though they seem to be similar.
Also it looks like padding is not supported on ANE at all, is that true that I should remove all padding from the model architecture in order for it to run on ANE?