How to use the iOS 12 batch API for classification of multiple images

Jekapa 0
Viewed 19

Up to now, I have used VNCoreMLRequest to classify images. Now, to improve performance for classification of multiple images, I want to switch to the batch API. It seems that the Vision framework does not support batches and instead, the MLModel has to be used directly.

However, I am struggling to call the API:

[coreMLModel predictionsFromBatch:<#(nonnull id<MLBatchProvider>)#> 
                          options:<#(nonnull MLPredictionOptions *)#> 
                            error:<#(NSError *__autoreleasing  _Nullable * _Nullable)#>]

Does anyone have a code sample, especially on how to set up the MLBatchProvider from a bunch of images?

P.S. Objective-C would be preferred, but Swift code is also welcome :-)

coreml objective-c xcode
1 Answers

I figured it out - hope my implementation can be useful for others (let me know if the code can be improved):

- (NSMutableArray*)classifyWithImageArray:(NSMutableArray*)smallImgs
{
    NSMutableArray *letterArray = [[NSMutableArray alloc] init];
    for (int i = 0; i < smallImgs.count; i++) {
        [letterArray addObject:@"?"];
    }
    MLImageConstraint *constraint = [(MLModel*)coreMLModel modelDescription].inputDescriptionsByName[@"input_1"].imageConstraint;
    NSMutableDictionary *featureDict = [[NSMutableDictionary alloc] init];
    NSMutableArray *featureProviders = [[NSMutableArray alloc] init];
    NSError *error = nil;
    
    for (int i = 0; i < smallImgs.count; i++)
    {
        MLFeatureValue *imageFeature = [MLFeatureValue featureValueWithCGImage:[(UIImage*)smallImgs[i] CGImage]
                                                                    constraint:constraint options:nil error:&error];
        NSAssert(!error, @"Error creating MLFeatureValue!");
        if (error) return letterArray;
        
        featureDict[@"input_1"] = imageFeature;

        MLDictionaryFeatureProvider *featureProvider = [[MLDictionaryFeatureProvider new] initWithDictionary:featureDict
                                                                                                       error:&error];
        NSAssert(!error, @"Error creating MLFeatureProvider!");
        if (error) return letterArray;

        [featureProviders addObject:featureProvider];
    }
    MLArrayBatchProvider *batchProvider = [[MLArrayBatchProvider alloc] initWithFeatureProviderArray:featureProviders];

    MLArrayBatchProvider *predictions = (MLArrayBatchProvider*)[coreMLModel predictionsFromBatch:batchProvider
                                                                                           error:&error];
    for (int i = 0; i < predictions.array.count; i++) {
        MLFeatureValue *classLabel = [predictions.array[i] featureValueForName:@"classLabel"];
        letterArray[i] = [classLabel stringValue];
    }
    return letterArray;
}

I noticed a small performance improvement (4 %) when going from VNCoreMLModel to MLModel, but unfortunately no further improvement when switching from sequential single predictions in a loop to the batch prediction. I assume this is due to my use case with small images and a not too complex neural net.

Jekapa0
Related