System.Runtime.InteropServices.SEHException HResult=0x80004005 on BruteForce matching images with SIFT

I am using BruteForce KNN matching to compare a set of images.

I have this method, based on the EMGU github:

public long Match(DescriptorInfo modelImage, DescriptorInfo observedImage)
{
    int k = 2;
    var matches = new VectorOfVectorOfDMatch();
    var mask = new Mat(matches.Size, 1, DepthType.Cv8U, 1);
    mask.SetTo(new MCvScalar(255));

    // Brute force, slower but more accurate
    // You can use KDTree for faster matching with slight loss in accuracy

    _matcher.Add(modelImage.Descriptors);

    _matcher.KnnMatch(observedImage.Descriptors, matches, k, null);
    mask = new Mat(matches.Size, 1, DepthType.Cv8U, 1);
    mask.SetTo(new MCvScalar(255));
    Features2DToolbox.VoteForUniqueness(matches, (double)this.Threshold, mask);

    _nonZeroCount = CvInvoke.CountNonZero(mask);
    if (_nonZeroCount >= 4)
    {
        try
        {
            _nonZeroCount = Features2DToolbox.VoteForSizeAndOrientation(modelImage.Keypoints, observedImage.Keypoints,
                    matches, mask, 1.5, 20);
        }
        catch (Exception)
        {
            // No way to solve this for now, ignore.
        }
    }

    return _nonZeroCount;}

On occassion I get an error on this line:

_nonZeroCount = Features2DToolbox.VoteForSizeAndOrientation(modelImage.Keypoints, observedImage.Keypoints,
                    matches, mask, 1.5, 20);

The exception I get only details this:

System.Runtime.InteropServices.SEHException
  HResult=0x80004005
  Message=External component has thrown an exception.
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

I am comparing a list of images, it happens on only a few of them so my guess would be that somehow there's an issue with matching these.

I am using

• SIFT as detector
• BFMatcher - DistanceType set to L2, crosscheck set to false

The exception info is not helpful at all, how can I pin point -and hopefully solve- what's going wrong?