Swift nested filter optimization?

I was trying to do something in Swift that would be easy in Objective-C using KVC. The new Contacts framework added in iOS9 is for the most part easier to use than the old AddressBook API. But finding a contact by its mobile phone number seems to be difficult. The predicates provided for finding contacts are limited to the name and the unique identifier. In Objective-C you could get all the contacts and then use an NSPredicate to filter on a KVC query. The structure is:

CNContact->phoneNumbers->(String, CNPhoneNumber->stringValue)

Presume in the code below that I fetched the contacts via:

let keys = [CNContactEmailAddressesKey,CNContactPhoneNumbersKey, CNContactFormatter.descriptorForRequiredKeysForStyle(.FullName)]
let fetchRequest = CNContactFetchRequest(keysToFetch: keys)
var contacts:[CNContact] = []
try! CNContactStore().enumerateContactsWithFetchRequest(fetchRequest) { ...

I want to compare the stringValue to a known value. Here's what I have so far from a playground:

import UIKit
import Contacts

let JennysPhone    = "111-867-5309"
let SomeOtherPhone = "111-111-2222"
let AndAThirdPhone = "111-222-5309"

let contact1 = CNMutableContact()
contact1.givenName = "Jenny"
let phone1 = CNPhoneNumber(stringValue: JennysPhone)
let phoneLabeled1 = CNLabeledValue(label: CNLabelPhoneNumberMobile, value: phone1)
contact1.phoneNumbers.append(phoneLabeled1)

let contact2 = CNMutableContact()
contact2.givenName = "Billy"
let phone2 = CNPhoneNumber(stringValue: SomeOtherPhone)
let phoneLabeled2 = CNLabeledValue(label: CNLabelPhoneNumberMobile, value: phone2)
contact2.phoneNumbers.append(phoneLabeled2)

let contact3 = CNMutableContact()
contact3.givenName = "Jimmy"
let phone3 = CNPhoneNumber(stringValue: SomeOtherPhone)
let phoneLabeled3 = CNLabeledValue(label: CNLabelPhoneNumberMobile, value: phone3)
contact3.phoneNumbers.append(phoneLabeled3)

let contacts = [contact1, contact2, contact3]

let matches = contacts.filter { (contact) -> Bool in
    let phoneMatches = contact.phoneNumbers.filter({ (labeledValue) -> Bool in
        if let v = labeledValue.value as? CNPhoneNumber
        {
            return v.stringValue == JennysPhone
        }
        return false
    })
    return phoneMatches.count > 0
}

if let jennysNum = matches.first?.givenName
{
    print("I think I found Jenny:  \(jennysNum)")
}
else
{
    print("I could not find Jenny")
}

This does work, but it's not efficient. On a device I would need to run this in a background thread, and it could take a while if the person has a lot of contacts. Is there a better way to find a contact by phone number (or email address, same idea) using the new iOS Contacts framework?