I'm trying to import a large data set of around 80k objects. I'm trying to follow Apple example
I have two issues:
- In the code example there is a comment:
// taskContext.performAndWait runs on the URLSession's delegate queue
// so it won’t block the main thread.
But in my case I'm not using URLSession to fetch the JSON. The file is bundled with the app. In this case how to make sure the import won’t block the main thread. Should I create a custom queue ? Any example ?
In the example it's just importing an array of entities. But in my case I need to import just one entity that has 70k object in a relation to many.
So what I want to achieve is:
- If there is a
ContactBookdon't import anything because we have already imported the JSON. - If there is no
ContactBookcreate one and import all the 70kContactobject to thecontactsrelation of theContactBook. This should happen in batches like in the example and should not block the UI.
What I have tried:
private func insertContactbookIfNeeded() {
let fetch: NSFetchRequest<Contactbook> = ContactBook.fetchRequest()
let contactBookCount = (try? context.count(for: fetch)) ?? 0
if contactBookCount > 0 {
return
}
let contacts = Bundle.main.decode([ContactJSON].self, from: "contacts.json")
// Process records in batches to avoid a high memory footprint.
let batchSize = 256
let count = contacts.count
// Determine the total number of batches.
var numBatches = count / batchSize
numBatches += count % batchSize > 0 ? 1 : 0
for batchNumber in 0 ..< numBatches {
// Determine the range for this batch.
let batchStart = batchNumber * batchSize
let batchEnd = batchStart + min(batchSize, count - batchNumber * batchSize)
let range = batchStart..<batchEnd
// Create a batch for this range from the decoded JSON.
let contactsBatch = Array(contacts[range])
// Stop the entire import if any batch is unsuccessful.
if !importOneBatch(contactsBatch) {
assertionFailure("Could not import batch number \(batchNumber) range \(range)")
return
}
}
}
private func importOneBatch(_ contactsBatch: [ContactJSON]) -> Bool {
var success = false
// Create a private queue context.
let taskContext = container.newBackgroundContext()
taskContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
// NOT TRUE IN MY CASE: (Any suggestion ??)
// taskContext.performAndWait runs on the URLSession's delegate queue
// so it won’t block the main thread.
print("isMainThread: \(Thread.isMainThread)") // prints true
taskContext.performAndWait {
let fetchRequest: NSFetchRequest<ContactBook> = ContactBook.fetchRequest()
fetchRequest.returnsObjectsAsFaults = true
fetchRequest.includesSubentities = false
let contactBookCount = (try? taskContext.count(for: fetchRequest)) ?? 0
var contactBook: ContactBook?
if contactBookCount > 0 {
do {
contactBook = try taskContext.fetch(fetchRequest).first
} catch let error as NSError {
assertionFailure("can't fetch the contactBook \(error)")
}
} else {
contactBook = ContactBook(context: taskContext)
}
guard let book = contactBook else {
assertionFailure("Could not fetch the contactBook")
return
}
// Create a new record for each contact in the batch.
for contactJSON in contactsBatch {
// Create a Contact managed object on the private queue context.
let contact = Contact(context: taskContext)
// Populate the Contact's properties using the raw data.
contact.name = contactJSON.name
contact.subContacts = NSSet(array: contactJSON.subContacts { subC -> Contact in
let contact = Contact(context: taskContext)
contact.name = subC.name
})
book.addToContacts(contact)
}
// Save all insertions and deletions from the context to the store.
if taskContext.hasChanges {
do {
try taskContext.save()
} catch {
print("Error: \(error)\nCould not save Core Data context.")
return
}
// Reset the taskContext to free the cache and lower the memory footprint.
taskContext.reset()
}
success = true
}
return success
}
The problem is that this is very slow because in each batch I fetch the workbook (which is getting bigger in each iteration) to be able to insert new batch of contacts in the contact book. Is there a efficient way to avoid fetching the workbook in each batch ? also any suggestion to make this is faster ? increase the batch size ? create a background queue ?
Update:
I have tried to create a ContactBook once in insertWordbookIfNeeded and pass it to importOneBatch with each iteration but I get:
Thread 1: Exception: "Illegal attempt to establish a relationship
'contactBook' between objects in different contexts
