How to batch more than 500 operations in python for firestore?

user_5482999 0
Viewed 2037

I'm creating documents from web scraping in python and uploading them to Firestore. To do so I'm adding them to a dictionary and uploading them from a for loop in python, one by one(ideally would be better to upload the collection at once, but that doesn't seem an option). I want to use batches, however they have the 500 limit per batch and I need to do more than 100,000 operations. The operations are merely set() operations and a couple of update() Is there a function to know the current size of the batch so I can reinitialize it? What is the best way to use batches for more than 500 operations in python?

google-cloud-firestore firebase python
3 Answers

The best way I found to deal with the 500 batch limit, while working with python, is to put all my data I want to send to Firestore in a 'Flat' dictionary so I can work with every unique document. This dictionary has as key for every document the form of: 'collection_document_collection_document...' while the value for that key would be a dictionary with:

{'action': 'set', 'reference': reference, 'document': {}}

The 'action' can be 'set', 'update' or 'delete', the 'reference' key is an actual Firestore reference, and the 'document' is just the document. For instance this are 2 documents in different locations.

{
    'user_data_roger':
    {'action': 'set', 'reference': db.collection('user_data').document('roger'), 'document': {'name': 'Roger', 'age': 37}},
    'user_data_roger_works_april':
    {'action': 'update', 'reference': db.collection('user_data').document('roger').collection('works').document('april'), 'document': {'is_valid': True, 'in_progress': True, 'level':5}},
}

After processing all the data I need I want to split the dictionary in arrays of 500 items and then add all of those Items to the batch, using the 'action' key for the batch.

# Convert dictionary to a list
dictionary_list = []
for item in dictionary:
    dictionary_list.append(dictionary.get(item))
# Split List in lists containing 500 items per list
list_to_batch = [dictionary_list[item:item+500] for item in range(0, len(dictionary_list), 500)]
# Finally iterate through the 'list_to_batch' add each item to the batch and commit using a for loop
for item in list_to_batch:
    batch = db.batch()
    for document in item:
        if document['action'] == 'set':
            batch.set(document['reference'], document['value'])
        elif draw['action'] == 'update':
            batch.update(document['reference'], document['value'])
        else:
            batch.delete(document['reference'], document['value'])
    # Finally commit the batch
    batch.commit()

In my particular case after processing all the data I needed, I ended up with more than 700,000 operations, so watch out for the billing :-D

user_54829990

The maximum number of operations in a Batch is 500. If you need more operations, you'll need multiple batches.

There is no API to determine the current number of operations in a Batch. If you need that, you will have to track it yourself.

user_2091030

You can not exceed the 500 operations limit, but you have some simple ways to split your operations into small batches.

Let's say you need to update every document in a collection. This works:

collection_ref = firestore_client.collection(col)

batch = firestore_client.batch()

for i, doc in enumerate(collection_ref.list_documents()):
    if i % 500 == 499:
        batch.commit()
        batch = firestore_client.batch()
    batch.update(doc, {"some field": "some value"})

batch.commit()
user_87211690
Related