Efficiently recover data from Firestore

Viewed 63

As a college project, I have a couple of Raspberry Pi's collecting weather and environment data throughout the campus.

They upload data to Firestore in this arrangement:

Devices
    ├───DeviceId1
    │   ├───2021-06-14
    │   │   ├───13.00
    │   │   │   └───JSON FILE
    │   │   ├───13.05
    │   │   │   └───JSON FILE
    │   │   ├───13.10
    │   │   │   └───JSON FILE
    │   │   └───13.15
    │   │       └───JSON FILE
    │   ├───2021-06-15
    │   │   ├───13.00
    │   │   │   └───JSON FILE
    │   │   ├───13.05
    │   │   │   └───JSON FILE
    │   │   ├───13.10
    │   │   │   └───JSON FILE
    │   │   └───13.15
    │   │       └───JSON FILE
    │   └───2021-06-16
    │       ├───13.00
    │       │   └───JSON FILE
    │       ├───13.05
    │       │   └───JSON FILE
    │       ├───13.10
    │       │   └───JSON FILE
    │       └───13.15
    │           └───JSON FILE
    ├───DeviceId2
    │   ├───2021-06-14
    │   │   ├───13.00
    │   │   │   └───JSON FILE
    │   │   ├───13.05
    │   │   │   └───JSON FILE
    │   │   ├───13.10
    │   │   │   └───JSON FILE
    │   │   └───13.15
    │   │       └───JSON FILE
    │   ├───2021-06-15
    │   │   ├───13.00
    │   │   │   └───JSON FILE
    │   │   ├───13.05
    │   │   │   └───JSON FILE
    │   │   ├───13.10
    │   │   │   └───JSON FILE
    │   │   └───13.15
    │   │       └───JSON FILE
    │   └───2021-06-16
    │       ├───13.00
    │       │   └───JSON FILE
    │       ├───13.05
    │       │   └───JSON FILE
    │       ├───13.10
    │       │   └───JSON FILE
    │       └───13.15
    │           └───JSON FILE
    └───DeviceId3
        ├───2021-06-14
        │   ├───13.00
        │   │   └───JSON FILE
        │   ├───13.05
        │   │   └───JSON FILE
        │   ├───13.10
        │   │   └───JSON FILE
        │   └───13.15
        │       └───JSON FILE
        ├───2021-06-15
        │   ├───13.00
        │   │   └───JSON FILE
        │   ├───13.05
        │   │   └───JSON FILE
        │   ├───13.10
        │   │   └───JSON FILE
        │   └───13.15
        │       └───JSON FILE
        └───2021-06-16
            ├───13.00
            │   └───JSON FILE
            ├───13.05
            │   └───JSON FILE
            ├───13.10
            │   └───JSON FILE
            └───13.15
                └───JSON FILE

Every device records a JSON file on Firestore every 5 minutes or so, every day. And every JSON file has exactly 1000 lines of data.

What I'm doing now is creating an webpage with flask that graphs the data from Firestore so my peers can easily access it.

This is my code snippet to retrieve temperature data from Firebase:

cred = credentials.Certificate("myFirestoreCreds.json")

firebase_admin.initialize_app(cred, {
'projectId': "myproject",
})    

db = firestore.client()

device = 'DeviceID'


collections = db.collection('Devices').document(device).collections()
for dates in collections:
    for hours in dates.stream():
        doc_ref = db.collection(u'Devices').document(device).collection(dates.id).document(hours.id)
        doc = doc_ref.get()
        if doc.exists:
            bd = doc.to_dict()
        else:
            print(u'No such document!')
        //Code that plots the data goes here

I quickly stumbled upon on the 503 error that, from what I gathered, prevents me for querying database longer than 30 seconds or retrieving more than 50.000 lines of data, not sure about which one really.

So, how can I tweak my code to efficiently query Firestore?

EDIT: I menaged to work around error 503 by breaking up my code:

#gets all dates from a specific device
datesList = []
datehour = defaultdict(list)
collections = db.collection('Devices').document(device).collections()
for dates in collections:
    datesList.append(dates.id)

#gets all hours from days
assist = []
for day in datesList:
    docs  = db.collection(u'Devices').document(device).collection(day).stream()
    assist = []
    for doc in docs:
        assist.append(doc.id)
    datehour[str(day)] = assist

for key in datehour.keys():
    for date in datehour[key]:
        doc_ref = db.collection(u'Devices').document(device).collection(key).document(date )
        doc = doc_ref.get()
        if doc.exists:
           bd = doc.to_dict()
        else:
           print(u'No such document')

But this solution is stupidly inefficient and takes about 6 minutes to retrieve all the data of 3 days of logging one device.

With that in mind, my question is still valid: How can I tweak my code to efficiently query Firestore?

0 Answers
Related