I have data acquired from collection points (sourceId), which record data (samples) from different probes (probeId). I need to structure the data as samples by probeId by sourceId.
Specification
- Insert often
- Read occasionally
- Never update or delete
- Read by indexing with a list of
sourceIdsand a list ofprobeIds - Insert atomically, including when
probeIdis not in the list of probes and the probe sub-document needs to be created
Question
What is the best structure to achieve the above? Particularly, how can the preferred structure facilitate achieving point 5? I have not been able to get upsert to match only for probeId and ignore samples and meta.
So far I have considered the following 2 structures:
{'sourceId': ObjectId(),
'probes': [
{
'probeId': ObjectId(),
'meta': {...},
'samples': [...]
},
...
]
}
and
{'sourceId': ObjectId(),
'probes': [
{
ObjectId(): { # probeId key
'meta': {...},
'samples': [...]
}
}
]
}
Currently I use the first structure and search like this:
db.collection.aggregate(
{'$match': {'sourceId': {'$in': source_id_list}}},
{'$project': {'probes': {'$filter': {'input': '$probes', 'cond': {'$in': ['$$this.probeId', probe_id_list]}}}}}
)
For clarity point 5 should work as follows (assuming using the first structure):
Before inserting:
{'sourceId': ObjectId('1'),
'probes': [
{
'probeId': ObjectId('2'),
'meta': {...},
'samples': [...]
}
]
}
After inserting a sample for probeId=3 for sourceId=1 (note that not only has the sample been added but also the containing subdocument):
{'sourceId': ObjectId('1'),
'probes': [
{
'probeId': ObjectId('2'),
'meta': {...},
'samples': [...]
},
{
'probeId': ObjectId('3'),
'meta': {},
'samples': [<NEW SAMPLE DATA>]
}
]
}