Structure nested list of objects for searching by object key

Viewed 30

I have data acquired from collection points (sourceId), which record data (samples) from different probes (probeId). I need to structure the data as samples by probeId by sourceId.

Specification

  1. Insert often
  2. Read occasionally
  3. Never update or delete
  4. Read by indexing with a list of sourceIds and a list of probeIds
  5. Insert atomically, including when probeId is not in the list of probes and the probe sub-document needs to be created

Question

What is the best structure to achieve the above? Particularly, how can the preferred structure facilitate achieving point 5? I have not been able to get upsert to match only for probeId and ignore samples and meta.

So far I have considered the following 2 structures:

{'sourceId': ObjectId(),
  'probes': [
    {
      'probeId': ObjectId(),
      'meta': {...},
      'samples': [...]
    },
    ...
  ]
}

and

{'sourceId': ObjectId(),
  'probes': [
    {
      ObjectId(): {                # probeId key
        'meta': {...},
        'samples': [...]
      }
    }
  ]
}

Currently I use the first structure and search like this:

  db.collection.aggregate(
    {'$match': {'sourceId': {'$in': source_id_list}}},
    {'$project': {'probes': {'$filter': {'input': '$probes', 'cond': {'$in': ['$$this.probeId', probe_id_list]}}}}}
  )

For clarity point 5 should work as follows (assuming using the first structure):

Before inserting:

{'sourceId': ObjectId('1'),
  'probes': [
    {
      'probeId': ObjectId('2'),
      'meta': {...},
      'samples': [...]
    }
  ]
}

After inserting a sample for probeId=3 for sourceId=1 (note that not only has the sample been added but also the containing subdocument):

{'sourceId': ObjectId('1'),
  'probes': [
    {
      'probeId': ObjectId('2'),
      'meta': {...},
      'samples': [...]
    },
    {
      'probeId': ObjectId('3'),
      'meta': {},
      'samples': [<NEW SAMPLE DATA>]
    }
  ]
}
0 Answers
Related