How to use AggregateToCollection() to store the results of an IMongoQueryable in a collection

user_4852638 0
Viewed 719

Unless I'm missing something, there appear to be large gaps in the documentation for the C# MongoDb driver.

I'm trying to take an IMongoQueryable (which is the result of various LINQ Where, Select, etc. operations) and store the results in a collection, on the database side. Of course, I could iterate through it on the client side and persist it that way, but even in batches that's not efficient, and in shell it's quite an easy operation to use $merge or $out.

On a collection, there's the method AggregateToCollection<TResult>() which is exactly what I want I believe, but it takes a PiplineDefinition<TDocument, TResult> parameter that I have little idea how to actually generate from the IMongoQueryable.

I've been going down the rabbit hole of using GetExecutionModel() on the IMongoQueryable to get a BsonDocument or Json string, but then I still have no clue how to turn that into the PipelineDefinition that I need!

What I would have thought I would find, would be an extension method on IMongoQueryable that would allow me to send it to be merged into a collection.

So far to be able to send an IMongoQueryable to a collection, I have something along the lines of:

    var executionModelDocument = queryable.GetExecutionModel().ToBsonDocument();

    // somehow turn the document into pipeline stages and a pipeline??

    await _database.GetCollection<TDocument>().AggregateToCollectionAsync<TResult>(pipeline);

How do I actually make this happen?

mongodb iqueryable linq c#
1 Answers

Okay, well this was ultimately easier than expected but still frustrating.

First of all, it turns out you can get the pipeline from an IMongoQueryable as JSON simply by calling ToString() on it. While I say JSON, it's not quite - the output contains the full JSON for the pipeline as an array of stages, but has a label in front and contains the JSON within parentheses. I took a shortcut here and just took a dirty substring:

var queryableJson = queryable.ToString();
var trimmedDocument = queryableJson.Substring(10, queryableJson.Length - 11); // TODO: more reliably get the true json rather than blindly removing what should be "aggregate(" and ")"

Next, I re-serialized the JSON back into an array of BsonDocument and made a PipelineDefinition out of it (a BsonDocument[] can be implicitly cast to a PipelineDefinition):

PipelineDefinition<TDocument, TResult> pipelineQueryable = BsonSerializer.Deserialize<BsonDocument[]>(trimmedDocument);

Note that while the documentation at http://mongodb.github.io/mongo-csharp-driver/2.4/reference/driver/definitions/#pipelines says I could have implicitly cast a single BsonDocument for the pipeline rather than an array of stages, this is not true unless there's an overload in another namespace that I didn't find.

Now that we have a defined pipeline for the IMongoQueryable so far, we can simply add stages to it to achieve the result that we want (in this case, to merge the results of the pipeline into another collection). You can specify properties of the MergeStageOptions<TResult> object to control behaviour, but the defaults worked fine for me:

var stageMerge = PipelineStageDefinitionBuilder.Merge<TResult, TResult>(_database.GetCollection<TResult>(), new MergeStageOptions<TResult>());
var mergePipeline = pipelineQueryable.AppendStage(stageMerge);

Using our newly enhanced pipeline, we can apply it to the source collection to merge the output into the destination collection:

_database.GetCollection<TDocument>().AggregateToCollection(mergePipeline);

For simplicity, what I've demonstrated here is not asynchronous, however I've taken advantage of the awaitable methods in the driver in my actual code as the synchronous methods simply wrap the awaitable versions.

That's really all there was to it! When I get more time, I'll go back and try and skip the serialization-deserialization step as it's noticeably slow and shouldn't be necessary. I also plan on turning this into an extension method for fluency's sake.

user_48526380
Related