$lookup is applied again on all subfields of a looked up object

I'm using Spring Data MongoDB and applying $lookup on a field with ObjectId value, what I'm getting is the referenced document like I want but also the fields of that document are also populated, $lookup is applied again on the fields of the referenced document which causes the query to take a very long time to execute (around 5mins).

The query works fine using a raw query from a mongo client without spring data MongoDB, the lookup is applied only once and I get a document that has ObjectId value in the fields instead of those fields being populated too.

This is the raw query

{$match : {_id : new ObjectId("63282356bb1b311dd0e63f67")}},
    {$lookup : {
    "from":"paper",
    "let":{
        "userId":"$_id"
     },
    "pipeline":[
        {$facet:{
        sent:[
         {$match:{
            "$expr":{"$eq":["$to", "$$userId"]}
        }}
        ],
        }}
],
    "as":"papers"
}}
])

And I'm translating this to spring data mongodb like this

   Aggregation aggregation = Aggregation.newAggregation(
                matchStage,
                aggregationOperationContext -> {
                    Document lookupPipeline = new Document("$lookup",
                            new Document("from", "paper")
                                    .append("let", new Document("userId", "$_id"))
                                    .append("pipeline", Arrays.<Object> asList(
                                            new Document("$facet",
                                                    new Document("sent",Arrays.<Object>asList(
                                                            new Document("$match", new Document("$expr",
                                                                    new Document("$eq", Arrays.<Object>asList("$to", "$$userId"))
                                                            ))
                                                    )
                                                    )
                                            )
                                    ))
                                    .append("as", "papers")
                    );
                    return aggregationOperationContext.getMappedObject(lookupPipeline);
                }
                );

I'm using the custom aggregation because using $lookup with the pipeline field is not supported in spring data MongoDB as of now

Also I'm using @DocumentReference

 public class User{
    @ReadOnlyProperty
    @DocumentReference
    private List<Paper> papers;
}