.Net core gRPC unmanaged memory usage

I have a simple service that listens to RabbitMQ, calls some gRPC services and does other stuff like DB updates etc. I started noticing Kubernetes pods failing with OOM exception so I took out all the logic from Consume method and controller constructor and started adding back parameters one by one. Everything is normal until I pass my gRPC client to constructor- then unmanaged memory starts going up with each new message and never goes down (the Consume method is still empty). Any ideas what I'm doing wrong? My client is added in Program.cs like this:

services.AddGrpcClient<Notification.Api.Notification.NotificationClient>(o => { o.Address = new Uri(sso.GrpcOptions.ApiBaseAddress); })
         .ConfigureChannel(c =>
             {
                 c.LoggerFactory = LoggerFactory.Create(logging =>
                    {
                        logging.AddConsole();
                        logging.SetMinimumLevel(LogLevel.Warning);
                    });
                    c.HttpHandler = new SubdirectoryHandler(new HttpClientHandler()
                        {                                
                        }, sso.GrpcOptions.NotifSubdirectory);
                    });