OpenCV holds the memory longer than it is needed

I have a program that does something like:

#include <iostream>
#include "opencv2/core.hpp"

int main() {
    for (int i = 0; i < 10; ++i) {
        cv::Mat mats[10];
        for (auto & m : mats) {
            m = cv::Mat::zeros(5000, 5000, CV_8U);
            // use the mats ...
        }
    }

    getchar(); // at this point, process holds hundreds of megabytes of memory
}

After this code block ends, the Mats are out of scope and there are definitely no other references to the data, the process somehow still takes hundreds of megabytes of RAM. If I do this Instead:

#include <iostream>
#include "opencv2/core.hpp"

int main() {
    for (int i = 0; i < 10; ++i) {
        cv::Mat mats[10];
        for (auto & m : mats) {
            m.data = new uchar[5000 * 5000];
            // use the memory ...
            memset(m.data, i, 5000 * 5000);
        }
        for (auto & m : mats) {
            delete m.data;
        }
    }

    getchar(); // at this point, process holds almost zero memory
}

the process takes almost zero memory after this code. I am guessing OpenCV has some internal memory management where it doesn't free the memory for potential future use. That's a problem for me because I need this memory for other things after this code run. This process serves as a server so it stays up indefinitely. Is there a way to force OpenCV to free this memory?

I am using OpenCV 4.3.0 on Ubuntu 16.04