Im trying to request a big load of data and then parse it into a report. The problem is that the data I'm requesting has 27 million lines of records, each having 6 joins, which when loaded via Entity framework uses all of the server RAM. Ive implemented a pagination system to buffer the processing into smaller chunks like you would do with an IO operation.
I request 10,000 records, writing them to a file stream (to disk) and them I'm trying to clear the 10,000 records from memory as they're no longer needed.
I'm having trouble garbage collecting the database context. I've tried disposing the object, nulling the reference and then creating a new context on the next batch of 10,000 records. This does not seem to work. (this was recommended by one of the devs on ef core: https://github.com/aspnet/EntityFramework/issues/5473)
The only other alternative I see is to use a raw SQL query to achieve what I want. I'm trying to build the system to deal with any request size and the only variable factor will be the time it takes to produce the reports. Is there something I can do with the EF context to get rid of loaded entities?
private void ProcessReport(ZipArchive zip, int page, int pageSize)
{
using (var context = new DBContext(_contextOptions))
{
var batch = GetDataFromIndex(page, pageSize, context).ToArray();
if (!batch.Any())
{
return;
}
var file = zip.CreateEntry("file_" + page + ".csv");
using (var entryStream = file.Open())
using (var streamWriter = new StreamWriter(entryStream))
{
foreach (var reading in batch)
{
try
{
streamWriter.WriteLine("write data from record here.")
}
catch (Exception e)
{
//handle error
}
}
}
batch = null;
}
ProcessReport(zip, page + 1, pageSize);
}
private IEnumerable<Reading> GetDataFromIndex(int page, int pageSize, DBContext context)
{
var batches = (from rb in context.Reading.AsNoTracking()
//Some joins
select rb)
.Skip((page - 1) * pageSize)
.Take(pageSize);
return batches
.Includes(x => x.Something)
}