Reading thousands of objects with EF Core FAST

Viewed 2793

I am reading 40,000 small objects / rows from SQLite with EF core, and it's taking 18 seconds, which is too long for my UWP app. When this happens CPU usage on a single core reaches 100%, but the disk reading speed is circa 1%.

var dataPoints =  _db.DataPoints.AsNoTracking().ToArray();

Without AsNoTracking() the time taken is even longer.

DataPoint is a small POCO with a few primitive properties. Total amount of data I am loading is 4.5 MB.

    public class DataPointDto
    {
        [Key]
        public ulong Id { get; set; }

        [Required]
        public DateTimeOffset TimeStamp { get; set; }

        [Required]
        public bool trueTime { get; set; }

        [Required]
        public double Value { get; set; }
   }

Question: Is there a better way of loading this many objects, or am I stuck with this level of performance?

Fun fact: x86 takes 11 seconds, x64 takes 18. 'Optimise code' shaves off a second. Using Async pushes execution time to 30 seconds.

3 Answers

Performance test on 26 million records (1 datetime, 1 double, 1 int), EF Core 3.1.5:

  • Anonymous types or tuples as suggested in the accepted answer = About 20 sec, 1.3GB RAM

  • Struct = About 15 sec, 0.8GB RAM

Related