How to calculate the average duration between two DateTimes and group the average by Date in separate list in C#

So, I have the following Model structure:

public class MyAverageData
{
    public DateTime startDate { get; set; }
    public DateTime? endDate { get; set; }
}

And I populate this data from another list like this:

var myAvgList = from x in myBiglist
                where x.filter == "Required"
                select new MyAverageData
                {
                   startDate = x.startDate,
                   endDate = x.endDate
                };

var noNulls = myAvgList.Where(x => x.endDate.HasValue);

The noNulls list looks like this:

[0]
endDate: {18/05/2021 5:32:12 PM}
startDate: {24/06/2020 9:34:16 PM}

[1]
endDate: {18/05/2021 5:32:12 PM}
startDate: {24/06/2020 9:28:57 PM}

[2]
endDate: {18/05/2021 5:32:12 PM}
startDate: {24/06/2020 9:22:35 PM}

[3]
endDate: {18/05/2021 5:32:12 PM}
startDate: {23/06/2020 5:44:01 PM}

[4]
endDate: {18/05/2021 5:32:12 PM}
startDate: {23/06/2020 5:36:44 PM}

I am attempting to find the average time between the endDate and startDate like this:

if(noNulls.IsAny())
{
    var avgTime = noNulls.Average(x => (x.endDate.Value - x.startDate).TotalMilliseconds);
}

This gives the average of the entire noNulls list. How can I find the average duration between two dates and then group them based on the date. Basically I want a list (based on the above data) that would like something like this:

[0]
Date: {23/06/2020}
AverageTime: {2500}

[1]
Date: {24/06/2020}
AverageTime: {5500}

How can I achieve this?