FetchXML Aggregate Sum Issue

I have a requirement to sum/aggregate total hours entered by day for the week for each user. Each user can have multiple entries for each day depending on which task(s) they are entering hours for. The data looks similar to this (simplified)

Table 1 (SystemUser)

Table 2 (TimeEntry)

My results should look like this:

My fetch looks like this currently (I have additional code to figure out which day of the week it is to build the dates dynamically based on the workweek). I start with SystemUser and do outer joins to TimeEntry to make sure if they didn't put in any time for a day, I still get the user back.

<fetch aggregate="true">
<entity name="systemuser" >
    <attribute name="fullname" alias="FullName" groupby="true" />
    <link-entity name="timeentry" from="ownerid" to="systemuserid" link-type="outer" alias="TimeEntriesMonday" >
        <attribute name="duration" alias="mondayTotal" aggregate="sum" />
        <filter>
            <condition attribute="timeentrydate" operator="on" value="2022-09-05" />
        </filter>
    </link-entity>
    <link-entity name="timeentry" from="ownerid" to="systemuserid" link-type="outer" alias="TimeEntriesTuesday" >
        <attribute name="duration" alias="tuesdayTotal" aggregate="sum" />
        <filter>
            <condition attribute="timeentrydate" operator="on" value="2022-09-06" />
        </filter>
    </link-entity>
</entity>

What I find is it doubles the amounts (and so on as I add in the rest of the week).

I have tried various things, such as "distinct" which doesn't work as if a user enters two entries of "4" hours for a given day it will only return one. I've also tried using inner joins but obviously that doesn't work.

I thought providing a table alias every time I join it along with a filter condition for the date would handle this correctly.