Laravel Eloquent Nested Eager Load Apply Order By

Given this data:

{
    "current_page": 1,
    "data": [
      {
        "id": 1,
        "delivery_id": 1,
        "deliveries_sub": {
          "first_delivery_date": "08/31/2022",
          "delivery" {
            "day": "Monday",
            "location": {
              "direction": "North"
            }
          }
        }
      },
      {
        "id": 2,
        "delivery_id": 2,
        "deliveries_sub": {
          "first_delivery_date": "09/28/2022",
          "delivery" {
            "day": "Friday",
            "location": {
              "direction": "South"
            }
          }
        }
      },
      {
        "id": 3,
        "delivery_id": 2,
        "deliveries_sub": {
          "first_delivery_date": "08/31/2022",
          "delivery" {
            "day": "Wednesday",
            "location": {
              "direction": "Northeast"
            }
          }
        }
      },
      {
        "id": 4,
        "delivery_id": 3,
        "deliveries_sub": {
          "first_delivery_date": "09/02/2022",
          "delivery" {
            "day": "Tueday",
            "location": {
              "direction": "North"
            }
          }
        }
      }
    ],
    "first_page_url": "http://localhost/mypage/list?page=1",
    "from": 1,
    "last_page": 1,
    "last_page_url": "http://localhost/mypage/list?page=1",
    "links": [
      {
        "url": null,
        "label": "« Previous",
        "active": false
      },
      {
        "url": "http://localhost/mypage/list?page=1",
        "label": "1",
        "active": true
      },
      {
        "url": null,
        "label": "Next »",
        "active": false
      }
    ],
    "next_page_url": null,
    "path": "http://localhost/mypage/list",
    "per_page": 10,
    "prev_page_url": null,
    "to": 4,
    "total": 4
  }

I would like to sort it by the first_delivery_date. I have fetched this one using:

DeliveryDays::with('deliverySub.delivery.location')
->orderBy('deliverySub.first_delivery_date')
->paginate(10);

It seems that the orderBy clause is not working. I've also tried this approach:

DeliveryDays::with(['deliverySub' => function ($query) {
  $query->orderBy('first_delivery_date');
}, 'deliverySub.delivery.location']) 
->paginate(10);

But it also doesn't work. My desired result would be the sorted first_delivery_date data:

{
        "current_page": 1,
        "data": [
          {
            "id": 1,
            "delivery_id": 1,
            "deliveries_sub": {
              "first_delivery_date": "08/31/2022",
              "delivery" {
                "day": "Monday",
                "location": {
                  "direction": "North"
                }
              }
            }
          },
          {
            "id": 3,
            "delivery_id": 2,
            "deliveries_sub": {
              "first_delivery_date": "08/31/2022",
              "delivery" {
                "day": "Wednesday",
                "location": {
                  "direction": "Northeast"
                }
              }
            }
          },
          {
            "id": 4,
            "delivery_id": 3,
            "deliveries_sub": {
              "first_delivery_date": "09/02/2022",
              "delivery" {
                "day": "Tueday",
                "location": {
                  "direction": "North"
                }
              }
            }
          },
          {
            "id": 2,
            "delivery_id": 2,
            "deliveries_sub": {
              "first_delivery_date": "09/28/2022",
              "delivery" {
                "day": "Friday",
                "location": {
                  "direction": "South"
                }
              }
            }
          }
        ],
        "first_page_url": "http://localhost/mypage/list?page=1",
        "from": 1,
        "last_page": 1,
        "last_page_url": "http://localhost/mypage/list?page=1",
        "links": [
          {
            "url": null,
            "label": "« Previous",
            "active": false
          },
          {
            "url": "http://localhost/mypage/list?page=1",
            "label": "1",
            "active": true
          },
          {
            "url": null,
            "label": "Next »",
            "active": false
          }
        ],
        "next_page_url": null,
        "path": "http://localhost/mypage/list",
        "per_page": 10,
        "prev_page_url": null,
        "to": 4,
        "total": 4
      }

What would be the best approach for sorting the results? Is it within the eager load or after fetching the results? Thanks.