After upgrade to MySQL 8, JSON large queries fail

After upgrading to MySQL 8 w/ the Ubuntu 20.04 distribution upgrade, any MySQL JSON data type queries returning a large JSON data payload are failing when using primary keys to sort/order. The Java SQL exception chain I get starts with a Out of sort memory exception. I set the sort buffer to 1024K but do not know if I should increase my sort buffer much based on things I have read saying setting the sort buffer large can cause performance issues. I am doing minor sorting on data using a primary key, and it's still causing the sort buffer error. I can remove the sort by Date clause and it works, but it's not the order I want the data in. Is it because I need to create a Primary key in the subquery I'm creating somehow?

java.sql.SQLException: Out of sort memory, consider increasing server sort buffer size

The query I am trying to run is as follows:

SELECT
                Date, Type, GeoJSON FROM (
                SELECT Date, 'R' AS Type, RunGeoJSON as GeoJSON FROM Core.Fitness WHERE RunGeoJSON IS NOT NULL AND (CommonRoute = 0 OR CommonRoute IS NULL) UNION ALL
                SELECT Date, 'C' AS Type, CycGeoJSON as GeoJSON FROM Core.Fitness WHERE CycGeoJSON IS NOT NULL AND (CommonRoute = 0 OR CommonRoute IS NULL) UNION ALL
                SELECT Date, 'A' AS Type, AltGeoJSON as GeoJSON FROM Core.Fitness WHERE AltGeoJSON IS NOT NULL AND (CommonRoute = 0 OR CommonRoute IS NULL)) as tmp
                ORDER BY Date DESC; 

But if I simply just remove the order by, like this, I don't get the sort buffer error:

SELECT
                Date, Type, GeoJSON FROM (
                SELECT Date, 'R' AS Type, RunGeoJSON as GeoJSON FROM Core.Fitness WHERE RunGeoJSON IS NOT NULL AND (CommonRoute = 0 OR CommonRoute IS NULL) UNION ALL
                SELECT Date, 'C' AS Type, CycGeoJSON as GeoJSON FROM Core.Fitness WHERE CycGeoJSON IS NOT NULL AND (CommonRoute = 0 OR CommonRoute IS NULL) UNION ALL
                SELECT Date, 'A' AS Type, AltGeoJSON as GeoJSON FROM Core.Fitness WHERE AltGeoJSON IS NOT NULL AND (CommonRoute = 0 OR CommonRoute IS NULL)) as tmp;

Again, Date is a primary key (which I am ordering by):

describe Fitness;
+------------------+---------------+------+-----+---------+-------+
| Field            | Type          | Null | Key | Default | Extra |
+------------------+---------------+------+-----+---------+-------+
| Date             | date          | NO   | PRI | NULL    |       |