Elasticsearch Nested 2 Step Sorting

Given the following data with nested objects (members within teams), I need to do a 2 step sort:

1. Return the youngest member of each team.
2. Sort the teams by the name of that youngest member.

I have a query below that is close: it does get the youngest member of each team, but then it sorts the teams using the names of all the members, not just the one selected per team.

What would the query be to do this?
And would such a query be performant assuming there was a lot of data? (Probably a few million objects each having 1-3 nested objects.)

Note: Although it's not clear in this simple example, I cannot simply store the youngest member, since in my real world case, the sorting of the nested objects is determined by a formula that includes an external parameter. This is just a very simplified example of the many sorts like this I would have to do on a larger data set, where I need to get the single best matching nested document for each outer document sorted in one way, but then sort the outer objects based on some other property of that selected nested object.

Data

PUT nested_test
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "team": { "type": "keyword", "index": true, "doc_values": true },
      "members": {
        "type": "nested",
        "properties": {
          "name": { "type": "keyword", "index": true, "doc_values": true },
          "age": { "type": "integer", "index": true, "doc_values": true}
        }
      }
    }
  }
}
 
PUT nested_test/_doc/1
{
    "team" : "A" ,
    "members" :
      [
        { "name" : "Curt" , "age" : "34" } ,
        { "name" : "Dave" , "age" : "33" }
      ]
}
 
PUT nested_test/_doc/2
{
    "team" : "B" ,
    "members" :
      [
        { "name" : "Alex" , "age" : "36" } ,
        { "name" : "Earl" , "age" : "32" }
      ]
}

PUT nested_test/_doc/3
{
    "team" : "C" ,
    "members" :
      [
        { "name" : "Brad" , "age" : "35" } ,
        { "name" : "Gary" , "age" : "31" }
      ]
}

Attempted Query

GET nested_test/_search?filter_path=hits.hits._source.team,hits.hits.sort.*,hits.hits.inner_hits.members.hits.hits._source.*,hits.hits.inner_hits.members.hits.hits.sort.*
{
  "query": {
    "bool": {
      "filter": [
        {
          "nested": {
            "path": "members",
            "query": {
                 "match_all" : { } 
            } ,
            "inner_hits": {
              "size": 1,
              "sort": {
                "members.age": { "order": "asc" }
              }
            }         
          }
        }
      ]
    }
  }
  ,
  "sort": [
    { "members.name": {
        "order": "asc" ,
        "nested": {
          "path": "members",
          "filter": { "match_all" : { } }
        }
    } } 
  ]
}

Results (If the query was correct, the teams would be in A, B, C order, but they are B, C, A)

{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "team" : "B"
        },
        "inner_hits" : {
          "members" : {
            "hits" : {
              "hits" : [
                {
                  "_source" : {
                    "name" : "Earl",
                    "age" : "32"
                  }
                }
              ]
            }
          }
        }
      },
      {
        "_source" : {
          "team" : "C"
        },
        "inner_hits" : {
          "members" : {
            "hits" : {
              "hits" : [
                {
                  "_source" : {
                    "name" : "Gary",
                    "age" : "31"
                  }
                }
              ]
            }
          }
        }
      },
      {
        "_source" : {
          "team" : "A"
        },
        "inner_hits" : {
          "members" : {
            "hits" : {
              "hits" : [
                {
                  "_source" : {
                    "name" : "Dave",
                    "age" : "33"
                  }
                }
              ]
            }
          }
        }
      }
    ]
  }
}