How do I sort associations (on field or custom field)

I've got the following code for automatically loading streams (dynamic product groups) with all products, but I would like to sort them based on a custom field.

public function search(Criteria $criteria, SalesChannelContext $salesChannelContext): EntitySearchResult
{
    $criteria->addAssociation('streams');

    return $this->decorated->search($criteria, $salesChannelContext);
}

How can I sort the results using $criteria (meaning through SQL), or do I have to sort it using standard PHP after getting the results?