Properly sorting multidimensional array using usort and spaceship operator

I need to sort the following array with following logic: If the 'score' is the same, I want to compare using 'time'. The array is as follows:

$user_scores = [ 82 => [ 'score' => 1, 'time' => 6.442 ],
                 34 => [ 'score' => 1, 'time' => 5.646 ],
                 66 => [ 'score' => 3, 'time' => 1.554 ]
               ]

The 'keys' in the above array are the 'user_ids', which I need to preserve in the sorted array. My best attempt so far is as below -

$result = usort( $user_scores, function ( $a, $b ) {
    if ( $a['score'] == $b['score'] ) {
        return $a['time'] == $b['time'];
        }
    return $a['score'] <=> $b['score'];
} );

Obviously, this isn't working, and I'm getting the $user_scores array with all keys replaced with 0, 1, 2 instead of user_ids ( 82, 34 and 66 ). The sorting ain't working either.

For the above array, my desired output would be $user_scores array :

$user_scores = [ 34 => [ 'score' => 1, 'time' => 5.646 ],
                 82 => [ 'score' => 1, 'time' => 6.442 ],
                 66 => [ 'score' => 3, 'time' => 1.554 ]
               ]

Would really appreciate if you could tell me how to make this work using the spaceship operator (if it makes sense). Thank you for your time and I look forward to your responses.

---UPDATE ---

The sorting logic required is like this:

1. Higher the score, higher would be the rank.
2. Higher the time, lower would be the rank.

It's basically sorting the results of quiz. The top scorers with the least amount of time would be at the top; and those with lower score and higher time would be at the bottom.