Eloquent: order by max difference?

I have a table stats like this:

id | post_id | num_yes | num_no
1  | 1       | 2       | 0
2  | 1       | 3       | 3
3  | 1       | 1       | 0

How can I order by the difference num_yes - num_no? The output should look be 1, 3, 2 (in terms of id).

I have this currently:

$posts = Post::join('stats', 'stats.id', '=', 'posts.id')
    ->where('post_id', 1)
    ->orderByRaw('MAX(stats.num_yes - stats.num_no)', 'DESC')
    ->get();

But this isn't working. How can I fix this?