rails group by and order by column

In my controller I'm getting all Extras grouped by Category:

def index
  @categories = Extra.all.group_by(&:category)
end

The result is something like an Array of Hashes:

{#<Category id:1, sort:2> => [#<Extra id:1>,#<Extra id:2],
 #<Category id:2, sort: 1> => [#<Extra id:3>,#<Extra id:4>]}

I want to sort by category "sort" column instead of id, which should look like this:

{#<Category id:2, sort:1> => [#<Extra id:3>,#<Extra id:4],
 #<Category id:1, sort: 2> => [#<Extra id:1>,#<Extra id:2>]}

When I try:

def index
  @categories = Extra.all.group_by(&:category).sort_by{|s| s[:sort]}
end

I get "no implicit conversion of Symbol into Integer". Is that because I used a symbol in the "sort_by"?