Queryset filter: retrieve manytomany field as list of each object

I want to retrieve every object of a model with only their id and a list of ids they have in a many to many field.

My models :

class Wordlist(models.Model):
  words = models.ManyToManyField(Word)


class Word(models.Model):
  word = models.CharField(max_length=256)

I have this code :

list(Wordlist.objects.all().annotate(w=F('words')).values_list('pk', 'w'))

And it gives me this :

[{'pk': 1, 'w': 7},
 {'pk': 1, 'w': 13},
 {'pk': 1, 'w': 17},
 {'pk': 2, 'w': 29},
 {'pk': 1, 'w': 42},
 {'pk': 3, 'w': 52},
 {'pk': 2, 'w': 65}
 ...
]

What i want is :

[{'pk': 1, 'w': [7, 13, 17, 42,...]},
 {'pk': 2, 'w': [29, 65,...]},
 {'pk': 3, 'w': [52,...]},
 ...
]

A simple solution would be to combine the dicts based on their ids but I don't think it's a good practice and very efficient, since we could have dozens of thousands dicts as a result.

Also, I wondered if it was possible to do the opposite with a single request; retrieving a list of Wordlist a word is in for each word in a request on Word.