Remove item from PopupMenuButton while it is open

I have a PopupMenuButton that displays some PopupMenuItem<String>'s generated from a List<String>. Each item has a delete button, which removes the String from the list. The problem is that the popup menu doesn't get rebuilt after deleting an item, until it's closed and opened again.

It seems that no matter what I do, even using a GlobalKey and calling key.currentState.setState(), it doesn't cause the popup menu to be rebuilt until it's closed and opened again.

  GlobalKey _favoritesKey = new GlobalKey();

  PopupMenuButton<String>(
      key: _favoritesKey,
      icon: Icon(Icons.bookmark_border),
      itemBuilder: (context){
          List<PopupMenuItem<String>> result = [];
          model.favorites.forEach((x){
              result.add(PopupMenuItem<String>(value: x, child: Row(
                  children: [
                      IconButton(icon: Icon(Icons.delete_outline), onPressed: (){
                          model.removeFavorite(x);
                          _favoritesKey.currentState?.setState((){});
                          setState(() {});
                      }),
                      Text(x)
                  ]
              )));
          });
          return result;
      },
      onSelected: (x){
          // Do something with the selected value
      },
  )

How can I make the popup menu rebuild itself while it is opened?