Using $watch to update data, ng-repeat not reflecting changes

I have a list of fooditems in $scope.raw and I want to show this data in columns so I'm changing the structure a bit. I do this in the sortStuff() function and store the updated data in $scope.allfood. There's a $watch that calls sortStuff() every time anything changes in $scope.raw (I'm using drag and drop to change the food category):

$scope.$watch('raw', function(){
    $scope.allfood = $scope.sortStuff();
    console.log($scope.allfood);
}, true);

This is what happens when food is dragged around:

receive:function(event, ui) {
    var issueScope = angular.element(ui.item).scope();
    scope.$apply(function() {
        var recp = _.find(scope.raw, function(lineitem){
            return lineitem.name === issueScope.receipe.name;
        })
        recp.cat = scope.col.name;
    })

    $(ui.item).remove(); // remove DOM
}

Basically, I search for the right object inside $scope.raw and change cat to new category for the food. I also delete the dom element because I'm counting on ng-repeat to refresh the view. This seems to work fine: console.log inside $watch shows that the object is being moved to the right category and the data looks what it should look like. However, visually, ng-repeat doesn't reflect the data.

Here's the jsfiddle.

Dragging an item from B to C works fine. Dragging one from A to B, makes two items from B disappear... the results are very inconsistent and I have no idea what is happening.

Any ideas what is going wrong? Or maybe any suggestions for a better way to do this?