Deploying Azure Firewall IP Group changes fails with conflict

I am attempting to deploy an Azure Firewall with a Policy, a Rule and a set of IPGroups. When I deploy the ARM templates to start everything works.. Later If I want to change something in one of the IPGroups, and I try to deploy that IPGroup change, the Azure Deployment fails with a Status: Conflict with message:

{
    "status": "Failed",
    "error": {
        "code": "ResourceDeploymentFailure",
        "message": "The resource operation completed with terminal provisioning state 'Failed'."
    }
}

I've attempted to both manage the IPGroups distinctly in their own ARM Template, and place them in with the Azure Policy Rule Collection ARM Template with a DependsOn to see if deploying them all together would help, but either way we just get "Conflict".. I Guess I am wondering what is the appropriate way to update an IPGroup that is a part of a Firewall Network rule? If I can't simply update the IPGroup?

Here is an example of my full ARM Template for my Policy with the IPGroups..

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
      "firewallPolicyName": {
            "defaultValue": "[concat('onelucki-fw-parent-policy', uniqueString(resourceGroup().id))]",
            "type": "String"
        },
        "DevSubnets": {
            "defaultValue": "DevSubnets",
            "type": "String"
        },
        "AzureSubnets": {
            "defaultValue": "AzureSubnets",
            "type": "String"
        }
    },
    "variables": {
        "fwPolicyName": "[parameters('firewallPolicyName')]"
    },
    "resources": [
          {
            "type": "Microsoft.Network/ipGroups",
            "apiVersion": "2020-05-01",
            "name": "AzureSubnets",
            "location": "centralus",
            "tags": { "Zone": "MixedZones" },
            "properties": {
                "ipAddresses": [
                    "10.99.1.1"
                ]
            }
        },
        {
            "type": "Microsoft.Network/ipGroups",
            "apiVersion": "2020-05-01",
            "name": "DevSubnets",
            "location": "centralus",
            "tags": { "Zone": "Dev" },
            "properties": {
                "ipAddresses": [
                    "10.99.2.2"
                ]
            }
        },
        {
            "type": "Microsoft.Network/firewallPolicies",
            "apiVersion": "2020-11-01",
            "name": "[parameters('firewallPolicyName')]",
            "location": "centralus",
            "properties": {
                "sku": {
                    "tier": "Standard"
                },
                "threatIntelMode": "Alert"
            }
        },
        {
            "type": "Microsoft.Network/firewallPolicies/ruleCollectionGroups",
            "apiVersion": "2020-11-01",
            "name": "[concat(parameters('firewallPolicyName'), '/DefaultNetworkRuleCollectionGroup')]",
            "location": "westus",
            "dependsOn": [
                "[resourceId('Microsoft.Network/ipGroups', parameters('AzureSubnets'))]",
                "[resourceId('Microsoft.Network/ipGroups', parameters('DevSubnets'))]",
                "[resourceId('Microsoft.Network/firewallPolicies', parameters('firewallPolicyName'))]"
            ],
            "properties": {
                "priority": 200,
                "ruleCollections": [
                    {
                        "ruleCollectionType": "FirewallPolicyFilterRuleCollection",
                        "action": {
                            "type": "Allow"
                        },
                        "rules": [
                            {
                                "ruleType": "NetworkRule",
                                "name": "DemoRule",
                                "ipProtocols": [
                                    "TCP"
                                ],
                                "sourceAddresses": [],
                                "sourceIpGroups": [
                                    "/subscriptions/<subscriptionIDHere>/resourceGroups/onelucki-fw/providers/Microsoft.Network/ipGroups/DevSubnets"
                                ],
                                "destinationAddresses": [],
                                "destinationIpGroups": [
                                     "/subscriptions/<subscriptionIDHere>/resourceGroups/onelucki-fw/providers/Microsoft.Network/ipGroups/AzureSubnets"
                                ],
                                "destinationFqdns": [],
                                "destinationPorts": [
                                    "135",
                                    "445"
                                ]
                            }
                        ],
                        "name": "DemoDeployRuleCollection",
                        "priority": 1300
                    }
                ]
            }
        }
    ]
}