So after some extensive poking around the DB, I have managed to hack together a solution. I couldn't find a way to do it through the UI, but there are two key things that have to be done in the DB to make it work.
First, you need to enable the feature, which is enabled in the repository settings. While I couldn't find a setting in the UI, if you adjust anything from the default settings on the repo, a JSON object is saved to the dbo.tbl_PolicyConfigurationRevision table which includes an "optimizedByDefault" element which will be null, change it from null to true and the feature will now be enabled on the repo in question.
Alternately, this can be accomplished using the Policy Configuration endpoint for the TFS Rest API as described here. It's a bit more involved and still involves tweaking the JSON, but it goes through officially exposed channels without requiring direct DB manipulation and will properly version the configuration change.
Second, you need to specify what the "important" refs are. There is a stored proc and a custom data type to help with this. dbo.prc_UpdateGitLimitedRefCriteria takes the partitionId, dataspaceId, repositoryId and two custom table data type records for the exactRefs matches and the namespaceRefs matches. Build up the tables with your important refs in them and call the stored proc to add them to the list.
It appears that this ends up going through the prc_QueryGitRefs stored proc for filtering the refs, so you can look in there if you need more detail about how to format them so that processing works correctly.
After a bit more experimentation, it appears that there is an undocumented API endpoint for limitedRefCriteria as well that supports getting and updating ref criteria.
/tfs/*collectionName*/_apis/git/repositories/*repositoryId*/limitedRefCriteria
Interestingly, once I knew the "optimized" keyword, it looks like using _optimized in the repo name where the article describes using _full will also work to reverse the feature. Ex:
https://.visualstudio.com/Project/_git/_optimized/Repo
If you don't have the feature on by default but use the _optimized path, it will return the filtered branches. This might also be a safer path for those that don't want to mess with tweaking the settings json and only want to add records via stored proc to the limited refs criteria.