In ASP.NET, how do I add a CORS header for only TTF files?

I have a page with an iframe that shows an external page. The external page is configured to download a CSS file from my server.

In the CSS, I added a @font-face selector:

@font-face {
    font-family: "Special Font";
    src: url("<%= Request.Url.GetLeftPart(UriPartial.Authority) + "/fonts/specialfont.ttf" %>");
}

This downloads and shows the font fine in Chrome, but in Firefox, it downloads the font, but refuses to use it. Doing a little bit of research shows that this problem is a cross-origin policy issue. One of the solutions mentioned here:

http://enable-cors.org/

Is to enable the CORS header. However, the solution it provided is site-wide:

<?xml version="1.0" encoding="utf-8"?>
    <configuration>
        <system.webServer>
            <httpProtocol>
                <customHeaders>
                    <add name="Access-Control-Allow-Origin" value="*" />
                </customHeaders>
            </httpProtocol>
        </system.webServer>
</configuration>

Whereas I would only like to enable it for only .TTF files. Is there a way to do this, either through using a HttpHandler or some other method?