Use HTACCESS to block all referrers except one domain

I want to block all traffic to my website except referrers from a specific domain. Here is what I have. It works for http connections but not working for https connections. Is there something that needs to be done differently if the referrer is an https?

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http?://www\.example\.com/ [NC]
RewriteCond %{HTTP_REFERER} !^https?://www\.example\.com/ [NC]
RewriteRule .* - [F]