Blazor WASM: Secure a video stream - controller only accessible for authorized users

Info: I'm using the default Visual Studio (16.10.0 Preview 1.0) ASP.NET hosted Blazor WASM template with IdentityServer4 individual accounts.(ASP.NET Core 6 Preview 1)

I'd like the video stream only to be accessible by authorized users but [Authorize] attribute doesn't work here on the controller, I'm getting a Status Code: 401, while the user has logged in and thus has been authorized by the application. [Authorize] works with controllers for database access for example.

I think maybe it doesn't work with the video tag because the client accesses the controller directly from the video tag option: <source src="/VideoStream/@videoFileName" so no authorization headers are being sent. I've seen the same problem when devs want to protect/authorize downloading a file. I've read some blogs on that topic and tried some code snippets, but none of them work in my Blazor WASM app.

Without the [Authorize] attribute and if you know the path to the controller and the filename of the video (e.q. /VideoStream/test.mp4) anybody can access/download the video by just putting the link in the browser.

In this example the video isn't served from the (client side) wwwroot folder, but from a server side folder, see: var videoPathFile

Client side video tag

<video id="videostream" poster="@videoPoster" class="videostyle" oncontextmenu="return false;" controls disablePictureInPicture controlsList="nodownload">
    <source src="/VideoStream/@videoFileName" type="@ContentType;codecs=@Codecs" />
    Your browser does not support the video tag.naar een nieuwere versie.
</video>

/Server/Controllers/VideoStreamController.cs

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.FileProviders;
using Microsoft.Extensions.Logging;
using System.IO;

namespace Mediatheek.Server.Controllers
{
    //[ApiController] 
    [Route("[controller]")]
    public class VideoStreamController : ControllerBase
    {
        private readonly IWebHostEnvironment env;
        private readonly ILogger<VideoStreamController> logger;
        public VideoStreamController(ILogger<VideoStreamController> logger, IWebHostEnvironment env)
        {
            this.logger = logger;
            this.env = env;
        }

        //[Authorize] // Doesn't work here, even when [ApiController] is uncommented
        [HttpGet("{file}")]
        public IActionResult StreamVideo(string file)
        {
            var provider = new PhysicalFileProvider(env.ContentRootPath);
            var videoPathFile = Path.Combine(provider.Root, "Files", "Videos", $"{file}.mp4");
            var fileResponse = PhysicalFile(videoPathFile, "application/octet-stream");
            fileResponse.EnableRangeProcessing = true;
            fileResponse.FileDownloadName = "video.mp4"; // rename original filename, just for testing
            return fileResponse;
        }
    }
}