C# API - Provide download of files from Azure Blobstorage

I am currently working on a problem I've encountered while using Azure Blob Storage together with C# API. I also didn't find a fitting solution in the questions here since most of them just download files once and they're done.

What I want to achieve is to have an API as a proxy for handling file downloads for my mobile clients. Therefore I need fast response / fast first byte responses since the mobile applications have a rather low timeout of five seconds.

    [HttpGet, Route("{id}")]
    [Authorize(Policy = xxxxx)]
    public async Task<FileStreamResult> Get(Guid tenantId, Guid id)
    {
        if (tenantId == default)
        {
            throw new ArgumentException($"Tenant id '{tenantId}' is not valid.");
        }

        if (id == default)
        {
            throw new ArgumentException($"Package id '{id}' is not valid.");
        }

        var assetPackage = await _assetPackageService.ReadPackage(myenum.myvalue, tenantId, id).ConfigureAwait(false);
        if (assetPackage == null)
        {
            return File(new MemoryStream(), "application/octet-stream");
        }
        return File(assetPackage.FileStream, assetPackage.ContentType);
    }

        public async Task<AssetPackage> ReadPackage(AssetPackageContent packageContent, Guid tenantId, Guid packageId)
        {
            var blobRepository = await _blobRepositoryFactory.CreateAsync(_settings, tenantId.ToString())
                .ConfigureAwait(false);

            var blobPath = string.Empty;

            //some missing irrelevant code

            var blobReference = await blobRepository.ReadBlobReference(blobPath).ConfigureAwait(false);

            if (blobReference == null)
            {
                return null;
            }

            var stream = new MemoryStream();
            await blobReference.DownloadToStreamAsync(stream).ConfigureAwait(false);
            stream.Seek(0, SeekOrigin.Begin);
            return new AssetPackage(packageContent, stream, blobReference.Properties.ContentType);
        }

I am aware that MemoryStream is terrible for downloading and stuff since it consumes the files into memory before distributing it to the client.

How would you tackle this? Is there a easy solution to have my API act as a proxy rather than downloading the whole file and then let the client download it again from my api?