Download only files with specified extensions from Azure Git repository

Viewed 74

I need to download files with specified extensions from Azure Git repository programmatically (C#, .NET Framework 4.8). Files are located in different server folders and sometimes in different branches. I was able to achieve this goal with the following code:

var connection = new VssConnection(new Uri(collectionUri), new VssBasicCredential("", personalAccessToken));
using (var ghc = connection.GetClient<GitHttpClient>())
{
    string[] extensionsToDownload = { "xml", "xslt" }; // just for example, real cases are not limited to these extensions
    string branch = "my-branch-name";
    GitVersionDescriptor version = new GitVersionDescriptor { Version = branch };

    // Get all items information
    GitItemRequestData data = new GitItemRequestData
    {
        ItemDescriptors = new []
        {
            new GitItemDescriptor
            {
                Path = "/Some/Path",
                RecursionLevel = VersionControlRecursionType.Full,
                VersionType = GitVersionType.Branch,
                Version = branch
            },
            new GitItemDescriptor
            {
                Path = "/Another/Path",
                RecursionLevel = VersionControlRecursionType.Full,
                VersionType = GitVersionType.Branch,
                Version = branch
            }
        }
    };
    var items = ghc.GetItemsBatchAsync(data, project: projectName, repositoryId: repoName).Result;
    
    // filter returned items by extension
    List<GitItem> filteredItems = items
        .SelectMany(item => item)
        .Where(item => item.GitObjectType == GitObjectType.Blob && extensionsToDownload.Contains(item.Path.Split('.').Last())).ToList();
    
    // download zipped items and extract
    foreach (var item in filteredItems)
    {
        using (var stream = ghc.GetItemZipAsync(
            project: projectName, repositoryId: repoName, path: item.Path, includeContent: true, versionDescriptor: version).Result)
        {
            ZipArchive archive = new ZipArchive(stream);
            foreach (ZipArchiveEntry entry in archive.Entries)
            {
                entry.ExtractToFile(Path.Combine(localFolder, entry.FullName.Trim('/')), true);
            }
        }
    }
    

However, it means that each item requires a separate API call. Definitely not good for performance. I thought "There should be a way to batch download all items at once".

GetBlobsZipAsync method seemed like what I exactly needed. However, attempt to use it had failed miserably. All I got was VssUnauthorizedException: 'VS30063: You are not authorized to access https://dev.azure.com'. Very strange because calling GetBlobZipAsync for each individual item id works perfectly (but it's almost the same as initial solution with the same far-from-ideal performance).

Dictionary<string, string> idToNameMappings = filteredItems.ToDictionary(k => k.ObjectId, v => Path.Combine(localFolder, v.Path.Trim('/')));

foreach (var item in filteredItems)
{
    using (var stream = ghc.GetBlobsZipAsync(idToNameMappings.Select(i => i.Key), project: projectName, repositoryId: repoName).Result)
    {
        ZipArchive archive = new ZipArchive(stream);
        foreach (ZipArchiveEntry entry in archive.Entries)
        {
            entry.ExtractToFile(entry.FullName, idToNameMappings[entry.FullName], true);
        }
    }
}

Another option is to download all items as zip archive and filter it on the client side:

foreach (var desc in data.ItemDescriptors)
{
    using (var stream = ghc.GetItemZipAsync(projectName, repoName, null, desc.Path, desc.RecursionLevel, versionDescriptor:version).Result)
    {
        ZipArchive archive = new ZipArchive(stream);
        foreach (ZipArchiveEntry entry in archive.Entries)
        {
            if (extensionsToDownload.Contains(entry.FullName.Split('.').Last()))
            {
                entry.ExtractToFile(Path.Combine(localFolder, entry.FullName.Trim('/')), true);
            }
        }
    }
}

But it's even worse because the repository contains a large amount of data files (including some binary content). Downloading several hundreds MB of data to get less than 10 MB of xml files doesn't seem to be very efficient.

So at the moment I gave up and decided to stick with initial solution. But maybe there's something I overlooked?

0 Answers
Related