Wikipedia user agent problem when downloading images

user_1432808 0
Viewed 348

I am trying to download about 250 images from wikipedia with a c# .net console application. After downloading 3 I get this error.

System.Net.WebException: 'The remote server returned an error: (403) Forbidden. Please comply with the User-Agent policy: https://meta.wikimedia.org/wiki/User-Agent_policy. '

I have read their User-Agent_policy page and added a user agent that complies with what they say. (to the best of my ability, I'm not a web-dev)

They say, make it descriptive, include the word bot if its a bot, include contact details in parentheses, all of which I have done.

I am also waiting 5 seconds in between each image.. I just really really dont wanna download them by hand in my browser.

    static void DownloadImages()
    {
        var files = Directory.GetFiles(@"C:\projects\CarnivoraData", "*", SearchOption.AllDirectories);
        var client = new WebClient();
        client.Headers.Add("User-Agent", "bot by <My Name> (<My email address>) I am downloading an image of each carnivoran once (less than 300 images) for educational purposes");

        foreach (var path in files)
        {
            //Console.WriteLine(path);
            //Console.WriteLine(File.ReadAllText(path));

            AnimalData data = JsonSerializer.Deserialize<AnimalData>(File.ReadAllText(path));
            client.DownloadFile("https:" + data.Imageurl,@"C:\projects\CarnivoraImages\"+ data.Name +Path.GetExtension(data.Imageurl));
            System.Threading.Thread.Sleep(5000);
        }
    }

Any suggestions?

wikipedia user-agent webclient .net c#
1 Answers

Ok I got this to work. I think they key was using httpclient to download the files instead of webclient, and using DefaultRequestHeaders.UserAgent.ParseAdd

    var httpClient = new HttpClient();
    httpClient.DefaultRequestHeaders.UserAgent.ParseAdd("<My Name>/1.0 (<My Email>) bot");

I didnt even bother waiting between images, downloaded them all in about a minute.

Also as a bonus, heres how to download a file using httpclient (its a lot messier than webclient!)

    static async Task GetFile(HttpClient httpClient,string filepath, string url)
    {
        using (var stream = await httpClient.GetStreamAsync(new Uri(url)))
        {
            using (var fileStream = new FileStream(filepath, FileMode.CreateNew))
            {
                await stream.CopyToAsync(fileStream);
            }
        }
    }
user_14328080
Related