C# Webclient.DownloadData only returns garbage

Question

I have following code, to download the html source of a webadress, but when I run it I only get random characters and a lot of Questionmarks as an output.

The code:

 ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback(
                delegate
                {
                    return true;
                });
                using (WebClient webClient = new WebClient())
                {
                    webClient.Headers["User-Agent"] = "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 (.NET CLR 3.5.30729)";
                    webClient.Headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
                    webClient.Headers["Accept-Language"] = "en-us,en;q=0.5";
                    webClient.Headers["Accept-Encoding"] = "gzip,deflate";
                    webClient.Headers["Accept-Charset"] = "ISO-8859-1,utf-8;q=0.7,*;q=0.7";

                    var htmlData = webClient.DownloadData("https://de.WEBSITE.com/EXAMPLE");
                    var htmlCode = Encoding.UTF8.GetString(htmlData);
                    
                    Console.WriteLine(htmlCode);
                }

user_3567736 · Answer

It's just because you are downloading the gzip-compressed data. You should decompress data and then convert it to UTF-8.

webClient.Headers["User-Agent"] = "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 (.NET CLR 3.5.30729)";
webClient.Headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
webClient.Headers["Accept-Language"] = "en-us,en;q=0.5";
webClient.Headers["Accept-Encoding"] = "gzip,deflate";
webClient.Headers["Accept-Charset"] = "ISO-8859-1,utf-8;q=0.7,*;q=0.7";

var htmlData = webClient.DownloadData("https://de.WEBSITE.com/EXAMPLE");
using (var msi = new MemoryStream(htmlData))
using (var mso = new MemoryStream()) {
    using (var gs = new GZipStream(msi, CompressionMode.Decompress)) {
        gs.CopyTo(mso);
    }

    var htmlCode = Encoding.UTF8.GetString(mso.ToArray());
    Console.WriteLine(htmlCode);
}

C# Webclient.DownloadData only returns garbage

1 Answers