Cannot set some HTTP headers when using System.Net.WebRequest

If you need the short and technical answer go right to the last section of the answer.

If you want to know better, read it all, and i hope you'll enjoy...

I countered this problem too today, and what i discovered today is that:

1. the above answers are true, as:

   1.1 it's telling you that the header you are trying to add already exist and you should then modify its value using the appropriate property (the indexer, for instance), instead of trying to add it again.

   1.2 Anytime you're changing the headers of an HttpWebRequest, you need to use the appropriate properties on the object itself, if they exist.

Thanks FOR and Jvenema for the leading guidelines...

2. But, What i found out, and that was the missing piece in the puzzle is that:

   2.1 The WebHeaderCollection class is generally accessed through WebRequest.Headers or WebResponse.Headers. Some common headers are considered restricted and are either exposed directly by the API (such as Content-Type) or protected by the system and cannot be changed.

The restricted headers are:

• Accept
• Connection
• Content-Length
• Content-Type
• Date
• Expect
• Host
• If-Modified-Since
• Range
• Referer
• Transfer-Encoding
• User-Agent
• Proxy-Connection

So, next time you are facing this exception and don't know how to solve this, remember that there are some restricted headers, and the solution is to modify their values using the appropriate property explicitly from the WebRequest/HttpWebRequest class.

Edit: (useful, from comments, comment by user Kaido)

Solution is to check if the header exists already or is restricted (WebHeaderCollection.IsRestricted(key)) before calling add

I ran into this problem with a custom web client. I think people may be getting confused because of multiple ways to do this. When using WebRequest.Create() you can cast to an HttpWebRequest and use the property to add or modify a header. When using a WebHeaderCollection you may use the .Add("referer","my_url").

Ex 1

WebClient client = new WebClient();
client.Headers.Add("referer", "http://stackoverflow.com");
client.Headers.Add("user-agent", "Mozilla/5.0");

Ex 2

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Referer = "http://stackoverflow.com";
request.UserAgent = "Mozilla/5.0";
response = (HttpWebResponse)request.GetResponse();

Anytime you're changing the headers of an HttpWebRequest, you need to use the appropriate properties on the object itself, if they exist. If you have a plain WebRequest, be sure to cast it to an HttpWebRequest first. Then Referrer in your case can be accessed via ((HttpWebRequest)request).Referrer, so you don't need to modify the header directly - just set the property to the right value. ContentLength, ContentType, UserAgent, etc, all need to be set this way.

IMHO, this is a shortcoming on MS part...setting the headers via Headers.Add() should automatically call the appropriate property behind the scenes, if that's what they want to do.

WebRequest being abstract (and since any inheriting class must override the Headers property).. which concrete WebRequest are you using ? In other words, how do you get that WebRequest object to beign with ?

ehr.. mnour answer made me realize that the error message you were getting is actually spot on: it's telling you that the header you are trying to add already exist and you should then modify its value using the appropriate property (the indexer, for instance), instead of trying to add it again. That's probably all you were looking for.

Other classes inheriting from WebRequest might have even better properties wrapping certain headers; See this post for instance.

Note: this solution will work with WebClientSocket as well as with HttpWebRequest or any other class that uses WebHeaderCollection to work with headers.

If you look at the source code of WebHeaderCollection.cs you will see that Hinfo is used to keep information of all known headers:

private static readonly HeaderInfoTable HInfo = new HeaderInfoTable();

Looking at HeaderInfoTable class, you can notice all the data is stored into hash table

private static Hashtable HeaderHashTable;

Further, in static contructor of HeaderInfoTable, you can see all known headers are added in HeaderInfo array and then copied into hashtable.

Final look at HeaderInfo class shows the names of the fields.

internal class HeaderInfo {

    internal readonly bool IsRequestRestricted;
    internal readonly bool IsResponseRestricted;
    internal readonly HeaderParser Parser;

    //
    // Note that the HeaderName field is not always valid, and should not
    // be used after initialization. In particular, the HeaderInfo returned
    // for an unknown header will not have the correct header name.
    //

    internal readonly string HeaderName;
    internal readonly bool AllowMultiValues;
    ...
    }

So, with all the above, here is a code that uses reflection to find static Hashtable in HeaderInfoTable class and then changes every request-restricted HeaderInfo inside hash table to be unrestricted

        // use reflection to remove IsRequestRestricted from headerInfo hash table
        Assembly a = typeof(HttpWebRequest).Assembly;
        foreach (FieldInfo f in a.GetType("System.Net.HeaderInfoTable").GetFields(BindingFlags.NonPublic | BindingFlags.Static))
        {
            if (f.Name == "HeaderHashTable")
            {
                Hashtable hashTable = f.GetValue(null) as Hashtable;
                foreach (string sKey in hashTable.Keys)
                {

                    object headerInfo = hashTable[sKey];
                    //Console.WriteLine(String.Format("{0}: {1}", sKey, hashTable[sKey]));
                    foreach (FieldInfo g in a.GetType("System.Net.HeaderInfo").GetFields(BindingFlags.NonPublic | BindingFlags.Instance))
                    {

                        if (g.Name == "IsRequestRestricted")
                        {
                            bool b = (bool)g.GetValue(headerInfo);
                            if (b)
                            {
                                g.SetValue(headerInfo, false);
                                Console.WriteLine(sKey + "." + g.Name + " changed to false");
                            }

                        }
                    }

                }
            }
        } 

Basically, no. That is an http header, so it is reasonable to cast to HttpWebRequest and set the .Referer (as you indicate in the question):

HttpWebRequest req = ...
req.Referer = "your url";

I ran into same issue below piece of code worked for me

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

request.Headers["UserAgent"] = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; 
Trident/5.0)"

request.Headers.UserAgent.Add(new ProductInfoHeaderValue("my_string"));