Logging raw HTTP request/response in ASP.NET MVC & IIS7

Question

I'm writing a web service (using ASP.NET MVC) and for support purposes we'd like to be able to log the requests and response in as close as possible to the raw, on-the-wire format (i.e including HTTP method, path, all headers, and the body) into a database.

What I'm not sure of is how to get hold of this data in the least 'mangled' way. I can re-constitute what I believe the request looks like by inspecting all the properties of the HttpRequest object and building a string from them (and similarly for the response) but I'd really like to get hold of the actual request/response data that's sent on the wire.

I'm happy to use any interception mechanism such as filters, modules, etc. and the solution can be specific to IIS7. However, I'd prefer to keep it in managed code only.

Any recommendations?

Edit: I note that HttpRequest has a SaveAs method which can save the request to disk but this reconstructs the request from the internal state using a load of internal helper methods that cannot be accessed publicly (quite why this doesn't allow saving to a user-provided stream I don't know). So it's starting to look like I'll have to do my best to reconstruct the request/response text from the objects... groan.

Edit 2: Please note that I said the whole request including method, path, headers etc. The current responses only look at the body streams which does not include this information.

Edit 3: Does nobody read questions around here? Five answers so far and yet not one even hints at a way to get the whole raw on-the-wire request. Yes, I know I can capture the output streams and the headers and the URL and all that stuff from the request object. I already said that in the question, see:

I can re-constitute what I believe the request looks like by inspecting all the properties of the HttpRequest object and building a string from them (and similarly for the response) but I'd really like to get hold of the actual request/response data that's sent on the wire.

If you know the complete raw data (including headers, url, http method, etc.) simply cannot be retrieved then that would be useful to know. Similarly if you know how to get it all in the raw format (yes, I still mean including headers, url, http method, etc.) without having to reconstruct it, which is what I asked, then that would be very useful. But telling me that I can reconstruct it from the HttpRequest/HttpResponse objects is not useful. I know that. I already said it.

Please note: Before anybody starts saying this is a bad idea, or will limit scalability, etc., we'll also be implementing throttling, sequential delivery, and anti-replay mechanisms in a distributed environment, so database logging is required anyway. I'm not looking for a discussion of whether this is a good idea, I'm looking for how it can be done.

user_450159 · Answer

The following extension method on HttpRequest will create a string that can be pasted into fiddler and replayed.

namespace System.Web
{
    using System.IO;

    /// 
    /// Extension methods for HTTP Request.
    /// 
    /// See the HTTP 1.1 specification http://www.w3.org/Protocols/rfc2616/rfc2616.html
    /// for details of implementation decisions.
    /// 
    /// 
    public static class HttpRequestExtensions
    {
        /// 
        /// Dump the raw http request to a string. 
        /// 
        /// The  that should be dumped.       
        /// The raw HTTP request.
        public static string ToRaw(this HttpRequest request)
        {
            StringWriter writer = new StringWriter();

            WriteStartLine(request, writer);
            WriteHeaders(request, writer);
            WriteBody(request, writer);

            return writer.ToString();
        }

        private static void WriteStartLine(HttpRequest request, StringWriter writer)
        {
            const string SPACE = " ";

            writer.Write(request.HttpMethod);
            writer.Write(SPACE + request.Url);
            writer.WriteLine(SPACE + request.ServerVariables["SERVER_PROTOCOL"]);
        }

        private static void WriteHeaders(HttpRequest request, StringWriter writer)
        {
            foreach (string key in request.Headers.AllKeys)
            {
                writer.WriteLine(string.Format("{0}: {1}", key, request.Headers[key]));
            }

            writer.WriteLine();
        }

        private static void WriteBody(HttpRequest request, StringWriter writer)
        {
            StreamReader reader = new StreamReader(request.InputStream);

            try
            {
                string body = reader.ReadToEnd();
                writer.WriteLine(body);
            }
            finally
            {
                reader.BaseStream.Position = 0;
            }
        }
    }
}

user_128809 · Answer

use a IHttpModule:

    namespace Intercepts
{
    class Interceptor : IHttpModule
    {
        private readonly InterceptorEngine engine = new InterceptorEngine();

        #region IHttpModule Members

        void IHttpModule.Dispose()
        {
        }

        void IHttpModule.Init(HttpApplication application)
        {
            application.EndRequest += new EventHandler(engine.Application_EndRequest);
        }
        #endregion
    }
}

    class InterceptorEngine
    {       
        internal void Application_EndRequest(object sender, EventArgs e)
        {
            HttpApplication application = (HttpApplication)sender;

            HttpResponse response = application.Context.Response;
            ProcessResponse(response.OutputStream);
        }

        private void ProcessResponse(Stream stream)
        {
            Log("Hello");
            StreamReader sr = new StreamReader(stream);
            string content = sr.ReadToEnd();
            Log(content);
        }

        private void Log(string line)
        {
            Debugger.Log(0, null, String.Format("{0}
", line));
        }
    }

user_112407 · Answer

HttpRequest and HttpResponse pre MVC used to have a GetInputStream() and GetOutputStream() that could be used for that purpose. Haven't look into those part in MVC so Im not sure they are availavle but might be an idea :)

user_125090 · Answer

Agree with FigmentEngine, IHttpModule appears to be the way to go.

Look into httpworkerrequest, readentitybody and GetPreloadedEntityBody.

To get the httpworkerrequest you need to do this:

(HttpWorkerRequest)inApp.Context.GetType().GetProperty("WorkerRequest", bindingFlags).GetValue(inApp.Context, null);

where inApp is the httpapplication object.

user_58155 · Answer

I know it's not managed code, but I'm going to suggest an ISAPI filter. It's been a couple of years since I've had the "pleasure" of maintaining my own ISAPI but from what I recall you can get access to all this stuff, both before and after ASP.Net has done it's thing.

http://msdn.microsoft.com/en-us/library/ms524610.aspx

If a HTTPModule isn't good enough for what you need, then I just don't think there is any managed way of doing this in the required amount of detail. It's gonna be a pain to do though.

user_26226 · Answer

I agree with the others, use an IHttpModule. Take a look at the answer to this question, which does almost the same thing that you are asking. It logs the request and response, but without headers.

How to trace ScriptService WebService requests?

Logging raw HTTP request/response in ASP.NET MVC & IIS7

15 Answers