Variable generic return type in C#

Is there any way to have a method return any one of a number of generic types from a method? For example, I have the following:

public static T ParseAttributeValue<T>(this XElement element, string attribute)
    {
        if(typeof(T) == typeof(Int32))
        {
            return Int32.Parse(element.Attribute(attribute).Value);
        }

        if(typeof(T) == typeof(Double))
        {
            return Double.Parse(element.Attribute(attribute).Value);
        }

        if(typeof(T) == typeof(String))
        {
            return element.Attribute(attribute).Value;
        }

        if(typeof(T) == typeof(ItemLookupType))
        {
            return Enum.Parse(typeof(T), element.Attribute(attribute).Value);
        }
    }

(This is only a very quick mockup, I'm aware that any production code would need to be significantly more thorough in null checks etc...)

But the compiler doesn't like it, complaining that Int32 cannot be implicitly converted to T (it doesn't work with a cast either). I can understand that. At compile time it has no way to know what T is, but I'm checking it beforehand. Is there anyway I can make this work?