How to deserialize a field that's either a string or an array of elements?

I have some xml documents that contain a section that EITHER looks like this:

<MY_FIELD>
    <P>Some text I want here</P>
</MY_FIELD>

OR, it looks like this:

<MY_FIELD>Some text I want here</MY_FIELD>

For the latter, which is about 90% of the occurrences, this works:

[XmlElement("MY_FIELD")]
public string MyField { get; set; }

For the former, this seems to work:

[XmlArray("MY_FIELD")]
[XmlArrayItem("P")]
public string[] MyFieldPs { get; set; }

However, I can't figure out how to write something that gets me both. If I try to reference MY_FIELD twice (and thus have both MyField and MyFieldPs in my class), then I get the error

The XML element 'MY_FIELD' from namespace '' is already present in the current scope.

Is there a way for both to work at the same time?