XDocument.Validate not catching all errors against XSD

user_982354 0
Viewed 1991

I have a really strange problem validating an XML document against a valid XSD using C# XDocument.Validate or XMLReaderSettings with required configurations. The problem is: When there are errors in the XML document, the validation process fails to catch all errors under certain conditions and I can't find a pattern for this anomoly.

Here is my XSD:

<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified"
     targetNamespace="http://www.somesite.com/somefolder/messages"
     xmlns:xs="http://www.w3.org/2001/XMLSchema">
   <xs:element name="Message">
    <xs:complexType>
     <xs:sequence>
      <xs:element name="Header">
         <xs:complexType>
          <xs:sequence>
           <xs:element name="MessageId" type="xs:string" />
           <xs:element name="MessageSource" type="xs:string" />
          </xs:sequence>
       </xs:complexType>
    </xs:element>
    <xs:element name="Body">
       <xs:complexType>
          <xs:sequence>
             <xs:element name="Abc001">
                <xs:complexType>
                   <xs:sequence>
                    <xs:element name="Abc002" type="xs:string" />
                    <xs:element name="Abc003" type="xs:string" minOccurs="0" />
                    <!--<xs:element name="Abc004" type="xs:string" />-->
                    <xs:element name="Abc004">
                       <xs:simpleType>
                         <xs:restriction base="xs:string">
                           <xs:maxLength value="200"/>
                         </xs:restriction>
                      </xs:simpleType>
                    </xs:element>
                      <xs:element name="Abc005">
                         <xs:complexType>
                            <xs:sequence>
                              <xs:element name="Abc006" type="xs:unsignedShort" />
                              <xs:element name="Abc007">
                                <xs:complexType>
                                  <xs:sequence>
                                    <xs:element name="Abc008" type="xs:string"/>
                                    <xs:element name="Abc009" type="xs:string" minOccurs="0"/>
                                    <xs:element name="Abc010" type="xs:string"/>
                                  </xs:sequence>
                                </xs:complexType>
                              </xs:element>
                              <xs:element name="Abc011" type="xs:date" />
                              <xs:element name="Abc012">
                                <xs:complexType>
                                  <xs:sequence>
                                    <xs:element name="Abc013" type="xs:string" />
                                    <xs:element name="Abc014" type="xs:string" />
                                  </xs:sequence>
                                </xs:complexType>
                              </xs:element>
                            </xs:sequence>
                         </xs:complexType>
                      </xs:element>
                   </xs:sequence>
                </xs:complexType>
             </xs:element>
          </xs:sequence>
       </xs:complexType>
    </xs:element>
   </xs:sequence>
  </xs:complexType>
 </xs:element>
</xs:schema>

And here is the XML document being validated against this XSD:

<?xml version="1.0" encoding="utf-8"?>
<Message xmlns="http://www.somesite.com/somefolder/messages">
 <Header>
  <MessageId>Lorem</MessageId>
  <MessageSource>Ipsum</MessageSource>
 </Header>
 <Body>
  <Abc001>
   <Abc002>dolor</Abc002>
   <Abc003>sit amet</Abc003>
   <Abc004>consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.</Abc004>
   <Abc005>
    <Abc006>1234</Abc006>
    <Abc007>
     <Abc008>Ut enim</Abc008>
     <Abc009>ad</Abc009>
     <Abc010>minim</Abc010>
    </Abc007>
    <Abc011>1982-10-17</Abc011>
    <Abc012>
     <Abc013>veniam</Abc013>
     <Abc014>nostrud</Abc014>
    </Abc012>
   </Abc005>
  </Abc001>
 </Body>
</Message>

Now, when I introduce some validation errors into the XML and validate it against the XSD, it does find all the errors as expected. Here is the error-prone xml (I have marked where the errors are introduced):

<?xml version="1.0" encoding="utf-8"?>
<Message xmlns="http://www.somesite.com/somefolder/messages">
 <Header>
  <MessageId>Lorem</MessageId>
  <MessageSource>Ipsum</MessageSource>
 </Header>
 <Body>
  <Abc001>
   <Abc002>dolor</Abc002>
   <Abc003>sit amet</Abc003>
   
   <!--The value for Abc004 is increased beyond the allowed 200 characters-->
   
   <Abc004>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</Abc004>
   <Abc005>
    <Abc006>1234</Abc006>
    <Abc007>
     <Abc008>Ut enim</Abc008>
     <ABC009>AD</ABC009>
     
     <!--<Abc010>minim</Abc010>  Required element removed-->
    </Abc007>
    
    <!--Date formate below is wrong-->
    <Abc011>1982-10-37</Abc011>
    
    <Abc012>
     <Abc013>veniam</Abc013>
     <Abc014>nostrud</Abc014>
    </Abc012>
   </Abc005>

   <!--the element below is not allowed-->
   <Abc15>Not allowed</Abc15>
  </Abc001>
 </Body>
</Message>

and here is my resulting xml that shows all the errors:

<MessageResponse xmlns="http://www.somesite.com/somefolder/messages">
    <Result>false</Result>
    <Status>Failed</Status>
    <FaultCount>4</FaultCount>
    <Faults>
        <Fault>
            <FaultCode>ERR01</FaultCode>
            <FaultMessage>The 'http://www.somesite.com/somefolder/messages:Abc004' element is invalid - The value 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.' is invalid according to its datatype 'String' - The actual length is greater than the MaxLength value.</FaultMessage>
        </Fault>
        <Fault>
            <FaultCode>ERR02</FaultCode>
            <FaultMessage>The element 'Abc007' in namespace 'http://www.somesite.com/somefolder/messages' has invalid child element 'ABC009' in namespace 'http://www.somesite.com/somefolder/messages'. List of possible elements expected: 'Abc009, Abc010' in namespace 'http://www.somesite.com/somefolder/messages'.</FaultMessage>
        </Fault>
        <Fault>
            <FaultCode>ERR03</FaultCode>
            <FaultMessage>The 'http://www.somesite.com/somefolder/messages:Abc011' element is invalid - The value '1982-10-37' is invalid according to its datatype 'http://www.w3.org/2001/XMLSchema:date' - The string '1982-10-37' is not a valid Date value.</FaultMessage>
        </Fault>
        <Fault>
            <FaultCode>ERR04</FaultCode>
            <FaultMessage>The element 'Abc001' in namespace 'http://www.somesite.com/somefolder/messages' has invalid child element 'Abc15' in namespace 'http://www.somesite.com/somefolder/messages'.</FaultMessage>
        </Fault>
    </Faults>
</MessageResponse>

Here is the weird part. When I introduce one more error towards the beginning of the "Abc001" element, and also keep all the other existing errors, the result is totally messed up. Here is the XML with the newly introduced error:

<?xml version="1.0" encoding="utf-8"?>
<Message xmlns="http://www.somesite.com/somefolder/messages">
 <Header>
  <MessageId>Lorem</MessageId>
  <MessageSource>Ipsum</MessageSource>
 </Header>
 <Body>
  <Abc001>
   <!--newly introduced error - removed the following element-->
   <!--<Abc002>dolor</Abc002>-->
   <Abc003>sit amet</Abc003>
   <!--The value for Abc004 is increased beyond the allowed 200 characters-->
   <Abc004>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</Abc004>
   <Abc005>
    <Abc006>1234</Abc006>
    <Abc007>
     <Abc008>Ut enim</Abc008>
     <ABC009>AD</ABC009>
     <!--<Abc010>minim</Abc010>-->
    </Abc007>
    <Abc011>1982-10-37</Abc011>
    <Abc012>
     <Abc013>veniam</Abc013>
     <Abc014>nostrud</Abc014>
    </Abc012>
   </Abc005>
   <!--the element below is not allowed-->
   <Abc15>Not allowed</Abc15>
  </Abc001>
 </Body>
</Message>

and finally, here is the validation result:

<MessageResponse xmlns="http://www.somesite.com/somefolder/messages">
    <Result>false</Result>
    <Status>Failed</Status>
    <FaultCount>1</FaultCount>
    <Faults>
        <Fault>
            <FaultCode>ERR01</FaultCode>
            <FaultMessage>The element 'Abc001' in namespace 'http://www.somesite.com/somefolder/messages' has invalid child element 'Abc003' in namespace 'http://www.somesite.com/somefolder/messages'. List of possible elements expected: 'Abc002' in namespace 'http://www.somesite.com/somefolder/messages'.</FaultMessage>
        </Fault>
    </Faults>
</MessageResponse>

Here is my C# code I am using to validate:

public async Task<IMIDPreValidationAckMessage> ValidateXmlMessage( XDocument doc )
    {
        var result = new PreValidationAckMessage();
        result.Result = true;
        result.Status = "Succeeded";

        var xsd = HttpContext.Current.Server.MapPath( "~/message01.xsd" );

        try
        {
            var uri = new System.Uri(xsd);

            var localPath = uri.LocalPath;

            var docNameSpace = doc.Root.Name.Namespace.NamespaceName;

            XmlSchemaSet schemas = new XmlSchemaSet();
            schemas.Add( docNameSpace, localPath );

            XmlReaderSettings xrs = new XmlReaderSettings();
            xrs.ValidationType = ValidationType.Schema;
            xrs.ValidationFlags |= XmlSchemaValidationFlags.ReportValidationWarnings;
            xrs.Schemas = schemas;

            result.XSDNamespace = doc.Root.GetDefaultNamespace().NamespaceName;
            var errCode = 1;

            xrs.ValidationEventHandler += ( s, e ) =>
            {
                var msg = e.Message;
                result.Result = false;
                result.Status = "Failed";
                result.FaultCount++;
                result.Faults.Add( new Fault
                {
                    FaultCode = "ERR" + errCode++.ToString().PadLeft( 2, '0' ),
                    FaultMessage = e.Message
                } );
            };

            using ( XmlReader xr = XmlReader.Create( doc.CreateReader(), xrs ) )
            {
                while ( xr.Read() ) { }
            }
        }
        catch ( System.Exception ex )
        {
            result.Result = false;
            result.Status = "Unknown Error";
        }
        return result;
    }

Can someone please tell me what is wrong here?

xml xsd c#
2 Answers

I wrote this code which validates every single element of the xml message and tries to correct what can be corrected:

        static void ValidateMessage(XDocument xDoc)
    {
        var schemas = new XmlSchemaSet();
        schemas.Add("", @"Messages_Schema.xsd");
        schemas.Compile();

        var schemaElements = (XmlSchemaElement)schemas.GlobalElements.Values.OfType<XmlSchemaObject>()
            .FirstOrDefault(e => ((XmlSchemaElement)e).Name == xDoc.Root?.Name.LocalName);

        var xmlElement =
            (XmlSchemaSequence)((XmlSchemaComplexType)schemaElements?.ElementSchemaType)?.ContentTypeParticle;

        var elementNameList = new Dictionary<string, XmlSchemaObject>();

        AddElementsInDictionary(xmlElement?.Items.OfType<XmlSchemaObject>().ToList(), elementNameList, new List<string>());

        var errorList = new List<string>();
        bool elementFixed;

        do
        {
            errorList.Clear();

            ValidateElements(xDoc.Root?.Elements().ToList(), elementNameList, schemas, new List<string>(), errorList);

            errorList.ForEach(e =>
            {
                if (!ErrorLogs.Contains(e))
                    ErrorLogs.Add(e);
            });

            elementFixed = false;

            if (XElementsToDelete.Any() || XElementsToCut.Any() || XElementsInvalid.Any())
            {
                XElementsInvalid.ForEach(xElement =>
                {
                    var xParent = xElement.Parent;
                    var xParentSchema =
                        (xParent?.GetSchemaInfo()?.SchemaType as XmlSchemaComplexType)?.ContentTypeParticle as
                        XmlSchemaSequence;

                    var elementNameLists = new List<string>();

                    AddElementsInList(xParentSchema?.Items.OfType<XmlSchemaObject>().ToList(), elementNameLists);

                    var index = elementNameLists.IndexOf(xElement.Name.LocalName);
                    if (index == 0)
                    {
                        xElement.Remove();
                        var log = $"    Move: {xElement.Name} element to the top of the sequence";
                        if (!FixLogs.Contains(log))
                            FixLogs.Add(log);

                        elementFixed = true;
                        xParent?.AddFirst(xElement);
                    }
                    else
                    {
                        var xNextElement = xElement.NextNode as XElement;
                        var indexNext = elementNameLists.IndexOf(xNextElement?.Name.LocalName);

                        if (indexNext - index > 1)
                        {
                            var xPreviousElement = xElement.PreviousNode as XElement;
                            var indexPrevious = elementNameLists.IndexOf(xPreviousElement?.Name.LocalName);

                            do
                            {
                                indexPrevious -= 1;
                                var xLastValidElement = xParent?.Element(elementNameLists[indexPrevious]);
                                if (xLastValidElement == null) continue;
                                xPreviousElement?.Remove();
                                var log = $"    Move: {xPreviousElement?.Name} element after {xLastValidElement.Name}";
                                if (!FixLogs.Contains(log))
                                    FixLogs.Add(log);

                                elementFixed = true;
                                xLastValidElement.AddAfterSelf(xPreviousElement);
                                break;

                            } while (indexPrevious > 0);
                        }
                        else
                        {
                            do
                            {
                                index -= 1;
                                var xLastValidElement = xParent?.Element(elementNameLists[index]);
                                if (xLastValidElement == null) continue;
                                xElement.Remove();
                                var log = $"    Move: {xElement.Name} element after {xLastValidElement.Name}";
                                if (!FixLogs.Contains(log))
                                    FixLogs.Add(log);

                                elementFixed = true;
                                xLastValidElement.AddAfterSelf(xElement);
                                break;

                            } while (index > 0);
                        }
                    }
                });

                XElementsToDelete.ForEach(e =>
                {
                    e.Remove();
                    var log = $"    Delete: {e.Name} element";
                    if (!FixLogs.Contains(log))
                        FixLogs.Add(log);

                    elementFixed = true;
                });

                XElementsToCut.ForEach(e =>
                {
                    var schemaType = (XmlSchemaSimpleType)e.GetSchemaInfo()?.SchemaType;
                    var restriction = (XmlSchemaSimpleTypeRestriction)schemaType?.Content;
                    var enumFacets = restriction?.Facets.OfType<XmlSchemaMaxLengthFacet>();
                    var maxLengthFacet = enumFacets?.ToList().FirstOrDefault();

                    if (maxLengthFacet != null)
                    {
                        var maxLength = int.Parse(maxLengthFacet.Value);
                        var log = $"    Cut: {e.Name} value to maxLength: {maxLength}";
                        if (!FixLogs.Contains(log))
                            FixLogs.Add(log);

                        elementFixed = true;
                        e.Value = e.Value.Substring(0, maxLength);
                    }
                });

                if (!elementFixed)
                {
                    var log = "    Cannot fix:";
                    if (!FixLogs.Contains(log))
                        FixLogs.Add(log);

                    errorList.ForEach(e =>
                    {
                        if (!FixLogs.Contains($"        {e}"))
                            FixLogs.Add($"        {e}");
                    });
                }

            }

            XElementsToDelete.Clear();
            XElementsInvalid.Clear();
            XElementsToCut.Clear();

        } while (errorList.Count > 0 && elementFixed);

        Console.WriteLine($"Validating a {xDoc.Root?.Name.LocalName}");
        Console.WriteLine("");
        ErrorLogs.ForEach(e =>
        {
            Console.WriteLine("    {0}", e);
        });

        Console.WriteLine("");
        Console.WriteLine("Fixing");
        Console.WriteLine("");
        FixLogs.ForEach(e =>
        {
            Console.WriteLine("{0}", e);
        });

        Console.WriteLine("");
        Console.WriteLine("Message {0}",
            errorList.Count > 0 ? "did not validate" : "validated");

        Console.ReadKey();
    }


    private static void AddElementsInList(IEnumerable<XmlSchemaObject> schemaObjectList,
        ICollection<string> schemaObjectKeys)
    {
        schemaObjectList.ToList().ForEach(se =>
        {
            switch (se.GetType().ToString())
            {
                case "System.Xml.Schema.XmlSchemaElement":
                    var element = se as XmlSchemaElement;

                    var name = (se as XmlSchemaElement)?.QualifiedName.Name;

                    if (!schemaObjectKeys.Contains(name))
                        schemaObjectKeys.Add(name);

                    var elementSchemaType = element?.ElementSchemaType;
                    if (elementSchemaType != null)
                        AddElementsInList(new List<XmlSchemaObject> { elementSchemaType }, schemaObjectKeys);

                    break;

                case "System.Xml.Schema.XmlSchemaComplexType":

                    break;

                case "System.Xml.Schema.XmlSchemaSequence":
                    var sequence = se as XmlSchemaSequence;

                    var sequenceItems = sequence?.Items.OfType<XmlSchemaObject>().ToList();
                    if (sequenceItems != null)
                        AddElementsInList(sequenceItems, schemaObjectKeys);

                    break;

                case "System.Xml.Schema.XmlSchemaChoice":
                    var choice = se as XmlSchemaChoice;

                    var choiceItems = choice?.Items.OfType<XmlSchemaObject>().ToList();
                    if (choiceItems != null)
                        AddElementsInList(choiceItems, schemaObjectKeys);

                    break;

                case "System.Xml.Schema.XmlSchemaGroupRef":
                    var group = se as XmlSchemaGroupRef;

                    var groupParticle = group?.Particle;
                    if (groupParticle != null)
                        AddElementsInList(new List<XmlSchemaObject> { groupParticle }, schemaObjectKeys);

                    break;
            }
        });
    }

    private static void AddElementsInDictionary(IEnumerable<XmlSchemaObject> schemaObjectList,
        IDictionary<string, XmlSchemaObject> schemaObjectDictionary, IList<string> schemaObjectDictionaryKeys)
    {
        schemaObjectList.ToList().ForEach(se =>
        {
            switch (se.GetType().ToString())
            {
                case "System.Xml.Schema.XmlSchemaElement":
                    var element = se as XmlSchemaElement;

                    schemaObjectDictionaryKeys.Add((se as XmlSchemaElement)?.QualifiedName.Name);
                    var path = string.Join("/", schemaObjectDictionaryKeys);
                    if (!schemaObjectDictionary.ContainsKey(path))
                        schemaObjectDictionary.Add(path, se);

                    var elementSchemaType = element?.ElementSchemaType;
                    if (elementSchemaType != null)
                        AddElementsInDictionary(new List<XmlSchemaObject> { elementSchemaType },
                            schemaObjectDictionary, schemaObjectDictionaryKeys);

                    if (schemaObjectDictionaryKeys.Count > 0)
                        schemaObjectDictionaryKeys.RemoveAt(schemaObjectDictionaryKeys.Count - 1);

                    break;

                case "System.Xml.Schema.XmlSchemaComplexType":
                    var complexType = se as XmlSchemaComplexType;

                    var complexTypeParticle = complexType?.ContentTypeParticle;
                    if (complexTypeParticle != null)
                        AddElementsInDictionary(new List<XmlSchemaObject> { complexTypeParticle },
                            schemaObjectDictionary, schemaObjectDictionaryKeys);

                    break;

                case "System.Xml.Schema.XmlSchemaSequence":
                    var sequence = se as XmlSchemaSequence;

                    var sequenceItems = sequence?.Items.OfType<XmlSchemaObject>().ToList();
                    if (sequenceItems != null)
                        AddElementsInDictionary(sequenceItems, schemaObjectDictionary, schemaObjectDictionaryKeys);

                    break;

                case "System.Xml.Schema.XmlSchemaChoice":
                    var choice = se as XmlSchemaChoice;

                    var choiceItems = choice?.Items.OfType<XmlSchemaObject>().ToList();
                    if (choiceItems != null)
                        AddElementsInDictionary(choiceItems, schemaObjectDictionary, schemaObjectDictionaryKeys);

                    break;

                case "System.Xml.Schema.XmlSchemaGroupRef":
                    var group = se as XmlSchemaGroupRef;

                    var groupParticle = group?.Particle;
                    if (groupParticle != null)
                        AddElementsInDictionary(new List<XmlSchemaObject> { groupParticle }, schemaObjectDictionary,
                            schemaObjectDictionaryKeys);

                    break;

            }
        });
    }

    private static void ValidateElements(List<XElement> xElementList,
        IReadOnlyDictionary<string, XmlSchemaObject> schemaObjectDictionary, XmlSchemaSet schemas,
        IList<string> schemaObjectDictionaryKeys, ICollection<string> errorList)
    {
        xElementList.ForEach(xElement =>
        {
            schemaObjectDictionaryKeys.Add(xElement.Name.LocalName);
            var path = string.Join("/", schemaObjectDictionaryKeys);

            if (schemaObjectDictionary.ContainsKey(path))
            {
                var validateObject = schemaObjectDictionary[path];

                xElement.Validate(validateObject, schemas,
                    (o, e) =>
                    {
                        if (!errorList.Contains(e.Message))
                            errorList.Add(e.Message);

                        if (e.Message.Contains("has incomplete content"))
                        {
                            if (!XElementsToDelete.Contains((XElement)o))
                                XElementsToDelete.Add((XElement)o);
                        }

                        if (e.Message.Contains("has invalid child element"))
                        {
                            if (!XElementsInvalid.Contains((XElement)o))
                                XElementsInvalid.Add((XElement)o);
                        }

                        if (e.Message.Contains("actual length is greater than the MaxLength value"))
                        {
                            if (!XElementsToCut.Contains((XElement)o))
                                XElementsToCut.Add((XElement)o);
                        }
                    }, true);

                if (xElement.HasElements)
                    ValidateElements(xElement.Elements().ToList(), schemaObjectDictionary, schemas,
                        schemaObjectDictionaryKeys, errorList);
            }
            else
            {
                var log = $"The element '{xElement.Name.LocalName}' is unknown. It should be delete.";
                if (!errorList.Contains(log))
                    errorList.Add(log);

                if (XElementsInvalid.Contains(xElement))
                    XElementsInvalid.Remove(xElement);

                if (XElementsToCut.Contains(xElement))
                    XElementsToCut.Remove(xElement);

                if (!XElementsToDelete.Contains(xElement))
                    XElementsToDelete.Add(xElement);
            }

            if (schemaObjectDictionaryKeys.Count > 0)
                schemaObjectDictionaryKeys.RemoveAt(schemaObjectDictionaryKeys.Count - 1);
        });
    }
user_38210500
Related