I need to process XML files containing text with Romanian characters, (text like 'licenţe şi mărci').
After processing this text ends up in html documents. If at the final step I output the content using something like:
doc.write(path, encoding='utf-8', method='xml' )
NOTE: doc above is an instance of lxml.etree.ElementTree
I get something like:
<span id="123">licenţe şi mărci</span>
Everything looks fine (i.e. Romanian chars are not converted to html entities).
If I do use the html encoding method
doc.write(path, encoding='utf-8', method='html' )
I get something like:
<span id="128927">licenţe şi mărci</span>
Although the document looks ok in the browser, the text in the document is also used by another process which extracts the text and uses it for some diff-ing jobs which get completely confused by the html entities.
Although using the xml as the method argument to etree.ElementTree.write seems to solve my current problem, it feels wrong since I really want to generate html files and may behave badly in other contexts.
Is there a way to write a document as html (using method='html') without having lxml replace non ascii chars with html entities ?
After @mzjn 's question it turns out that the story is a bit more complicated.
Here's a minimal code that reproduces the problem:
from lxml import etree
from os import path
def process( method, out_name ):
# the doc
x = etree.XML('<span id="123">licenţe şi mărci</span>')
tree = etree.ElementTree(x)
#do some processing
modifiedTree = modifyDoc(tree)
# write the result
modifiedTree.write(out_name, encoding='utf-8', method=method)
def modifyDoc(tree):
root = tree.getroot()
doNothingTransformPath = _absolutePath('transform.xslt')
transform_doc = etree.parse(doNothingTransformPath)
transform = etree.XSLT(transform_doc)
return transform(root)
def _absolutePath(relative):
return path.abspath(path.join( __file__ , '..', relative))
if __name__ == '__main__':
process( 'xml', 'out1.xml')
process( 'html', 'out2.html')
where (in this example) the XSLT transform (doing nothing) is:
-- transform.xslt --
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="no"/>
<xsl:template match="/">
<xsl:copy-of select="/"/>
</xsl:template>
</xsl:stylesheet>
If I don't pass the document through the XSLT transform the output does not contain HTML entities, when passed through the modifyDoc function (i.e. applying an xslt transform) the output is changed as described above, (i.e. when using method='html' the output contains HTML entities).
So running the program above outputs two files:
-- out1.xml --
<span id="123">licenţe şi mărci</span>
and -- out2.html --
<span id="123">licenţe şi mărci</span>