What Characters Need to be Escaped in XML Documents?

1. Introduction

Some characters are treated specially when processing XML documents. These are the characters which are used to markup XML syntax; when they appear as a part of a document rather than for syntax markup, they need to be appropriately escaped. These characters are:

"	"		Double quote
'	'		Single quote
<	&lt;		Left angle bracket
>	&gt;		Right angle bracket
&	&amp;		Ampersand

2. Character Data

All text that is not markup constitutes character data of the document. Within character data, “&” and “<” must not appear except when used in markup. However “>”, “”” and “‘” can appear directly within character data without having to be encoded.

<?xml version="1.0"?>
<valid>"'></valid>
<?xml version="1.0"?>
<invalid>&</invalid>
<?xml version="1.0"?>
<invalid><</invalid>

However, “>” cannot appear in the form “]]>” unless it is a part of CDATA ending sections.

<?xml version="1.0"?>
<invalid>]]></invalid>

The following is valid as “>” has been encoded properly:

<?xml version="1.0"?>
<valid>]]&gt;</valid>

2. Attributes

Within attributes, “>” is valid.

<valid attrib=">"></valid>

However, ‘”‘ is not valid within double quotes; it must be encoded using “&quot;”.

<valid attrib="&quot;"></valid>

Similarly “‘” is not valid within single quotes. It must be encoded as “&apos;”:

<valid attrib='&apos;'></valid>

3. Comments

Comments can appear anywhere in a document outside of markup. Within comments, none of the 5 special characters must be escaped or encoded.

<valid><!-- '"<>& --></valid>

In addition, the string “–” must not appear within comments.

<invalid><!-- Hello -- there --></invalid>

A consequence of this rule is that a comment must not end with “—>” (three dashes followed by a right-angle-bracket). The following is invalid:

<invalid><!-- Hello there ---></invalid>

4. Processing Instructions

Processing Instructions (PI) are used to add instructions for XML processing applications. None of the 5 special characters must be encoded within PI statements.

<valid><?execute <>"'& ?></valid>

A further restriction in the case of processing instructions is that the instruction name must not be the string “xml”; this name is reserved for standardization of the XML specification itself.

<invalid><?xml hello ?></invalid>

CData

CDATA sections are used to escape blocks of text containing characters which would otherwise be recognized as markup. This section begin with the string “<![CDATA[” and end with the string “]]>”. Within a CData section, none of the 5 special characters must be encoded.

<valid><![CDATA[[<greeting>Hello world: <>&'" </greeting>]]></valid>

However, within CData sections, the string “]]>” must not appear except to end the section:

<invalid><![CDATA[[This string("]]>") must not appear here]]></invalid>

The same can be re-written with two CData sections as follows:

<valid><![CDATA[[This string("]]]]><![CDATA[[>") must not appear here]]></valid>

Conclusion

This article demonstrated what the predefined XML entities are and the various circumstances in which they can be used.

Parsing XML in Python

1. Introduction

XML can be parsed in python using the xml.etree.ElementTree library. This article shows you how to parse and extract elements, attributes and text from XML using this library.

While this library is easy to use, it loads the whole XML document into memory and hence may not be suitable for processing large XML files or where run-time memory is an issue.

2. Sample XML

Here is a short snippet of the sample XML we will be working with.

<?xml version="1.0"?>
<catalog>
  <book id="bk101">
    <author>Gambardella, Matthew</author>
    <title>XML Developer's Guide</title>
    <genre>Computer</genre>
    <price>44.95</price>
    <publish_date>2000-10-01</publish_date>
    <description>An in-depth look at creating applications 
    with XML.</description>
  </book>
  <book id="bk102">
    <author>Ralls, Kim</author>
    <title>Midnight Rain</title>
...

3. Getting Started

3.1 Parsing an XML File

It is very simple to parse an XML using ElementTree. The following returns an ElementTree object which contains all the XML artifacts.

import xml.etree.ElementTree as ET

tree = ET.parse('catalog.xml')

Once you have the ElementTree object, you can get the root Element object using the tree.getroot() method:

root = tree.getroot()

3.2 Parsing an XML String

We use the ElementTree.fromstring() method to parse an XML string. The method returns root Element directly: a subtle difference compared with the ElementTree.parse() method which returns an ElementTree object.

print ET.fromstring('<a><b>1</b>2</a>')
# prints "<Element 'a' at ...>"

4. Working with Elements

From the Element object, you can get the XML tag name using the tag property.

print root.tag    # outputs "catalog"

Get a list of child Elements from any Element object using element.findall(‘*’). You can find specific children by name by passing the name of the tag e.g. element.findall(‘book’).

For example, the following code recursively processes all the elements in the XML and prints the name of the tag.

def show(elem):
    print elem.tag
    for child in elem.findall('*'):
        show(child)

show(root)

The above code can be modified to show nicely indented output of the tag names:

def show(elem, indent = 0);
    print ' ' * indent + elem.tag
    for child in elem.findall('*'):
         show(elem, indent + 1)

show(root)

To find a single element by name, use elem.find(tagName):

print root.find('book').tag    # prints "book"

5. Working with Attributes

XML attributes can be extracted from an Element object using the element.items() method which returns a sequence of name, value pairs. (The name-value pairs are returned in random order, not in the order they appear in the XML.)

for attrName, attrValue in elem.items():
    print attrName + '=' + attrValue

To retrieve a single attribute value by name, use the elem.get(attrName) method:

print root.find('book').get('id')    # prints "bk101"

Get a list of all attribute names defined on the element using elem.keys(). Returns an empty list if no attributes are defined.

print root.find('book').keys()
# prints ['id']

To get all the attributes as a python dictionary, use the elem.attrib property:

print root.find('book').attrib
# prints {'id': 'bk101'}

6. Retrieving Element Text Content

You can retrieve an element’s text content using the elem.text property as follows:

print ET.fromstring('<a>Hello<b>1</b>2</a>').text
# prints "Hello"

print ET.fromstring('<a>Hello<b>1</b>2</a>').find('b').text
#prints "1"

Text appearing after the element’s end tag is retrieved using elem.tail property:

print ET.fromstring('<a>Hello<b>1</b>2</a>').find('b').tail
# prints "2"

7. Using Path to Extract Content

Some of the Element object methods support extracting content by using a syntax similar to XPath:

Retrieve a descendant element:

print root.find('book/author').text
# prints "Gambardella, Matthew"

To obtain the text of the first matching element, use the elem.findtext() method as follows:

print root.findtext('book/author')
#prints "Gambardella, Matthew"

Retrieve and process a list of matching elements using elem.findall():

for e in root.findall('book/author'):
    print e.text

# prints the following
Gambardella, Matthew
Ralls, Kim
Corets, Eva
Corets, Eva
Corets, Eva
Randall, Cynthia
Thurman, Paula
Knorr, Stefan
Kress, Peter
O'Brien, Tim
O'Brien, Tim
Galos, Mike

Find a specific element by position. (Position indexes start with 1).

print root.find('book[2]/author').text
# prints "Ralls, Kim"

Here is an example to find and concatenate all text content using a reduce operation:

reduce(lambda x, y: x + '|' + y.text, root.findall("book/author"), '')

# prints "|Gambardella, Matthew|Ralls, Kim|Corets, Eva|Corets, Eva|Corets, Eva|Randall, Cynthia|Thurman, Paula|Knorr, Stefan|Kress, Peter|O'Brien, Tim|O'Brien, Tim|Galos, Mike"

Conclusion

This article demonstrated some aspects of parsing XML with python. We showed how to parse an XML file or an XML string and extract elements, attributes and text content.

See Also

What does <![CDATA[]]> in XML mean?

Introduction

CDATA stands for Character Data. A CDATA section in XML is used to escape text containing characters which would otherwise be recognized as markup. It can appear anywhere character data can occur.

Markup

A CDATA section is marked up starting with “<![CDATA[” and ending with “]]>“. Any character data (other than “]]>“) can appear within the section without needing to be escaped. For example, angle brackets (<>) and ampersands (&) which indicate XML markup need not be escaped within a CDATA section.

For example, the following is a CDATA section. The angle brackets surrounding “greeting” and “/greeting” need not be escaped. When processing this XML, the parser receives the text “<greeting>Hello, world!</greeting>” as character data and not as markup.

<![CDATA[<greeting>Hello, world!</greeting>]]>

In addition, parameter entity references are recognized within CDATA sections. For example, assume the following parameter entity is defined:

<!ENTITY AnEntity "Sample entity data here">

Within the following CDATA section, the entity reference “%AnEntity” is recognized and the value is replaced within character data passed to the XML processor.

<![CDATA[My value is %AnEntity]]>

Nesting

A CDATA section may not be nested inside another because “]]>” may not appear directly except to end the CDATA section. The following is invalid:

<![CDATA[We need the ending "]]>" here.]]>

Instead the above can be written in two sections as follows:

<![CDATA[We need the ending "]]]]><![CDATA[>" here.]]>

Attribute Value

Within a Document-Type Definition (DTD), an attribute value may be declared to be of type CDATA as follows:

<!ATTLIST img
          src CDATA #REQUIRED>

This declaration states that an img element must have a src attribute whose value type is CDATA.

Where is it used?

CDATA sections are used when larger amounts of verbatim text need to appear within XML documents and processed verbatim. Smaller quantities of such text can be properly encoded to escape the XML characters, but for larger text, it helps to preserve the meaning of the text without having to do so.