How to Parse and Modify XML in Python?

We often require to parse data written in different languages. Python programming provides numerous libraries to parse or split data written in other languages. In this Python XML Parser Tutorial, you will learn how to parse XML using Python.

Here are all the topics that are covered in this tutorial:

So let’s get started. :)

What is XML?

XML stands for Extensible Markup Language. It is similar to HTML in its appearance but, XML is used for data presentation, while HTML is used to define what data is being used. XML is exclusively designed to send and receive data back and forth between clients and servers. Take a look at the following example:

EXAMPLE:

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
<food>
    <item name="breakfast">Idly</item>
    <price>$2.5</price>
    <description>
   Two idly's with chutney
   </description>
    <calories>553</calories>
</food>
<food>
    <item name="breakfast">Paper Dosa</item>
    <price>$2.7</price>
    <description>
    Plain paper dosa with chutney
    </description>
    <calories>700</calories>
</food>
<food>
    <item name="breakfast">Upma</item>
    <price>$3.65</price>
    <description>
    Rava upma with bajji
    </description>
    <calories>600</calories>
</food>
<food>
    <item name="breakfast">Bisi Bele Bath</item>
    <price>$4.50</price>
    <description>
   Bisi Bele Bath with sev
    </description>
    <calories>400</calories>
</food>
<food>
    <item name="breakfast">Kesari Bath</item>
    <price>$1.95</price>
    <description>
    Sweet rava with saffron
    </description>
    <calories>950</calories>
</food>
</metadata>

The above example shows the contents of a file which I have named as ‘Sample.xml’ and I will be using the same in this Python XML parser tutorial for all the upcoming examples.

Python XML Parsing Modules

Python allows parsing these XML documents using two modules namely, the xml.etree.ElementTree module and Minidom (Minimal DOM Implementation). Parsing means to read information from a file and split it into pieces by identifying parts of that particular XML file. Let’s move on further to see how we can use these modules to parse XML data.

xml.etree.ElementTree Module

This module helps us format XML data in a tree structure which is the most natural representation of hierarchical data. Element type allows storage of hierarchical data structures in memory and has the following properties:

ElementTree is a class that wraps the element structure and allows conversion to and from XML. Let us now try to parse the above XML file using python module.

Top 10 Trending Technologies to Learn in 2024 | Edureka

There are two ways to parse the file using ‘ElementTree’ module. The first is by using the parse() function and the second is fromstring() function. The parse () function parses XML document which is supplied as a file whereas, fromstring parses XML when supplied as a string i.e within triple quotes.

Using parse() function

As mentioned earlier, this function takes XML in file format to parse it. Take a look at the following example:

EXAMPLE:

import xml.etree.ElementTree as ET
mytree = ET.parse('sample.xml')
myroot = mytree.getroot()

As you can see, The first thing you will need to do is to import the xml.etree.ElementTree module. Then, the parse() method parses the ‘Sample.xml’ file. The getroot() method returns the root element of ‘Sample.xml’.

When you execute the above code, you will not see outputs returned but there will be no errors indicating that the code has executed successfully. To check for the root element, you can simply use the print statement as follows:

EXAMPLE:

import xml.etree.ElementTree as ET
mytree = ET.parse('sample.xml')
myroot = mytree.getroot()
print(myroot)

OUTPUT:   <Element ‘metadata’ at 0x033589F0>

The above output indicates that the root element in our XML document is ‘metadata’.

Using fromstring() function

You can also use fromstring() function to parse your string data. In case you want to do this, pass your XML as a string within triple quotes as follows:

import xml.etree.ElementTree as ET
data='''<?xml version="1.0" encoding="UTF-8"?>
<metadata>
<food>
    <item name="breakfast">Idly</item>
    <price>$2.5</price>
    <description>
   Two idly's with chutney
   </description>
    <calories>553</calories>
</food>
</metadata>
'''
myroot = ET.fromstring(data)
#print(myroot)
print(myroot.tag)

The above code will return the same output as the previous one. Please note that the XML document used as a string is just one part of ‘Sample.xml’ which I have used for better visibility. You can use the complete XML document as well.

You can also retrieve the root tag by using the ‘tag’ object as follows:

EXAMPLE:

print(myroot.tag)

OUTPUT:  metadata

You can also slice the tag string output by just specifying which part of the string you want to see in your output.

EXAMPLE:

print(myroot.tag[0:4])

OUTPUT:  meta

As mentioned earlier, tags can have dictionary attributes as well. To check if the root tag has any attributes you can use the ‘attrib’ object as follows:

EXAMPLE:

print(myroot.attrib)

OUTPUT: {}

As you can see, the output is an empty dictionary because our root tag has no attributes.

Find out our Python Training in Top Cities/Countries

Finding Elements of Interest

The root consists of child tags as well. To retrieve the child of the root tag, you can use the following:

EXAMPLE:

print(myroot[0].tag)

OUTPUT: food

Now, if you want to retrieve all first-child tags of the root, you can iterate over it using the for loop as follows:

EXAMPLE:

for x in myroot[0]:
     print(x.tag, x.attrib)

OUTPUT:

item {‘name’: ‘breakfast’}
price {}
description {}
calories {}

All the items returned are the child attributes and tags of food.

To separate out the text from XML using ElementTree, you can make use of the text attribute. For example, in case I want to retrieve all the information about the first food item, I should use the following piece of code:

EXAMPLE:

for x in myroot[0]:
        print(x.text)

OUTPUT:

Idly
$2.5
Two idly’s with chutney
553

As you can see, the text information of the first item has been returned as the output. Now if you want to display all the items with their particular price, you can make use of the get() method. This method accesses the element’s attributes.

EXAMPLE:

for x in myroot.findall('food'):
    item =x.find('item').text
    price = x.find('price').text
    print(item, price)

OUTPUT:

Idly $2.5
Paper Dosa $2.7
Upma $3.65
Bisi Bele Bath $4.50
Kesari Bath $1.95

The above output shows all the required items along with the price of each of them. Using ElementTree, you can also modify the XML files.

Modifying XML files

The elements present your XML file can be manipulated. To do this, you can use the set() function.  Let us first take a look at how to add something to XML.

Adding to XML:

The following example shows how you can add something to the description of items.

EXAMPLE:

for description in myroot.iter('description'):
     new_desc = str(description.text)+'wil be served'
     description.text = str(new_desc)
     description.set('updated', 'yes')

mytree.write('new.xml')

The write() function helps create a new xml file and writes the updated output to the same. However, you can modify the original file as well, using the same function. After executing the above code, you will be able to see a new file has been created with the updated results.

The above image shows the modified description of our food items. To add a new subtag, you can make use of the SubElement() method. For example, if you want to add a new specialty tag to the first item Idly, you can do as follows:

EXAMPLE:

ET.SubElement(myroot[0], 'speciality')
for x in myroot.iter('speciality'):
     new_desc = 'South Indian Special'
     x.text = str(new_desc)

mytree.write('output5.xml')

OUTPUT:

As you can see, a new tag has been added under the first food tag. You can add tags wherever you want by specifying the subscript within [] brackets. Now let us take a look at how to delete items using this module.

Deleting from XML:

To delete attributes or sub-elements using ElementTree, you can make use of the pop() method. This method will remove the desired attribute or element that is not needed by the user.

EXAMPLE:

myroot[0][0].attrib.pop('name', None)

# create a new XML file with the results
mytree.write('output5.xml')

OUTPUT:

The above image shows that the name attribute has been removed from the item tag. To remove the complete tag, you can use the same pop() method as follows:

EXAMPLE:

myroot[0].remove(myroot[0][0])
mytree.write('output6.xml')

OUTPUT:

The output shows that the first subelement of the food tag has been deleted. In case you want to delete all tags, you can make use of the clear() function as follows:

EXAMPLE:

myroot[0].clear()
mytree.write('output7.xml')

OUTPUT:

When the above code is executed, the first child of food tag will be completely deleted including all the subtags. Till here we have been making use of the xml.etree.ElementTree module in this Python XML parser tutorial. Now let us take a look at how to parse XML using Minidom.

Choose the Right XML Parsing Model

Choosing the right XML parsing model depends on the specific requirements and characteristics of the XML data you are working with. Here’s a summary of when to use each XML parsing model:

1. DOM (Document Object Model) Parsing:

   – Use DOM parsing when you need to access and modify the entire XML document or traverse it in various directions.

   – Suitable for small to medium-sized XML documents where memory usage is not a significant concern.

   – Provides ease of use and flexibility in navigating and manipulating XML data.

2. SAX (Simple API for XML) Parsing:

   – Use SAX parsing when dealing with large XML documents where memory efficiency is critical.

   – Suitable for scenarios where you need to process XML sequentially and don’t need to access the entire document at once.

   – Provides better performance and reduced memory overhead compared to DOM parsing.

3. StAX (Streaming API for XML) Parsing:

   – Use StAX parsing when you want a balanced approach, combining the advantages of both DOM and SAX parsing.

   – Suitable for handling large XML documents with the flexibility of navigating in both forward and backward directions.

   – Provides a compromise between memory efficiency and ease of use.

To make the right choice, consider factors like the size of your XML documents, the complexity of data access and manipulation, processing speed requirements, and memory constraints. If you need to work with small to medium-sized documents and require easy access to the entire structure, DOM parsing may be appropriate. For large documents with limited memory, SAX parsing is ideal. If you need more flexibility in navigating the XML structure without loading the entire document, StAX parsing strikes a balance between the other two models.

Explore Third-Party XML Parser Libraries

In order to aid developers in efficiently parsing and processing XML data, third-party XML parser libraries are pre-built software components. These libraries offer various functionalities and advantages over implementing XML parsing from scratch. Here are some popular third-party XML parser libraries:

1. Untangle: Convert XML to a Python Object

   – Purpose: untangle is a Python library that simplifies XML parsing by directly converting XML data into Python objects. It provides an intuitive way to access and manipulate XML elements and attributes using dot notation, making it easy to work with XML data in Python.

   – Key Features: The library offers a straightforward approach to converting XML to Python objects without requiring complex XML parsing code. It is particularly useful for handling small to medium-sized XML data files and simplifying data extraction tasks.

2. Xmltodict: Convert XML to a Python Dictionary

   – Purpose: xmltodict is a Python library that efficiently converts XML data into a Python dictionary. It simplifies XML parsing and provides a dictionary-based representation of the XML structure, making it easy to access and manipulate data using familiar dictionary methods.

   – Key Features: The library is well-suited for converting XML data into a more accessible and easily navigable Python dictionary format. It is widely used for handling XML data in web services and APIs, and it simplifies tasks like data extraction and transformation.

3. Lxml: Use ElementTree on Steroids

   – Purpose: lxml is a powerful and feature-rich Python library for XML and HTML parsing. It is built on top of the libxml2 and libxslt libraries and provides a fast and efficient XML processing solution. lxml supports both ElementTree and XPath APIs, making it suitable for a wide range of XML processing tasks.

   – Key Features: The library offers a wide range of XML processing capabilities, including parsing, validation, XPath querying, and XSLT transformations. It is well-regarded for its performance and versatility, making it a popular choice for handling XML data in Python.

4. BeautifulSoup:  Deal With Malformed XML

   – Purpose: BeautifulSoup is a Python library primarily designed for parsing HTML, but it can also handle XML documents. It excels at dealing with poorly formatted or malformed XML/HTML data and provides a flexible way to extract information from web pages.

   – Key Features: The library is renowned for its simplicity and ability to parse complex, real-world HTML and XML data, even when it’s not strictly compliant with standards. It is commonly used for web scraping, data extraction, and working with XML/HTML data from various sources.

Each of these third-party XML parser libraries offers unique features and advantages. The choice of library depends on your specific needs and preferences, such as whether you want to work with Python objects, dictionaries, require advanced XML processing capabilities, or need robust handling of malformed XML data.

Bind XML Data to Python Objects

To bind XML data to Python objects, you can define models using XPath expressions or generate models from an XML schema. Both approaches enable you to create Python classes that correspond to the XML structure and easily interact with the XML data. Let’s explore each method:

1. Define Models With XPath Expressions:

   – In this approach, you manually define Python classes and map them to specific XPath expressions to access XML elements and attributes. This allows you to directly bind XML data to Python objects.

   – You can use libraries like `lxml` or `xml.etree.ElementTree` for parsing XML data and XPath expressions for querying specific elements or attributes.

   – Here’s an example using `lxml`:

from lxml import etree
class Person:
def __init__(self, name, age, email):
self.name = name
self.age = age
self.email = email
# Sample XML data
xml_data = '''
<person>
<name>John Doe</name>
<age>30</age>
<email>john.doe@example.com</email>
</person>

# Parse XML data
root = etree.fromstring(xml_data)
# Map XML elements to Python class attributes using XPath
person = Person(name=root.xpath('name/text()')[0],
age=int(root.xpath('age/text()')[0]),
email=root.xpath('email/text()')[0])
print(person.name) # John Doe
print(person.age) # 30
print(person.email) # john.doe@example.com

2. Generate Models From an XML Schema:

   – In this approach, you use an XML schema (XSD) to automatically generate Python classes that correspond to the XML structure defined in the schema.

   – The XML schema serves as a blueprint for the XML data, and using code generation tools, you can create Python classes that closely match the schema elements and attributes.

   – `xmlschema` is a Python library that can help you generate Python classes from an XML schema.

   – Here’s an example:

from xmlschema import XMLSchema
# Sample XML Schema
xsd_schema = '''
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:element name="person">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="name" type="xs:string"/>
                <xs:element name="age" type="xs:int"/>
                <xs:element name="email" type="xs:string"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>
'''
# Load XML Schema and generate Python classes
schema = XMLSchema(xsd_schema)
Person = schema.to_dict()['person']
# Sample XML data
xml_data = '''
<person>
    <name>John Doe</name>
    <age>30</age>
    <email>john.doe@example.com</email>
</person>
'''
# Bind XML data to Python object
person = Person.fromstring(xml_data)
print(person.name)   # John Doe
print(person.age)    # 30
print(person.email)  # john.doe@example.com

Using either of these methods, you can easily bind XML data to Python objects, providing a more organized and convenient way to work with XML data in your Python code. Choose the approach that best fits your project requirements and preferences.

Defuse the XML Bomb With Secure Parsers

The term “XML Bomb” refers to a type of security attack that involves using a specially crafted XML document to overwhelm and crash an XML parser or consume excessive system resources. This attack exploits vulnerabilities in the XML parsing process, leading to denial-of-service (DoS) situations.

To defuse the XML Bomb and ensure secure XML parsing, it is essential to use secure XML parsers that implement various protective measures. Here are some strategies to mitigate XML Bomb attacks:

1. Limit Entity Expansion: XML documents can contain entity references that reference external entities or recursively expand to create large amounts of data. Secure XML parsers should limit entity expansion to prevent excessive memory usage.
2. Set Depth Limits: Secure XML parsers should enforce depth limits on elements and nesting levels to prevent attacks that create deeply nested structures, consuming excessive processing time.
3. Implement Entity Resolution: Secure XML parsers should resolve external entities and resources cautiously. Disable external entity resolution if it is not required.
4. Validate Input: Validate and sanitize XML inputs to ensure that only well-formed and expected XML documents are processed. Reject or handle malicious or malformed XML data appropriately.
5. Use Trusted Parsers: Choose XML parsers from reputable sources and libraries that have a history of addressing security concerns and providing regular updates.
6. Enforce Resource Limits: Set resource limits on XML parsing operations to restrict the memory and processing time allocated to parsing tasks.
7. Enable XML Security Features: Some XML parsers offer built-in security features like secure processing, which can help mitigate XML Bomb attacks.
8. Keep Parsers Updated: Ensure that your XML parsers are up-to-date with the latest security patches and updates to guard against known vulnerabilities.
9. Validate XML Size: If your application allows user-provided XML data, consider implementing size limitations on XML documents to prevent excessively large inputs.
10. Monitor and Log: Monitor XML parsing activities and log potential attacks or unexpected behavior for further analysis and response.

By employing these security measures and using secure XML parsers, you can protect your applications from XML Bomb attacks and ensure safe XML processing without risking resource exhaustion or denial-of-service situations. Security is a critical aspect of XML parsing, and proactive measures are essential to maintaining the integrity and availability of your systems.

xml.dom.minidom Module

This module is basically used by people who are proficient with DOM (Document Object module). DOM applications often start by parsing XML into DOM. in xml.dom.minidom, this can be achieved in the following ways:

Using the parse() function:

The first method is to make use of the parse() function by supplying the XML file to be parsed as a parameter. For example:

EXAMPLE:

from xml.dom import minidom
p1 = minidom.parse("sample.xml");

Once you execute this, you will be able to split the XML file and fetch the required data. You can also parse an open file using this function. EXAMPLE:

dat=open('sample.xml')
p2=minidom.parse(dat)

The variable storing the opened file is supplied as a parameter to the parse function in this case.

Using parseString() Method:

This method is used when you want to supply the XML to be parsed as a string. EXAMPLE:

p3 = minidom.parseString('<myxml>Using<empty/> parseString</myxml>')

You can parse XML using any of the above methods. Now let us try to fetch data using this module.

Finding Elements of Interest:

After my file has been parsed, if I try to print it, the output that is returned displays a message that the variable storing the parsed data is an object of DOM.

EXAMPLE:

dat=minidom.parse('sample.xml')
print(dat)

OUTPUT: <xml.dom.minidom.Document object at 0x03B5A308>

Accessing Elements using GetElementByTagName:

EXAMPLE:

tagname= dat.getElementsByTagName('item')[0]
print(tagname)

If I try to fetch the first element using the GetElementByTagName method, I will see the following output:

OUTPUT:

<DOM Element: item at 0xc6bd00> Please note that just one output has been returned because I have used [0] subscript for convenience which will be removed in the further examples. To access the value of the attributes, I will have to make use of the value attribute as follows:

EXAMPLE:

dat = minidom.parse('sample.xml')
tagname= dat.getElementsByTagName('item')
print(tagname[0].attributes['name'].value)

OUTPUT:

breakfast To retrieve the data present in these tags, you can make use of the data attribute as follows:

EXAMPLE:

print(tagname[1].firstChild.data)

OUTPUT: Paper Dosa You can also split and retrieve the value of the attributes using the value attribute. EXAMPLE:

print(items[1].attributes['name'].value)

OUTPUT: breakfast To print out all the items available in our menu, you can loop through the items and return all the items. EXAMPLE:

for x in items:
    print(x.firstChild.data)

OUTPUT: Idly Paper Dosa Upma Bisi Bele Bath Kesari Bath To calculate the number of items on our menu, you can make use of the len() function as follows:

EXAMPLE:

print(len(items))

OUTPUT: 5

The output specifies that our menu consists of 5 items.

This brings us to the end of this Python XML Parser Tutorial. I hope you have understood everything clearly.