3
\$\begingroup\$

I've been working on a lightweight XML schema parser, and have what I think is a moderately clean solution (some parts helped out by previous questions I posted here) so far for obtaining all schema details, but would like any criticism at all that could help further improve this code.

Below I have supplied the schema class I wrote, and then an example schema.txt file that the schema class will open if run as main. The schema class calls under "main" can be modified if you want to get a better look at the schema data structure, and I have some accompanying functions I have written for the class to pull out specific details that I haven't put here because I still need to do some work on them.

schema.py:

from lxml import etree

INDICATORS = ["all", "sequence", "choice"]
TYPES = ["simpleType", "complexType"]

class schema:

    def __init__(self, schemafile):
        if schemafile is None:
            print "Error creating Schema: Invalid schema file used"
            return

        self.schema = self.create_schema(etree.parse(schemafile))

    def create_schema(self, schema_data):
        def getXSVal(element): #removes namespace
            return element.tag.split('}')[-1]

        def get_simple_type(element):
            return {
                "name": element.get("name"),
                "restriction": element.getchildren()[0].attrib,
                "elements": [ e.get("value") for e in element.getchildren()[0].getchildren() ]
        }

        def get_simple_content(element):
            return {
                "simpleContent": {
                    "extension": element.getchildren()[0].attrib,
                    "attributes": [ a.attrib for a in element.getchildren()[0].getchildren() ]
                }
            }

        def get_elements(element):


            if len(element.getchildren()) == 0:
                return element.attrib

            data = {}

            ename = element.get("name")
            tag = getXSVal(element)

            if ename is None:
                if tag == "simpleContent":
                    return get_simple_content(element)
                elif tag in INDICATORS:
                    data["indicator"] = tag
                elif tag in TYPES:
                    data["type"] = tag
                else:
                    data["option"] = tag

            else:
                if tag == "simpleType":
                    return get_simple_type(element)
                else: 
                    data.update(element.attrib)

            data["elements"] = []
            data["attributes"] = []
            children = element.getchildren()        

            for child in children:
                if child.get("name") is not None:
                    data[getXSVal(child)+"s"].append(get_elements(child))
                elif tag in INDICATORS and getXSVal(child) in INDICATORS:
                    data["elements"].append(get_elements(child))
                else:
                    data.update(get_elements(child))

            if len(data["elements"]) == 0:
                del data["elements"]
            if len(data["attributes"]) == 0:
                del data["attributes"]

            return data

        schema = {}
        root = schema_data.getroot()
        children = root.getchildren()
        for child in children:
            c_type = getXSVal(child)
            if child.get("name") is not None and not c_type in schema:
                schema[c_type] = []
            schema[c_type].append(get_elements(child))
        return schema

    def get_Types(self, t_name):
        types = []
        for t in self.schema[t_name]:
            types.append(t["name"])
        return types

    def get_simpleTypes(self):
        return self.get_Types("simpleType")

    def get_complexTypes(self):
        return self.get_Types("complexType")


if __name__ == '__main__':
    fschema = open("schema.txt")

    schema = schema(fschema)

    print schema.get_simpleTypes()
    print schema.get_complexTypes()

schema.txt:

<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" version="3.0" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="main_object">
    <xs:complexType>
      <xs:choice maxOccurs="unbounded">
        <xs:element minOccurs="1" maxOccurs="1" name="source">
          <xs:complexType>
            <xs:all>
              <xs:element name="name" type="xs:string" />
              <xs:element name="group_id" type="xs:integer" />
              <xs:element minOccurs="0" name="description" type="xs:string" />
            </xs:all>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element minOccurs="1" maxOccurs="1" name="event">
          <xs:complexType>
            <xs:all>
              <xs:element name="date" type="xs:date" />
              <xs:element minOccurs="0" name="event_type" type="xs:string" />
              <xs:element minOccurs="0" name="event_hours" type="xs:string" />
            </xs:all>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="state">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="name" type="xs:string" />
            </xs:sequence>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="location">
          <xs:complexType>
            <xs:all>
              <xs:element name="address" type="simpleAddressType" />
              <xs:element minOccurs="0" name="directions" type="xs:string" />
              <xs:element minOccurs="0" name="hours" type="xs:string" />
            </xs:all>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="selection_item">
          <xs:complexType>
            <xs:sequence>
              <xs:element minOccurs="0" name="selection_type_id" type="xs:integer" />
              <xs:element minOccurs="0" maxOccurs="unbounded" name="option_id">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:integer">
                      <xs:attribute name="sort_order" type="xs:integer" />
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="custom_selection">
          <xs:complexType>
            <xs:choice maxOccurs="unbounded">
              <xs:element name="heading" type="xs:string" />
              <xs:sequence maxOccurs="unbounded">
                <xs:element name="response_id">
                  <xs:complexType>
                    <xs:simpleContent>
                      <xs:extension base="xs:integer">
                        <xs:attribute name="sort_order" type="xs:integer" />
                      </xs:extension>
                    </xs:simpleContent>
                  </xs:complexType>
                </xs:element>
              </xs:sequence>
            </xs:choice>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
        <xs:element name="response">
          <xs:complexType>
            <xs:all>
              <xs:element name="text" type="xs:string" />
              <xs:element minOccurs="0" name="sort_order" type="xs:integer" />
            </xs:all>
            <xs:attribute name="id" type="xs:integer" use="required" />
          </xs:complexType>
        </xs:element>
       </xs:choice>
      <xs:attribute fixed="3.0" name="schemaVersion" type="xs:decimal" use="required" />
    </xs:complexType>
  </xs:element>

<xs:complexType name="simpleAddressType">
    <xs:all>
        <xs:element minOccurs="0" name="location_name" type="xs:string"/> 
        <xs:element name="line1" type="xs:string"/>
        <xs:element minOccurs="0" name="line2" type="xs:string"/>
        <xs:element minOccurs="0" name="line3" type="xs:string"/>
        <xs:element name="city" type="xs:string"/>
        <xs:element name="state" type="xs:string"/>
        <xs:element name="zip" type="xs:string"/>
    </xs:all> 
</xs:complexType> 

<xs:complexType name="certified">
  <xs:simpleContent>
    <xs:extension base="xs:integer">
      <xs:attribute name="certification" type="certificationEnum" />
    </xs:extension>
  </xs:simpleContent>
</xs:complexType>

  <xs:simpleType name="certificationEnum">
    <xs:restriction base="xs:string">
      <xs:enumeration value="unofficial_partial"/>
      <xs:enumeration value="unofficial_complete"/>
      <xs:enumeration value="certified"/>
      <xs:enumeration value="Unofficial_partial"/>
      <xs:enumeration value="Unofficial_complete"/>
      <xs:enumeration value="Unofficial_Partial"/>
      <xs:enumeration value="Unofficial_Complete"/>
      <xs:enumeration value="Certified"/>
    </xs:restriction>
  </xs:simpleType> 

  <xs:simpleType name="yesNoEnum">
    <xs:restriction base="xs:string">
      <xs:enumeration value="yes"/>
      <xs:enumeration value="no"/>
      <xs:enumeration value="Yes"/>
      <xs:enumeration value="No"/>
      <xs:enumeration value="YES"/>
      <xs:enumeration value="NO"/>
    </xs:restriction>
  </xs:simpleType> 

</xs:schema>
\$\endgroup\$
1
  • \$\begingroup\$ You're kind of reinventing the wheel, IMO. Have you seen aaronsw.com/2002/xmltramp ? \$\endgroup\$ Commented Apr 28, 2012 at 2:10

2 Answers 2

Reset to default
3
\$\begingroup\$ 
from lxml import etree

INDICATORS = ["all", "sequence", "choice"]
TYPES = ["simpleType", "complexType"]

class schema:

Python convention is to name classes using CamelCase.

    def __init__(self, schemafile):
        if schemafile is None:
            print "Error creating Schema: Invalid schema file used"
            return

Use exceptions report errors in python. Don't print problems to standard output and then try to continue. Nothing good will come of it. Actually, you don't even need to check for None, because it'll fail on the next line anyways.

        self.schema = self.create_schema(etree.parse(schemafile))

    def create_schema(self, schema_data):
        def getXSVal(element): #removes namespace
            return element.tag.split('}')[-1]

Shouldn't you at least verify that the namespace was correct?

        def get_simple_type(element):
            return {
                "name": element.get("name"),
                "restriction": element.getchildren()[0].attrib,
                "elements": [ e.get("value") for e in element.getchildren()[0].getchildren() ]
        }

It looks like you are using a dictionary like an object. Perhaps you should actually be creating a SimpleType object with these attributes.

        def get_simple_content(element):
            return {
                "simpleContent": {
                    "extension": element.getchildren()[0].attrib,
                    "attributes": [ a.attrib for a in element.getchildren()[0].getchildren() ]
                }
            }

        def get_elements(element):

I've go no idea what this function is trying to do

            if len(element.getchildren()) == 0:
                return element.attrib

            data = {}

            ename = element.get("name")
            tag = getXSVal(element)

            if ename is None:

It seems strange that you check for the name, but don't do anything with it

                if tag == "simpleContent":
                    return get_simple_content(element)

Its confusing the way you sometimes return something, other times you add into a dictionary. 

                elif tag in INDICATORS:
                    data["indicator"] = tag
                elif tag in TYPES:
                    data["type"] = tag
                else:
                    data["option"] = tag

            else:
                if tag == "simpleType":
                    return get_simple_type(element)
                else: 
                    data.update(element.attrib)

I don't really follow what the theory for this condition is. I do see the same code showing up multiple times which makes me wonder if it can be refactored to be cleaner.

            data["elements"] = []
            data["attributes"] = []
            children = element.getchildren()        

            for child in children:

Combine the last two lines

                if child.get("name") is not None:
                    data[getXSVal(child)+"s"].append(get_elements(child))
                elif tag in INDICATORS and getXSVal(child) in INDICATORS:
                    data["elements"].append(get_elements(child))
                else:
                    data.update(get_elements(child))

            if len(data["elements"]) == 0:
                del data["elements"]
            if len(data["attributes"]) == 0:
                del data["attributes"]

Do you really want to do this? It seems to me that it'll make code harder to write that uses the data

            return data

These long function as inner functions smell bad. The suggest perhaps they should be in another class or something.

        schema = {}
        root = schema_data.getroot()
        children = root.getchildren()
        for child in children:
            c_type = getXSVal(child)
            if child.get("name") is not None and not c_type in schema:
                schema[c_type] = []

If the name is None, won't that cause the next line to have an error?

            schema[c_type].append(get_elements(child))

Instead use schema.setdefault(c_type,[]).append(get_elements(child)) it'll take care adding the list the first time you append.

        return schema

    def get_Types(self, t_name):

Python convetion is lowercase_with_underscores for method names

        types = []
        for t in self.schema[t_name]:
            types.append(t["name"])
        return types

I'd use return [t["name"] for t in self.schema[t_name]]

    def get_simpleTypes(self):
        return self.get_Types("simpleType")

    def get_complexTypes(self):
        return self.get_Types("complexType")


if __name__ == '__main__':
    fschema = open("schema.txt")

I suggest using with to make sure it gets closed

    schema = schema(fschema)

    print schema.get_simpleTypes()
    print schema.get_complexTypes()

My overall problem with your approach is that you are converting the xml schema into a bunch of unstructured dictionaries. The result isn't going to be much easier to work then the original XML objects. It's gonna be a real pain writing code to work with the schema representation you've produced. Without knowing the overall goal of why you are parsing the schema, I can't really suggest how to improve it.

On your new vesrion

    try:
        self.schema = self.create_schema(etree.parse(schemafile))
    except:
        print "Error creating Schema: Invalid schema file used"

NO! Don't catch the exception. Let the program just die. Never ever ever ever just print to the screen where an error happens.

Here my version of what you've done. 

from lxml import etree
from copy import copy
SCHEMA_SPACE = "{http://www.w3.org/2001/XMLSchema}"

class Schema:

    def __init__(self, schemafile):
        self.root = etree.parse(schemafile)

    def findall(self, path):
        return self.root.findall( path.replace("xs:", SCHEMA_SPACE) )

    def find(self, path):
        return self.root.find( path.replace("xs:", SCHEMA_SPACE) )

    def names_of(self, nodes):
        return [node.get("name") for node in nodes]

    def get_Types(self, t_name):
        return self.names_of( self.findall(t_name) ) 

    def get_simpleTypes(self):
        return self.get_Types("xs:simpleType")

    def get_complexTypes(self):
        return self.get_Types("xs:complexType")

    def get_elements_of_attribute(self, attribute):
        return self.names_of(self.findall(".//xs:element/xs:complexType/xs:" + attribute + "/../.."))

    def get_element_attributes(self, name): 

        node = self.find(".//xs:element[@name='" + name + "']")
        if node is None:
            node = self.find(".//xs:complexType[@name='" + name + "']")

        if node is None:
            return None
        else:
            return node.attrib


if __name__ == '__main__':
    with open("schema.txt") as f:

        schema = Schema(f)

        print schema.get_simpleTypes()
        print schema.get_complexTypes()
        print schema.get_elements_of_attribute("all")

        print schema.get_element_attributes("source")
        print schema.get_element_attributes("contact_id")

Copying all the data into python dictionaries is not helping. It is much easier to extract this information directly from the XML.

\$\endgroup\$
5
  • \$\begingroup\$ Thanks for all your suggestions. Stuff I agree and changed: camel case class names, starting exception, all code consolidation suggestions, probably classes for simpleType and simpleContent objects. Stuff I should explain more: This is the core data structure, I wrote a bunch of helper methods to quickly access all properties contained within to parse XML docs, produce flat files from them, validate database data according to the schema, and some other stuff. The dictionary/sometime distinction is done based on whether the object is a property of the root element or is a new element itself \$\endgroup\$ Commented Apr 18, 2012 at 15:38
  • \$\begingroup\$ Hopefully that explains most of the conditions, also why frequently "name" is ignored. The inner class function get_element had to be recursive and check all conditions but you are right, I could parse out some of the logic. You mentioned repeated code but I couldn't find another place where I use that conditional logic in this code segment, am I missing something obvious? I changed "get_Type" to "get_type", original used because the xsd standards for "simpleType" and "complexType" are written like that. Given this, do you have other suggestions on how I could better handle the parsing logic? \$\endgroup\$ Commented Apr 18, 2012 at 15:45
  • \$\begingroup\$ @MikeJ, re code-duplication: upon a second look, I just can't read. To give any better suggestions, I'd have to see how you are using this data structure you are constructing. \$\endgroup\$ Commented Apr 18, 2012 at 15:49
  • \$\begingroup\$ I added in more of the helper functions I had written and more example function calls to hopefully show a better idea of the how I'm using this data structure. \$\endgroup\$ Commented Apr 18, 2012 at 19:29
  • \$\begingroup\$ @MikeJ, see edit. Basically, your data structure isn't helping you perform any of those tasks. Its actually easy to extract from the original xml document. \$\endgroup\$ Commented Apr 20, 2012 at 17:37
2
\$\begingroup\$

Well, your code is amazing in that it proves you can handle the most commonly used 10% subset of XML Schema in a few dozen lines of code. But where are you planning to go with this? As you try to increase your coverage of XSD features, your design won't scale. Perhaps that doesn't matter - there's no harm in starting small and being prepared to refactor as it grows.

From my own experience of writing a schema processor, I would recommend keeping a close correspondence between the architecture of your code and the architecture of the spec. That is, implement a data structure corresponding to the schema component model described in the spec; have a process that constructs the SCM from the raw XML representation, and that enforces all the XML representation constraints; have another process that examines the SCM for consistency (enforcing the SCM component constraints), and then another process that's concerned with using the SCM for validation, including generating a finite state automaton or whatever. You can still do it incrementally, of course (e.g leave out named model groups, attribute groups, substitution groups, etc initially) but it's probably a good idea to get the overall structure in place early.

I would follow the XSD 1.1 spec rather than XSD 1.0, partly because the new features are nice for users, but also because the spec is much clearer - still dense, but now penetrable.

\$\endgroup\$
1
  • \$\begingroup\$ I think I am following a similar approach to what you are mentioning. I am writing a class and series of helper method to parse the schema itself, and then quickly access all underlying attributes. This is for a personal project, but I would like to try to make this go farther, so I think I might need to introduce classes for simpleContent, and other types for this to scale better. I think I am somewhat following your approach, I use this to parse XML docs, produce flat files from them, validate database data according to the schema, and a few other tasks. \$\endgroup\$ Commented Apr 18, 2012 at 15:13

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.