This work is licensed under a Creative Commons Attribution 3.0 Unported License (including images & stylesheets). The source is available on Github.

What Version of Clojure Does This Guide Cover?

This guide covers Clojure 1.4 and Leiningen 2.x.


Try as you might, XML is difficult to avoid. This is particularly true in the Java ecosystem. This guide will show you how to parse XML with the minimum amount of pain using the excellent tools available in Clojure.

Parsing NZB files

For the purpose of the tutorial I have chosen a simple and fairly well known XML file format: NZB. An NZB file is used to describe files to download from NNTP servers. In this tutorial we will take a basic NZB document and turn it into a Clojure map.

Let us start by creating a new project (for details on using Leiningen, see this guide:

$ lein new nzb

Now edit project.clj to contain the following:

(defproject nzb "0.1.0-SNAPSHOT"
  :description ""
  :url ""
  :license {:name "Eclipse Public License"
            :url ""}
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [org.clojure/ "0.1.1"]])

We are including a dependency on, which is a "system for filtering trees, and XML trees in particular".

Make a dir called dev-resources at the root of your project, and create a file named example.nzb inside of it. This will be the file we use to test our code (taken from wikipedia). dev-resources is by convention the location to store file resources you use during development / testing.

Put the following XML in the example.nzb file:

<?xml version="1.0" encoding="iso-8859-1" ?>
<!-- <!DOCTYPE nzb PUBLIC "-//newzBin//DTD NZB 1.1//EN" ""> -->
<nzb xmlns="">
   <meta type="title">Your File!</meta>
   <meta type="tag">Example</meta>
 <file poster="Joe Bloggs &lt;bloggs@nowhere.example&gt;" date="1071674882" subject="Here's your file!  abc-mr2a.r01 (1/2)">
     <segment bytes="102394" number="1"></segment>
     <segment bytes="4501" number="2"></segment>

Note The eagle eyed among you will notice that I have commented out the DOCTYPE declaration, as this causes an Exception to be thrown. I will show you how to get around this towards the end of the tutorial.

Let's write a high level test to illustrate more clearly what we are trying to do. Open up the test/nzb/core_test.clj file and make enter the following:

(ns nzb.core-test
  (:use clojure.test
  (:require [ :as io]))

(deftest test-nzb->map
  (let [input (io/resource "example.nzb")]
    (is (= {:meta {:title "Your File!"
                   :tag "Example"}
            :files [{:poster "Joe Bloggs <bloggs@nowhere.example>"
                     :date 1071674882
                     :subject "Here's your file!  abc-mr2a.r01 (1/2)"
                     :groups ["alt.binaries.newzbin"
                     :segments [{:bytes 102394
                                 :number 1
                                 :id ""}
                                {:bytes 4501
                                 :number 2
                                 :id ""}]}]}
           (nzb->map input)))))

This should be fairly self-explanatory, I have directly translated the XML into Clojure data structures of maps and vectors. If we were to just use the clojure.xml library to parse the NZB file, we get a tree based representation. For example:

(-> "example.nzb" io/resource io/file xml/parse)
{:tag :nzb,
 :attrs {:xmlns ""},
 [{:tag :head,
   :attrs nil,
   [{:tag :meta, :attrs {:type "title"}, :content ["Your File!"]}
    {:tag :meta, :attrs {:type "tag"}, :content ["Example"]}]}
  {:tag :file,
   {:poster "Joe Bloggs <bloggs@nowhere.example>",
    :date "1071674882",
    :subject "Here's your file!  abc-mr2a.r01 (1/2)"},
   [{:tag :groups,
     :attrs nil,
     [{:tag :group, :attrs nil, :content ["alt.binaries.newzbin"]}
      {:tag :group, :attrs nil, :content ["alt.binaries.mojo"]}]}
    {:tag :segments,
     :attrs nil,
     [{:tag :segment,
       :attrs {:bytes "102394", :number "1"},
       :content [""]}
      {:tag :segment,
       :attrs {:bytes "4501", :number "2"},
       :content [""]}]}]}]}

That's great, and can sometimes be enough. But I would rather work with the representation I have in the test. To do that, we need a way of traversing this tree and picking out the pieces of information we require. The and libraries are perfect for this. The documentation for the library on github is nice, but it initially left me a little confused as to how to go about using the library (not being familiar with zippers).

A Simple Example

Zippers allow you to easily traverse a data structure. Let's play with it in a REPL and start with the root node of our NZB file:

(require '[ :as io])
(require '[clojure.xml :as xml])
(require '[ :as zip])
(require '[ :as zip-xml])

(def root (-> "example.nzb" io/resource io/file xml/parse zip/xml-zip))

Now we have a zipper for the root element of our document, we can start traversing it for information. The two main functions we will use for this are xml-> and xml1->. The former returns a sequence of items based on the predicates given to it, the latter returning the first matching item. As an example, let's get the meta data from the NZB document root and create a Clojure map:

(into {}
      (for [m (zip-xml/xml-> root :head :meta)]
        [(keyword (zip-xml/attr m :type))
         (zip-xml/text m)]))
;; => {:title "Your File!", :tag "Example"}

A couple of things are happening here. First of all we use xml-> to return a sequence of <meta> tags that live under the <head> tag:

(zip-xml/xml-> root :head :meta)

We use the for list comprehension macro to evaluate each item in the sequence. For each item we find the contents of the :type attribute using the attr function:

(keyword (zip-xml/attr m :type))

This returns us the contents of the attribute as a string, which we turn into a keyword to use as the key in the map. We then use the text function to get the textual contents of the meta tag:

(zip-xml/text m)

We make a tuple of these values, and pass the resulting sequence to into to build the map.

Putting It Together

Using only these functions, we can parse the raw XML into the Clojure data structure from our unit test. If you like, open ./src/nzb/core.clj, and make the changes as you read along.

First let's define our nzb->map function from the test, and pull in the code we have already written for parsing the metadata of the NZB:

(ns nzb.core
  (:require [clojure.xml :as xml]
            [ :as io]
            [ :as zip]
            [ :as zip-xml]))

(defn meta->map
  (into {}
        (for [m (zip-xml/xml-> root :head :meta)]
          [(keyword (zip-xml/attr m :type))
           (zip-xml/text m)])))

(defn file->map
  ;; TODO

(defn nzb->map
  (let [root (-> input
    {:meta  (meta->map root)
     :files (mapv file->map (zip-xml/xml-> root :file))}))

The only new thing here is the use of io/input-stream to allow us to use anything as input that the io/input-stream supports. These are currently OutputStream, File, URI, URL, Socket, byte array, and String arguments. See the docs for details.

Now let's fill in the file->map function:

(defn segment->map
  {:bytes  (Long/valueOf (zip-xml/attr seg :bytes))
   :number (Integer/valueOf (zip-xml/attr seg :number))
   :id     (zip-xml/xml1-> seg zip-xml/text)})

(defn file->map
  {:poster   (zip-xml/attr file :poster)
   :date     (Long/valueOf (zip-xml/attr file :date))
   :subject  (zip-xml/attr file :subject)
   :groups   (vec (zip-xml/xml-> file :groups :group zip-xml/text))
   :segments (mapv segment->map
                   (zip-xml/xml-> file :segments :segment))})

Again, nothing new. We simply pick out the pieces of the document we wish to process using a combination of the xml1->, xml->, attr, and text functions. Run the test, and it should pass.

Prevent Parsing the DTD

Interestingly, if we uncomment the DTD declaration in the example.nzb file, our code now explodes with an Exception:

org.xml.sax.SAXParseException: The markup declarations contained or pointed to by the document type declaration must be well-formed

We can fix this by swapping out the SAXParserFactory and setting a feature to not validate the DTD. Here's how:

Update the ns declaration to include some required classes:

(ns nzb.core
  (:require [clojure.xml :as xml]
            [ :as io]
            [ :as zip]
            [ :as zip-xml])
  (:import (javax.xml.parsers SAXParser SAXParserFactory)))

Define a function to switch out the SAXParserFactory:

(defn startparse-sax
  "Don't validate the DTDs, they are usually messed up."
  [s ch]
  (let [factory (SAXParserFactory/newInstance)]
    (.setFeature factory "" false)
    (let [^SAXParser parser (.newSAXParser factory)]
      (.parse parser s ch))))

Update our nzb->map definition to use it:

(defn nzb->map
  (let [root (-> input
                 (xml/parse startparse-sax)
    {:meta  (meta->map root)
     :files (mapv file->map (zip-xml/xml-> root :file))}))

Yay, our test passes again.

Query Predicates

There are a few other useful functions in the ns we haven't yet looked at, namely: text=, attr=, and tag=. These functions allow you to construct query predicates to run against a given node. As an example, let's pull out the first file segment from the example.nzb file using the attr= function:

(zip-xml/xml1-> root
                (zip-xml/attr= :number "1"))

From the root node of the document we reach down into :file, :segments, and :segment in turn, then use the attr= query predicate to match a :segment with a value of "1".

Interestingly enough, the other two query predicates have shortcuts for their use. You have already been using the tag= query predicate every time you use a keyword to locate a tag. To use the text= predicate easily, just use a string. For example, to retrieve the second :segment based on its content of

(zip-xml/xml1-> root
;; ... the resulting node

Finally, you can combine these query predicates to match multiple things on a given node by using a vector:

(zip-xml/xml1-> root
                [(zip-xml/attr= :number "1")
                 (zip-xml/attr= :bytes "102394")]

Here we are matching on both the :number attribute being "1", and the :bytes attribute being "102394". Obviously, you can use strings here to match against content too.

Creating New Predicates

OK, now let's suppose we want to use some kind of numerical comparison in our XML (like we might do with XPath). As it stands, we have no way to do that with the built-in functions but we can easily define our own.

Let's start with a general function for comparing attribute values:

(defn attr-fn
  [attrname f test-val & [conv-fn]]
  (fn [loc]
    (let [conv-fn (or conv-fn identity)
          val (conv-fn (zip-xml/attr loc attrname))]
      (f val test-val))))

This function takes an attribute name (attrname), a function for making a comparison (f), a value to test agains (test-val) and optionally a conversion function. Imagine our example.nzb file had 100 segments, and we only wanted to get segments over 75. We could now achieve this using our general function:

(zip-xml/xml-> root
               (attr-fn :number > 75 #(Long/valueOf %))

Let's provide a helper for this to make the syntax clearer:

(defn attr>
  [attrname val]
  (attr-fn attrname > val #(Long/valueOf %)))

(zip-xml/xml-> doc
               (attr> :number 75)

We could build a whole suite of helper functions for examining XML nodes, if we are unlucky enough to be required to do so :)


I hope these simple examples have given you an idea of the ease with which you can process XML using Clojure, and how simple it is to extend the tools already provded in interesting directions.


Gareth Jones, 2012 (original author)