XQuery and Unit testing - Part II (xqUnit)

So I've started using MarkLogic as the basis for my unit testing framework, the choice being premised on this is the system I will be coding most of my XQuery against.

In terms of the framework (which I have named xqUnit) I am trying to base it as much as possible on the xUnit approach. I've been reading Gerard Meszaroses book in order to get a better understanding of this, and it is very well written:



Ultimately I want to have a test runner which is web based with a nice progress bar that goes green. I also want my test framework to be testable. However my first step is to write a spike to see how this problem could be solved:

(: first ever version :)
declare namespace xqunit = "http://xqunit.FigmentEngine.com/v0.0.1"

(: a function to take some method code and some test code, join them together and run them :)
define function xqunit:execute($method as node(), $test as node())
{
 let $code := string-join(($method/text(), $test/text()),
  codepoints-to-string(13))

 let $result :=
  try
  {
   <result> { xdmp:eval($code) } </result>
  }
  catch ($exception)
  {
   <result exception="{$exception/err:format-string}"> { $exception/err:stack/err:frame[1] } </result>
  }

 return <test name="{$test/@name}"> { $result } </test>
}

(: a test definition :)
let $test :=
 <test>
  <method>
   let $book:= /bib/book
  </method>
  <exercise name="count nodes">
   return count($book) = 4
  </exercise>
  <exercise name="find by name">
   return $book[1]/title = "TCP/IP Illustrated"
  </exercise>
  <exercise name="find by author">
   return count($book[author/last = "Stevens"]) = 2
  </exercise>
  <exercise name="find by author case sensitivty">
   return count($book[author/last = "stevens"]) = 0
  </exercise>
 </test>

(: exercise each test and show the results of the run :)
for $exercise in $test/exercise
let $result := xqunit:execute($test/method, $exercise)
return <testrun> { $result } </testrun>

The installation of MarkLogic includes some test data which is based on the XML Query Use Cases so I have used them as the basis for my initial datasets. This code shows my basic approach, which is:

• a xml schema (not defined yet) which defines a test with a method and a set of exercises (I'm need a better name for this)
• the framework takes the test method and combines the code in the test methd with the code in the exercise
• the framework executes the joined code (xdmp:eval is MarkLogic's dynamic method)
• this results in either a result or an exception if the code is invalid (this uses MarkLogics extension to XQuery in order to catch the exception). Catching exceptions like this is useful for picking up errors in the code, but could also be used for testing (think xqunit:expectException)
• this process is repeated for all exercises and output as a set of xml results

In principle this approach works, though a few concerns appear:

• the test, method and exercise are badly named, I need to align this better with xUnit
• the approach could be inefficient (the method gets re-executed each time)
• it looks like method is more of a setup and the exercises are the tests..
• the exercises have to return true or false for success, this could be more obvious if we had code like "return xqunit:assertEqual(count($book), 4)"

Reading the code, it seems obvious what a test is and what it is trying to do - so I am happy with that. The approach needs to consider two kinds of testing:

• data testing, where we are testing that the data in the database is correct
• function testing, where we are testing that a function works correctly

Some tests may fall into both camps, but I think it may be a useful way of loooking at the user stories for the framework.

This seems a good spike for now, so I've started the process of building a site http://xqunit.figmentengine.com/ and hopefully I can start to drive through a story based approach.