gembin

Abstract

The XQuery 1.0 specification describes a powerful language for accessing and manipulating XML. In its initial form XQuery lacks two interesting features: fast token-based text search and node-level update. Specifications for these features are under development at the W3C with release dates after XQuery 1.0. This paper looks at the mechanisms by which XQuery can support search and update based on our experience at Mark Logic implementing these features.

Are Search and Update Needed?

To begin, let us ask the question, are search and update capabilities needed in an XQuery environment? The answer depends on the use case. When using XQuery to federate queries against data-oriented RDBMS stores, text-based search isn't required and updates can be handled through direct SQL calls. The XQuery in this situation acts as a read-only view, a convenience rather than a primary access and manipulation language.

In other use cases the need can be acute. Take for example O'Reilly Media, a Mark Logic customer with a custom publishing site called SafariU that's targeted at helping university professors create custom course readers. Professors access book and article content (held as Docbook within Mark Logic) by performing advanced searches to locate items of interest -- chapters, sections, or code examples. Professors gather together these items into a new book, reviewing HTML and later PDF copies, and eventually receiving a stack of their custom books at the university bookstore.

SafariU requires fast text search so professors can find content to select and update to track the new hierarchical book structure and content usage for royalty reports. In fact, before the first professor visits SafariU, O'Reilly uses XQuery update calls to "decorate" the book and article content: adding <indexterm> elements on articles, adding estimated page counts to content pieces, etc.

Mark Logic has opted to provide search and update using function calls -- rather than language extensions -- so as to conform to the XQuery standard as much as possible. We have been able to make search and update calls efficient, executing quickly even against gigabyte documents in terabyte collections. In the rest of this paper we'll explain the Mark Logic approaches to search and update and compare and contrast that approach with the preliminary W3C proposals.

Section Title

If XQuery search were to provide only what a typical search engine provides, it would be fairly easy. You specify the search word or phrase, possibly add a basic document selection constraint, and retrieve a relevance ordered list of documents containing the search term(s). Using the Mark Logic search function calls and the input() function to return a sequence of all document nodes, the expression would look like this:

 

cts:search(input(), "word or phrase")

 

Yet XQuery enables so much more. First, XQuery has direct access to and can utilize the XML document structure. This lets you determine what subset of content should be searched. As shown in Example 1, you can look for the most relevant <sect1> blocks discussing "cougars" where the <sect1> blocks must be within chapters about "veterinary medicine" in books published after 2003.

 

cts:search(

input()/book[meta/pubdate >= 2003]

       

//chapter[cts:contains(title, "veterinary medicine")]//sect1,

"cougars

)

 

Second, XQuery can access the items returned from each search and programmatically manipulate each item for rendering or to feed into another subsequent search. In the cougar example, the search expression might return a sequence of <sect1> elements, then the query might continue on to extract all the figure blocks within those sections for display, along with any paragraph linking to the figure. Example 2 shows what that query would look like, and how a theoretical print-references() might be implemented:

 

for $s in

cts:search(...)[1 to 10]

for $f in $s//figure

return (

print-figure($f),

print-references($f)

)

 

define function print-figure($f as element(figure)) { ... }

 

define function print-references($f as element(figure))

{

let $id := $f/@id

for $link in $f/ancestor::book//link[@linkend=$id]

return

<p>

 

{ string-join($link/ancestor::para[1]//text(), " ") }

</p>

}

 

The power to control what's searched and what's returned proves especially valuable when documents have useful semantic structure. Figure 1 shows a screen shot of a query for "tennis elbow" executed against a collection of medical textbooks:

The results were rendered with custom XQuery logic. Start at the most relevant <para> elements, walk up the document hierarchy to the nearest containing section, then render that block as HTML. Clickable breadcrumbs were added based on the full ancestral history of the matching <para>. The search returns paragraphs, but for the end result we've opted to include full sections.

By comparison, a search engine presented with a large number of Docbook files would return simply URL pointers to which files were most relevant. Any post processing would be entirely manual and external to the search engine.

input()/book[cts:contains(title, "java servlet")]

     

//example[cts:contains(title, "black and white")]

    

//programlisting

 

 

cts:search(

input()/book[meta/pubdate >= 2003]

      

//chapter[cts:contains(title, "veterinary medicine")]//sect1,

"cougars"

)[1 to 10]

 

 

//title[cts:contains(., "XQuery search")]

 

 

cts:search(//title, "XQuery search")

 

 

(: Simple strings are treated as

cts:word-query() :)

cts:word-query("XQuery search")

 

(: How to use a

cts:word-query() :)

cts:search(input(),

cts:word-query("rhinos"))

 

(: Indicates to use stemming so "searching" would be a match also :)

cts:word-query("XQuery search", "stemmed")

 

(: Either wine color is acceptable, red 2x preferred :)

cts:or-query(

cts:word-query("white wine", "stemmed", 0.5),

cts:word-query("red wine", "stemmed", 1.0)

)

 

(: Both explorers should appear :)

cts:and-query(

cts:word-query("Lewis", "case-sensitive"),

cts:word-query("Clark", "case-sensitive")

)

 

(: Same as previous. Capitalized words trigger case sensitivity. :)

cts:and-query(

cts:word-query("Lewis"),

cts:word-query("Clark")

)

 

(: Look for "spring" with relevance weighting by location :)

cts:or-query(

cts:element-word-query(xs:QName("title"), "spring", (), 1.0),

cts:element-word-query(xs:QName("subtitle"), "spring", (), 0.3),

cts:element-word-query(xs:QName("para"), "spring", 0.1)

)

 

(: Use

cts:near-query() to specify a term distance :)

cts:near-query(

(cts:word-query("white wine", "stemmed", 0.5),

cts:word-query("red wine", "stemmed", 1.0)),

2)

 

 

for $x in

cts:search(//para, "spring break")

return <p score="{cts:score($x)}"> { $x/* } </p>

 

Relation of Mark Logic Search to the W3C FTTF Activity

The Free Text Task Force of the W3C XQuery Working Group has published a draft recommendation for XQuery text search. The recommendation describes a surface syntax for XML search which integrates with the existing XQuery grammar. The basic component of this syntax is the "ftcontains" predicate relation. This predicate maps very closely to the Mark Logic cts:contains() predicate. For example, the FTTF proposed search expression:

 

/books/book[title

ftcontains ("dog" with stemming) && "cat"]/author

 

maps directly to the Mark Logic syntax

 

/books/book[cts:contains(title,

 

cts:and-query((cts:word-query("dog","stemmed"),"cat")))]/author

 

The Mark Logic search function API acts effectively as an implementation API for the W3C draft recommendation. It would be premature to start implementing the FTTF grammar given the lack of maturity and large number of unresolved issues in that recommendation.

The FTTF grammar also uses the "ftcontains" predicate relation as an integral part of the FLWOR expression. For example, the set of author children of book elements, where the book element has a title child containing both "dog" and "cat":

 

for $b in //book

where $b/title

ftcontains "dog" && "cat"

return $b/author

 

maps to the Mark Logic function API:

 

for $b in

cts:search(//book,

 

cts:element-query(QName("title"),

cts:and-query(("dog","cat"))))

return $b/author

 

A key difference, however, is that the Mark Logic cts:search() function returns results in (decreasing) relevance order by default, whereas the FLWOR expression returns results in document order unless an explicit order by clause is included in the expression. The fully equivalent FTTF expression would be:

 

for $b in //book

score $s as $b/title

ftcontains "dog" && "cat"

where $s > 0

order by $s descending

return $b/author

 

which is surprisingly more verbose than the lower-level Mark Logic implementation. De-coupling the search and the relevance expression is potentially very useful, but in practice the two expressions are almost always equal. The EBNF grammar allows that the right-hand-side of the "ftcontains" relation may be any expression, but the specification restricts the expression to be a Boolean combination of ft expressions.

 

You <b>phrase through</b> a bold tag.

 

 

A GPS <footnote>Global Positioning System</footnote> device should match a search for "GPS device".

 

Mark Logic Update

Mark Logic provides a handful of functions to enable document-level and node-level update. A single query can execute any number of these functions.

Three basic functions perform document-level updates. These functions all use the Mark Logic standard "xdmp" namespace prefix, as shown in Example 6.

 

(: Loads a document from the

filesystem to the specified URI.

The URI is what would be used as a parameter to

fn:doc().

 

Replaces any existing document at that URI. :)

xdmp:load($filepath, $uri)

 

(: Loads a dynamically constructed document or element node :)

xdmp:document-insert($uri, $root as node())

 

(: Deletes the specified document :)

xdmp:document-delete($uri)

 

Three other functions provide node-level insert abilities. Through the three basic calls shown in Example 7 a node can be added at any location:

 

(: Inserts the new node as the last child of the parent :)

xdmp:node-insert-child($parent as node(), $new as node())

 

(: Inserts the node as a following sibling to the reference node :)

xdmp:node-insert-after($sibling as node(), $new as node())

 

(: Inserts the node as a preceding sibling to the reference node :)

xdmp:node-insert-before($sibling as node(), $new as node())

 

A final pair of functions in Example 8 provide the rest of the node-level updates:

 

(: Replace the old with the new :)

xdmp:node-replace($old as node(), $new as node())

 

(: Delete the node (and its descendents) from the document :)

xdmp:node-delete($old as node())

There are additional methods to get and set document properties (used for WebDAV), to add and remove documents from collections, and to alter document permissions. Example 9 utilizing several of these calls appears below. It's an implementation of a named queue:

 

define function

enqueue($name as xs:string, $i as item()) as empty() {

(: wrap $i in an element() so it can be added to a doc,

: whether it's already an element or not: works for attributes, etc.

:)

let $n := <entry>{$i}</entry>

let $queue := doc($name)/queue

return

if (exists($queue))

  

then

xdmp:node-insert-child($queue, $n)

  

else

xdmp:document-insert($name, <queue>{$n}</queue>)

}

 

define function

dequeue($name as xs:string) as item()? {

(: get the first child element under the queue root, and unwrap it :)

let $node := doc($name)/queue/entry[1]

return

 

if (exists($node))

    

then (xdmp:node-delete($node), $node/@*, $node/node())

    

(: if there's nothing on the queue, return the empty sequence :)

    

else ()

}

 

Everything is straightforward in this example except perhaps the return value from dequeue(). The "then" clause returns a sequence of three items: the result of the xdmp:node-delete() call (always empty), the $node/@* (any possible attribute on the <entry> in case the item stored was an attribute), and $node/node() (any nodes under the <entry> in case the item stored was an element, PI, text node, etc).

To conform to the spirit of the XQuery specification in disallowing side effects, updates are not made visible to the query environment executing the update. This requires some getting used to, but doesn't hinder productivity.

Updates occur after the query completes without error. Should a query produce an uncaught error, the set of updates from that query do not occur. Mark Logic does not yet provide explicit commit/rollback support. However, Mark Logic provides a try/catch expression so that errors can be caught and processed while allowing execution to continue. Its syntax:

 

try {

$try-expression

}

catch ($exception as element()) {

$catch-expression

}

 

Example 10 loads files from a list of filenames, reporting errors but not letting any errors stop the other files from being loaded:

 

for $file in $files

return

try {

  

<li>

 

{ xdmp:load($file, $file) }

 

{ $file } loaded

  

</li>

}

catch ($exception) {

  

<span>

    

Problem while loading { $file }:<br/>

    

<i xmlns:e="http://marklogic.com/xdmp/error">

      

{$exception/e:message/text()}

    

</i>

  

</span>

}

 

Conclusion

Interested parties can download evaluation and community licensed copies of Mark Logic Server from http://xqzone.marklogic.com. Full function documentation for Mark Logic search and update functions can be found at http://xqzone.marklogic.com/pubs/.

Biography

Jason Hunter

Mark Logic

Jason Hunter works as a Lead Applications Engineer at Mark Logic. He's author of Java Servlet Programming (O'Reilly). He's a member of the expert groups responsible for Servlet, JSP, JAXP, and XQJ API development, and has assisted the W3C XQuery Working Group. He is also an Apache Member and as Apache's representative to the Java Community Process Executive Committee he established a landmark agreement allowing open source Java. He co-created the open source JDOM library to enable optimized Java and XML integration.