The declaration part of the query must contain a variable declaration for every variable mentioned in the conditions. Each variable declaration is enclosed in parentheses. The components of a variable declaration are separated by whitespace. The first component is an optional quantifier; the possible quantifiers are forall and exists, which have their usual logical meanings. The second component is the name of the variable. A variable name is a $ character followed by an arbitrary number of letters and digits, which can include underscore ( _ ) and language-specific characters. The final component is an optional type restriction. This is either a simple string expressing the type of a node in the NOM or a disjunction of these types separated using the pipe symbol (|).

($a)

($a word)($b sentence)

($a word)($b word)

($a word | phrase | sentence)

($b sentence)(forall $a word)

Formal definition:

	declarations     :=  declarations var_declaration
	declarations     :=  var_declaration
	var_declaration  :=  ( variable )
	var_declaration  :=  ( variable typedeclaration )
	typedeclaration  :=  types
	typedeclaration  :=  type
	types            :=  types | type

Grammar for forall/exists missing. We want to use some rule system with optionality or this is going to take forever - check what docbook supports?

The condition part is a Boolean expression over property tests, structural relations, and temporal relations. Parentheses are only needed if a lower precedence relation should be executed first. The strongest binding operator is negation ( ! ). The operators are listed in the order of their precedence below:

For convenience, there are a number of syntactic literals for each operator. The different forms for the same operator are completely synomynous. The implication operator -> is the weakest binding operator and, again, is provided for convenience; $a -> $b is logically equivalent to !a | b.

Formal definition:

	match_condition    := 
	match_condition    := ( match_condition )
	match_condition    := ! match_condition
	var_declaration    := match_condition & match_condition
	var_declaration    := match_condition | match_condition
	var_declaration    := match_condition -> match_condition
	var_declaration    := property_test
	var_declaration    := structural_relation
	var_declaration    := time_relation

A property may be an attribute value, a constant, or the result of a function applied to an element.

The query language contains a number of functions that take elements and return some property of the element. These functions have to do with properties that are special in NXT: timing, identification, and textual content. Function names can be given in upper or lower case, and the operators = and == are synonymous. Where the data storage format for a corpus stores these properties as XML attributes, they can also be queried using the form variable @attribute. The functional form is preferred because it is the same across all data sets and leaves the user in no doubt that these are the attributes holding the special properties.

The functions are as follows:

TEXT($w)

ID($w)

TIMED($w)

START($w)

END($a)

DURATION($a)

CENTER($a)

Formal definition:

	property    := " number_or_string " 
	property    := variable @ attribute
	property    := TEXT(  variable )
	property    := ID( variable )
	property    := TIMED( variable )
	property    := START( variable )
	property    := END( variable )
	property    := DURATION( variable )
	property    := CENTER( variable )

Property existence tests check for the existence of some property.

$w@pos

TIMED($a)

START($a)

END($a)

TEXT($a)

It is actually possible to test for the existence of any property, but the other possible existence tests are not useful; ids, string, and numbers always exist, and duration and center properties exist for elements that are timed.

Formal definition:

	property_test    := variable @ attribute
	property_test    := TIMED( variable )
	property_test    := START( variable )
	property_test    := END( variable )
	property_test    := TEXT( variable )

The next set of property tests compare the equality and order of two values. Property expressions are weakly typed. The value resulting from an expression will be interpreted as a floating-point number when possible. If it cannot be converted into a number or if the value is compared to a pattern given by a regular expression, the value will be treated as a string. A number is always unequal to a string. Strings are themselves alphabetically ordered, and are case-sensitive. Strings starting with upper case letters are less than strings with upper case letters. As a result, "2" == "2.0" is true, while "Two" == "two" is false.

($x): $x@cat=="NP"

($x)($y): $x@cat==$y@cat

($x)($y): $x@cat==$y@cat & $x!= $y

Formal definition:

	property_test    := property  ==  property
	property_test    := property  !=  property
	property_test    := property  <  property
	property_test    := property  >  property
	property_test    := property  <=  property
	property_test    := property  >=  property

The final set of property tests compare string values against regular expressions. Regular expressions are enclosed by slashes (/). NXT's regular expression implementation uses Java regular expressions underneath, so it is not possible to give a definitive syntax for the patterns here, instead, see the Java 1.5 Regular Expression Documentation or equivalent documentation for your Java version.

($a): text($a) ~ /th.*/

($a): text($a) ~ /[dD](as|er)/

($a): text($a) ~ /.+([0-9A-Z])+.*/

($a): text($a) ~ /\.*/

Formal definition:

	property_test    := property ~ / pattern / 
	property_test    := property !~ / pattern / 

The simplest structural relation asserts the identity or non-identity of two elements. Since the default evaluation strategy allows different variables to be bound to the same element, the != operator is sometimes necessary to exclude unwanted results. The == operator is less useful and was mainly added for the sake of symmetry.

	structural_relation    :=    variable == variable
	structural_relation    :=    variable =! variable

The basic structural relation is the dominance relation ^. To describe that an element a dominates an element b the dominance operator ^ is be used. In other words a is an ancestor of b.

	structural_relation    :=   variable ^ variable
	structural_relation    :=   variable ^ distance variable

Two elements are in a precedence relation if they have a common ancestor element, which can be a normal element or the root element of a layer. An element $x precedes another element $y if some ancestor of $x (or $x itself) is a preceding sibling of some ancesor of $y (or $y itself).

	structural_relation    :=    variable <> variable

Some examples:

($a)($b): $a ^ $b & $a != $b

($s syntax)($w word): $s ^1 $w

($a)($b): $a ^0 $b

($a)($b): $a ^-2 $b

($a word)($b word): $a <> $b

There is a bug when querying over multiple observations - the implementation considers times in different observations to be comparable, so that it's possible to get the result that an element in one observation is before some element in another. This is easy to get around: query on one observation at a time, or declare the reserved attribute for observation names for your corpus and add a test for the same observation as an extra query term - e.g.($f@obs = $g@obs), if the attribute declared is obs.

The search GUI (whether called stand-alone or from a search menu) can't display results if some subquery in a complex query only has query matches that are bound with forall - e.g. ($f foo):($f@att="val")::(forall $g bar):!($g ^ $f)

The immediate precedence operator is missing. Immediate precedence is equivalent to ($f foo)($g foo)(forall $h foo): ($f<<$g) &amp;&amp; (($h=$f) || ($h=$g) || ($h<<$f) || ($g<<$h))

but, of course, this is cumbersome and can be too slow and memory-intensive for practical purposes, depending on the data set. Some common uses of the operator are covered by the NGramCalc utility.Another work-around is to create one XML tree from the NXT data thatrepresents the information required and query it using XPath. Export to LPath and tgrep2 would also be reasonable and are not difficult to implement. If you need to match on regular expressions of XML elements in order to add markup, (so, for instance, saying "find syntactic constituents with one determiner, followed by one or more adjectives, followed by one noun, and wrap a new tag around them"), but you can always use something like fsgmatch (from the LTG; new release, currently in beta, is called lxtransduce) and then modify the metadata to match. Remember, the data is just XML, amenable to all of the usual XML processing techniques.

The arithmetic operators are missing.

At present, users who need them add new attributes to theirdata set and then carry on as normal. For instance, a researcher looking at how often bar elements start in the 10 seconds after foo elements end might add an "adjusted start" attribute to bar elements that take 10 secondsoff their official start times, and then use the query ($f foo)($b bar):(START($b) > END($f)) && ($b@adjustedstart < END($foo))

This stylesheet, run on a specific individual coding in the context of the MONITOR project, is an example of how this can be done. It just copies everything, adding new attributes to feedback gaze codes. We used this general technique on the Switchboard data to get lengths for syntactic constituents, and on the Monitor data to get durations.

This method is inconvenient, particularly for the sort of exploratory study that wishes to consider several different time relationships. We don't think it is worth adding special loading routines that addtemporary attributes for adjusted start and end times, but we could include some utilities for command line searching based on adjustments passed in on the command line. For instance, java CountWithTimeOffset -q '($t turn)($f feedback):($t # $f)' -t feedback -d 50 could mean to count overlaps after feedback elements have been displaced 50 seconds forward. We are considering whether this would be useful enough to supply.

At present (Apr 05) the query language parser fails to handle namespacing properly, so any elements and attributes that are namespaced will be difficult to work with. For the timing and id attributes, where the default names are in the nite: namespace, this doesn'tmatter, since they are exposed to query via e.g. START($x), but namespacing other tags and attributes would make working with them difficult until this is fixed.

NXT's query engine is slow and uses a great deal of memory. For instance, some of our more complicated syntactic querieson the Switchboard corpus take 10 seconds per dialogue, or over an hour and a half for the entire corpus.

This is partly a consequence of what it does - the query languageis solving a harder problem than languages that operate on trees and/or are limited in their use of left and right context. It istrue that the current implementation is not fully optimized, but this is not something we intend to look at in the immediate future. Our first choice strategy for addressing this problem is to look at mapping NQL queries to XQuery for implementation, and addition of the missing operators, that way. Meanwhile, most of NXT's users are not actually engaged in real-time processing, and find that if they develop queries on a few observations using a GUI, they can then afford to run the queries at the command line in batch. The more they are interested in sparse phenomena, the less suitable this strategy is. For some query-based analyses, it is also useful to consider direct implementation using the NOM API, since the programmer can optimize for the analysis being performed.

Meanwhile, an hour and a half is OK for batch mode, but some of our queries areso common that we really want easy access to the results. We can get this by indexing. Using indices rather than the more complex syntactic queries makes querying roughly ten times faster. This will be even faster if one then selects not to load the syntax at all, which is possible if one doesn't need it for other parts of the subsequent query. You can choose not to load any part of the data by commenting out the <coding-file> tag for it in your local copy of the metadata file, or after NXT 1.3.0, by enabling lazy loading in your applications.

It's faster to use string equality than regular expression matching in the query language, and keep in mind the regular expressions have to match the entire string they are compared against, not just a substring of it.

The very desperate can write special purpose applications to evaluate their queries, which is faster especially for queries involving quantification. For instance,one user has adapted CountQueryResults to run part of the query he wants, but instead of returning the results, then checks the equivalent of hisforall conditions using navigation in the NOM.