NIEMatron

Editorial: Tablets in the Public Sector - Part 1

2011-03-18T11:53:00.001-07:00

As 2011 has been termed by some in the media as "The Year of the Tablet," (Fox, CIO & NYT) this is the first part in a series of articles meant to articulate the role Tablet PC's could potentially take in the Public Sector and how our NIEM community might react to this trend.

While not directly related to Schematron or XSLT, we would be remiss in ignoring modern trends in our application of these technologies. The fact is, that unless our tools and technologies are adapted to support these trends, they risk being left behind.

This first article will focus on the Tablet and the various roles it can potentially take in the Public Sector. Some in the media have simply divided the device's role into two oversimplified categories of Content Creation and Content Consumption. While it is true, devices of any type can do these two things to differing degrees, it is probably easier to understand Tablet technologies in terms of actual public sector environment use cases.

In the Boardroom

In meetings the Tablet most often replaces the paper meeting handouts. In one large city they estimated that over $1m per year could be saved in paper and printing costs by simply eliminating meeting handouts in favor of electronic meeting materials. An electronic medium also makes it easy to store, preserve and reexamine materials presented at a later date or by those not who were unable to physically attend the meeting.

An example of where this use case is already becoming a reality is in the NLETS board, where each of the board members is issued a tablet upon which they are provided meeting materials and access to any other critical information prior to and during each meeting.

It is also possible for the Tablet to take the role of the "notebook" or the medium where one takes notes about the meeting, however more often than not, the lack of a stylus or even keyboard (barring separate accessories) for rapid data entry makes this impractical at this point in time.

Current Advantages	Current Disadvantages
- Size & Weight - Paper Cost Savings - Ease of Later Referencing - Ease of Real-Time Update	- Poor Annotation Capabilities

In the Courtroom

There are a number of situations where paper was the predominant medium for hundreds of years only to very recently be replaced with more electronic options. In the late 1990's, attorney's began to adopt Notebook PC's yet they have never been fully accepted in that environment. Many courtrooms were never built with a PC in mind therefore the lack of power outlets, desk space and network connectivity quickly became impediments.

With the advent of Tablet technology, there is a renewed interest in this technology by defense and prosecution alike. An example trial was recently documented by the Mac Litigator where it highlighted the use of an Tablet in a four day Jury Trial. Many of the issues with a laptop are overcome by the use of an unobtrusive, always connected, long-lasting Tablet.

Current Advantages	Current Disadvantages
- Size & Desk Footprint - Paper Cost Savings - Battery Life - Real-Time Updates - Real-Time Reference	- Poor Data Input Capabilities - Possible Courtroom Distraction - Enterprise Security Risk

In the Field

The public sector employs a large number of individuals involved in on-site investigations of various kinds. Whether it is the city building inspector, a fire marshal or a police detective, they all have similar needs to gather and process data in the field. For years now, investigators of this kind have been using laptop computers to address this need, however this technology has always been relegated to the car or the office as it would get in the way of the investigator at the work site. Tablets seem to fit this niche well as their size, weight and connectivity is better suited to the job.

Some agencies are already beginning to investigate and adopt Tablet technology in this manner. The Knox County Sheriff replaced a number of their detectives' laptops with Tablets as described by an article on the NLETS website.

Wireless Tablet technology coupled with GPS, maps and satellite imagery makes Tablets extremely relevant to those public sector employees working in the construction field. With a Tablet, a building inspector or DOT foreman can walk onto a job site and determine where there may be troubles quickly and easily. These can also give real-time access to building codes and municipal ordnances as well as architectural images, plans and blueprints.

Current Advantages	Current Disadvantages
- Size & Weight - Real-Time Reference - Real-Time Transmission - Hardware Price Point	- Poor Data Input Capabilities - Lack of "Rugged" Models - Enterprise Security Risks

In the Classroom

It seems like whenever the topic of Tablets or eBooks is brought up, a spotlight is placed on the classroom and the typical college student hefting around hundreds of pounds worth of textbooks. While on the surface and in 10 second sound bites, this makes sense; the classroom setting has experienced significant struggles making Tablets and eBooks a reality in the classroom. A recent article by the Chronicle for Higher Learning points out a number of the difficulties experienced across several different institutions.

Regardless of the problems currently being experienced, as the they are addressed, the classroom is a very likely place for Tablet technology to take a strong future foothold. As a Melbourne Trinity College study points out, even with the disadvantages, the usage of Tablets are still expected to increase at the University level in the upcoming years.

Current Advantages	Current Disadvantages
- Real-Time Learning Feedback - Size & Weight - Battery Life - Hardware Price Point	- eTextbook Prices Still High - Poor Annotation Capabilities - Potential Classroom Distraction

In summary, Tablets can be seen and used in a number of environments within the public sector and one should not be surprised as they begin to become less of a consumer toy and more of an industry tool.

Please bear in mind that we fully acknowledge that the editorial opinions stated in the article above are based on industry generalizations and we fully understand that for every generalization or rule there is an exception.

Editorial: Growing Support for Asynchronous Transactions

2010-10-29T11:06:00.001-07:00

It's always exciting to see a prediction come true. . .especially when its one of your own. For a number of years now it has been obvious that the simple synchronous web services taught in books and college courses were never going to be sufficient enough to support the highly complex transactions required in the real world. In particular, transactions which require two-phase validation, such as those that leverage Schematron in addition to simple XML Schema validation.

It appears that others are leaning towards agreeing with this same premise. Microsoft is now publishing its Visual Studio Async in order to better handle these sorts of highly complex transactions which require higher network latency than your traditional synchronous request/response web service. Infoworld has a good article here that describes Microsoft's adoption of this new trend.

While not endorsing any specific platform, Microsoft's development tools have historically been a great indicator of market trends. This latest addition proves to be a good sign that the industry is headed towards better support of this highly complex transaction processing world in which we live.

XSLT: Transform XML into nc:ContactInformation Structure

2010-10-20T10:33:00.000-07:00

This is a short post to show how to leverage XSLT to convert a simple and generic XML file into NIEM-conformant XML as it pertains to the nc:ContactInformation block.

This is a very common situation where a "non-NIEM" data stream is received and needs to be converted to a conformant structure. Take the following sample non-NIEM XML instance:

<?xml version="1.0" encoding="UTF-8" ?>     
<SomeBatchOfStuff>
   <Person>
       <Name>John Doe</Name>
       <PhoneNumber>212-111-2222</PhoneNumber>
   </Person>
   <Person>
       <Name>Sally Smith</Name>
       <PhoneNumber>212-333-4444</PhoneNumber>
   </Person>
</SomeBatchOfStuff>

If this very logical structure needed to be converted into nc:Person and nc:ContactInformatoin elements (with an nc:PersonContactInformationAssociation object to link the two together), the following XSLT could be used:

<?xml version="1.0" encoding="UTF-8" ?>     
<xsl:stylesheet  xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ns0="SomeNonConformantDocumentNamespace" version="1.0" exclude-result-prefixes="xs">
    <xsl:output method="xml" encoding="UTF-8" indent="yes"/>
    <xsl:template match="/">
        <xsl:variable name="var1_instance_InputSchema" select="."/>
        <MyNIEMConformantDocument xmlns="MyNIEMDocumentNamespace" xmlns:i="http://niem.gov/niem/appinfo/2.0" xmlns:nc="http://niem.gov/niem/niem-core/2.0" xmlns:niem-xsd="http://niem.gov/niem/proxy/xsd/2.0" xmlns:s="http://niem.gov/niem/structures/2.0">
            <!-- Loop through the Persons and create an NIEM Conformant Person -->
            <xsl:for-each select="$var1_instance_InputSchema/SomeBatchOfStuff/Person">
                <xsl:variable name="NonConformantPerson" select="."/>
                <nc:Person>
                    <xsl:attribute name="s:id">
                        <xsl:value-of select="generate-id(.)"/>
                    </xsl:attribute>
                    <nc:PersonName>
                        <xsl:for-each select="$NonConformantPerson/Name">
                            <nc:PersonFullName>
                                <xsl:value-of select="string(.)"/>
                            </nc:PersonFullName>
                        </xsl:for-each>
                    </nc:PersonName>
                </nc:Person>
            </xsl:for-each>
            
            <!-- Loop through the phone numbers and create a NIEM Conformant Contact Information -->
            <xsl:for-each select="$var1_instance_InputSchema/SomeBatchOfStuff/Person/PhoneNumber">
                <nc:ContactInformation>
                    <xsl:attribute name="s:id">
                        <xsl:value-of select="generate-id(.)"/>
                    </xsl:attribute>
                    <nc:ContactTelephoneNumber>
                        <nc:FullTelephoneNumber>
                            <nc:TelephoneNumberFullID>
                                <xsl:value-of select="string(.)"/>
                            </nc:TelephoneNumberFullID>
                        </nc:FullTelephoneNumber>
                    </nc:ContactTelephoneNumber>
                </nc:ContactInformation>
            </xsl:for-each>
            
            <!-- Loop through the phone numbers and create a NIEM Conformant Contact Information Association -->
            <xsl:for-each select="$var1_instance_InputSchema/SomeBatchOfStuff/Person">
                <nc:PersonContactInformationAssociation>
                    <nc:PersonReference>
                    <xsl:attribute name="s:ref">
                        <xsl:value-of select="generate-id(.)"/>
                    </xsl:attribute>
                    </nc:PersonReference>
                    <nc:ContactInformationReference>
                        <xsl:attribute name="s:ref">
                            <xsl:value-of select="generate-id(./PhoneNumber)"/>
                        </xsl:attribute>
                    </nc:ContactInformationReference>
                </nc:PersonContactInformationAssociation>
            </xsl:for-each>
        </MyNIEMConformantDocument>
    </xsl:template>                    
</xsl:stylesheet>

The XSLT heavily leverages the XSLT generate-id() function in order to work its magic and result in the following NIEM-conformant XML file:

<?xml version="1.0" encoding="UTF-8" ?>   
<MyNIEMConformantDocument xmlns="MyNIEMDocumentNamespace" xmlns:i="http://niem.gov/niem/appinfo/2.0" xmlns:nc="http://niem.gov/niem/niem-core/2.0" xmlns:niem-xsd="http://niem.gov/niem/proxy/xsd/2.0" xmlns:s="http://niem.gov/niem/structures/2.0" xmlns:ns0="SomeNonConformantDocumentNamespace">
   <nc:Person s:id="d0e3">
      <nc:PersonName>
         <nc:PersonFullName>John Doe</nc:PersonFullName>
      </nc:PersonName>
   </nc:Person>
   <nc:Person s:id="d0e12">
      <nc:PersonName>
         <nc:PersonFullName>Sally Smith</nc:PersonFullName>
      </nc:PersonName>
   </nc:Person>
   <nc:ContactInformation s:id="d0e8">
      <nc:ContactTelephoneNumber>
         <nc:FullTelephoneNumber>
            <nc:TelephoneNumberFullID>212-111-2222</nc:TelephoneNumberFullID>
         </nc:FullTelephoneNumber>
      </nc:ContactTelephoneNumber>
   </nc:ContactInformation>
   <nc:ContactInformation s:id="d0e17">
      <nc:ContactTelephoneNumber>
         <nc:FullTelephoneNumber>
            <nc:TelephoneNumberFullID>212-333-4444</nc:TelephoneNumberFullID>
         </nc:FullTelephoneNumber>
      </nc:ContactTelephoneNumber>
   </nc:ContactInformation>
   <nc:PersonContactInformationAssociation>
      <nc:PersonReference s:ref="d0e3"/>
      <nc:ContactInformationReference s:ref="d0e8"/>
   </nc:PersonContactInformationAssociation>
   <nc:PersonContactInformationAssociation>
      <nc:PersonReference s:ref="d0e12"/>
      <nc:ContactInformationReference s:ref="d0e17"/>
   </nc:PersonContactInformationAssociation>
</MyNIEMConformantDocument>

Schematron: Supporting Form Validation in a Data Validation World

2010-03-31T11:39:00.001-07:00

NIEM is for all intents and purposes a highly object-oriented data model which may, or may not be used by form entry tools at the time of data capture. While this has enormous benefits, it can be detrimental if one wants to use Schematron for BOTH form and data validation in conjunction with NIEM.

In order to properly support form validation, source systems will often want to know exactly which field caused an error during the processing of business rules surrounding a form. While in many systems, the map between the source fields and the NIEM may be readily available, in cases where it is not or processing speed is critical, the data validation engine should be capable of furnishing this information back to the calling system.

Lets take the example of a citation data capture tool with the following example data entry UI:

NIEM supports the passing of a “footnote” on every element called the nc:Metadata element. nc:Metadata is a complex data type that in turn includes an element to store the source-system’s field name called nc:SourceIDText. The NIEM conformant XML instance to pass this would look something like the following:

<ns:CitationBatchDocument> 
  ...
  ...
  ...
  <ns:Citation> 
    ...
    ...
    <!-- Citation Number --> 
    <nc:ActivityIdentification> 
      <nc:IdentificationID s:metadata=”M1”>ABC123</nc:IdentificationID> 
    </nc:ActivityIdentification> 
    <!-- Citation Date --> 
    <nc:ActivityDate> 
      <nc:Date s:metadata=”M2”>2002-05-30</nc:Date> 
    </nc:ActivityDate> 
    ...
    ...
  </ns:Citation> 
  ...
  ...
  ...
  <nc:Metadata s:id=”M1”> 
    <nc:SourceIDText>CITE_NUM</nc:SourceIDText> 
  </nc:Metadata> 
  <nc:Metadata s:id=”M2”> 
    <nc:SourceIDText>CITE_DATE</nc:SourceIDText> 
  </nc:Metadata> 
  ...
  ...
</ <ns:CitationBatchDocument>

While passing the field name to the business rules engine is 1/2 the battle, one must also return the field name with any errors the data validation engine runs into. An example Schematron code snippet to support returning the field name to the source system in the diagnostics would appear something like the following:

...
...
...
<pattern id="eBasicCiteRules">
  <title>Check the minimum basic citation rules.</title>
  <rule context="cite:CitationBatchDocument/cite:Citation">
    <let name="CiteNumSource" value="/cite:CitationBatchDocument
                                     /nc:Metadata [@s:id = current()
                                     /nc:ActivityIdentification
                                     /nc:IdentificationID/@s:metadata]
                                     /nc:SourceIDText"/>
    <assert test="nc:ActivityIdentification/nc:IdentificationID and
        string-length(normalize-space (nc:ActivityIdentification/nc:IdentificationID))
        &gt; 0" diagnostics="eCiteIdDiag">
            Citations must have a Citation Number.
    </assert>
  </rule>
</pattern>
...
...
...
<diagnostics>
  <diagnostic id="eCiteIdDiag">
    |<value-of select=”@CiteNumSource”/>|
    Some technical error description goes here (e.g. XPath to error).
  </diagnostic>
</diagnostics>

What this will yield to the end user is the following error message:

Citations must have a Citation Number.

What the source system will also receive in the case of any errors would look like the following:

|CITE_NUM| Some technical error description goes here (e.g. XPath to error).

The field name passed back could then be used by the source system in helping guide end users to complete the form correctly (e.g. jump to the first field with an error). In the above example, simple “bar” delimiters are being used (|) but this could of course be changed to proper XML elements through the use of > and $lt; instead.

EDITED 2010-04-01: Adding current() to XPath in Schematron code snippet.

XSLT: Using the generate-id() Function

2010-02-11T10:35:00.001-08:00

NIEM utilizes ID and IDREF elements heavily throughout the data standard. While this is native to the W3C specification for XML Schema files (.XSD) and in no way “unique” to NIEM, it is used much more heavily in NIEM than in many other national and international standards.

When converting or transforming to NIEM from another data standard, it quickly becomes necessary to generate unique identifiers in a common and consistent manner for key “noun” elements such as Persons, Places, Vehicles, and the like. A number of home-grown functions are scattered around the Internet to do this, however a native XSLT function already exists to perform this task called generate-id().

Say the following non-NIEM-conformant XML payload is provided to a system processing citation data:

<CitationBatch>
  <Citation>
    <CitationNumber>123456</CitationNumber>
    <CitationDefendant>
      <FirstName>John</FirstName>
      <LastName>Doe</LastName>
      <PhoneNumber>123-456-7890</PhoneNumber>
    </CitationDefendant>
    <!-- Remainder Omitted -->
  </Citation>
<CitationBatch>

Within NIEM the <CitationDefendant> element above is termed the <j:CitationSubject> and includes a <nc:RoleOfPersonReference> rather than embedding all person information as child elements within the citation. Additionally, the phone number for any given person is contained within a <nc:ContactInformation> element.

The XSLT generate-id() function accepts a specific xml node as its input parameter and will consistently provide a unique ID for that node no matter where or how many times it is called from within the XSLT. For example, take the following XSLT snippets:

<xsl:for-each select="$xmlInputFile/CitationBatch/Citation">
        <xsl:variable name="xmlCiteNode" select="."/>
        <j:CitationSubject>
            <nc:RoleOfPersonReference>
                <xsl:attribute name="s:ref">
                    <xsl:value-of select="generate-id($xmlCiteNode/CitationDefendant)"/>
                </xsl:attribute>
            </nc:RoleOfPersonReference>
        </j:CitationSubject>
    </xsl:for-each>
    ....
    ....
    ....
    <xsl:for-each select="$xmlInputFile/CitationBatch/Citation/CitationDefendant">
        <xsl:variable name="xmlCiteSubjectNode" select="."/>
        <nc:Person>
            <xsl:attribute name="s:id">
                <xsl:value-of select="generate-id($xmlCiteSubjectNode)"/>
            </xsl:attribute>
        </nc:Person>
    </xsl:for-each>
    ....
    ....

Even though the generate-id() function is called in two places within the transform, using two different variable names, the function will return the same exact yet unique ID as the XPath for both variables resolve to the same element in the input schema. The output of the above would appear as the following:

....
....
<j:CitationSubject>
    <nc:RoleOfPersonReference s:ref="d0e8"/>
</j:CitationSubject>
....
....
<nc:Person s:id="d0e8"/>

This powerful function within XSLT dramatically ease ID and IDREF usage within XML and makes implementation of transforms to NIEM relatively trivial.

XSLT: Transform Date and Time Elements into nc:DateTime

2010-01-11T15:46:00.001-08:00

While NIEM practitioners tend to merge Date and Time elements together into single nc:DateTime elements, we often find that the outside world separates these into two fields in their XML data packages. For example, if someone were to use Java XForms or Microsoft InfoPath to capture data in an electronic form, it is common to separate these out into their component parts.

For example, assume a NIBRS report form exists and has discrete date and time values. Using XSLT to merge these is quite simple and can be done using the concat() function as show here:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet 
    version="1.0" 
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:ns="SomeNibrsOffenseReportNamespace"
    exclude-result-prefixes="ns">
    
    <xsl:output method="xml" encoding="UTF-8" indent="yes"/>
    
    <xsl:template match="/">
        <xsl:variable name="sInputSchema" select="."/>
        
        <OffenseReportDocument xmlns="SomeNiemOffenseReportNamespace" xmlns:j="http://niem.gov/niem/domains/jxdm/4.0" xmlns:nc="http://niem.gov/niem/niem-core/2.0">
          <j:Offense>
            <nc:ActivityDate>
              <nc:DateTime>
                <xsl:value-of select="concat(string($sInputSchema/ns:NibrsForm/ns:OffenseDate), 'T', string(sInputSchema/ns:NibrsForm/ns:OffenseTime))"/>
              </nc:DateTime>
            </nc:ActivityDate>
          </j:Offense>
        </OffenseReportDocument>
    </xsl:template>
</xsl:stylesheet>

In the above example, the concat() function allows us to merge the date (e.g. ‘2010-01-01’), the letter ‘T’, and the time (e.g. ‘12:00:00’) into a single string which in turn can be evaluated as a nc:dateTime element.

1-13-10 – Edit for typo

Schematron: Using the Number() Function Versus Casting

2010-01-07T15:52:00.001-08:00

There are situations where it becomes necessary to test the value of a numeric element to ensure it meets some minimum or maximum value. As Schematron is capable of treating any element as a string, it is generally a best practice to cast the value to a numeric data type first.

For example, on a citation or a complaint document it may be necessary to check the fine or bail amount to ensure it is greater than zero. This could be done with the following Schematron assert statement:

<assert test="xsd:double(nc:ObligationDueAmount) &gt; 0">
  Bail amount may not be less than zero.
</assert>

While the above would work when a value is provided in the nc:ObligationDueAmount element, an XSLT error would be raised in the following circumstances:

Value is blank or null
Value is omitted
Value is a string value

For this reason, it is often preferable to use the native XPath function number(). As described by Ms. Priscilla Walmsley in her O’Reilly book XQuery, this function will prevent the XSLT parser from throwing an error and instead return the value ‘NaN’ (Not a Number). The following would be the same way the Schematron test could be written using the number() function instead:

<assert test="number(nc:ObligationDueAmount) &gt; 0">
  Bail amount may not be less than zero.
</assert>
<assert test="nc:ObligationDueAmount and string-length(nc:ObligationDueAmount) &gt; 0">
  Bail amount may not be left blank or otherwise omitted.
</assert>

While a few more lines are required, this prevents a runtime parser error from being raised and causing havoc with the validation engine.

Article Title and Tag Updates

2009-12-26T10:19:00.001-08:00

As Schematron is so closely related to XSLT and other related languages such as XQuery, in the new year, we have decided to begin providing XSLT tips in addition to the Schematron tips. We will be updating the prior posts tags and titles to help differentiate the Schematron from the XSLT articles. We apologize for any inconvenience or confusion these updates may cause.

Schematron: Trim Whitespace When Testing String Length

2009-12-08T16:34:00.001-08:00

As discussed in previous posts, the string-length() > 0 test is useful in checking to be sure null values are not passed; a validation step that raw XSD does not natively perform. This ensures the following is NOT allowed:

<nc:PersonGivenName></nc:PersonGivenName>

It also prevents the following:

<nc:PersonGivenName/>

However, if white space is not trimmed, the following WOULD be allowed:

<nc:PersonGivenName>  </nc:PersonGivenName>

In order to trim leading and trailing white space, the built-in XSLT function normalize-space() can be used. This in effect eliminates the above scenario where spaces have been inserted into the string. This can be seen in the following example:

<assert test="string-length(normalize-space(nc:PersonGivenName)) &gt; 0"/>
  Person's first name may not be left blank.
</assert>

Be aware, that this function also eliminates redundant spaces between characters (including duplicate carriage returns and line-feeds) so a custom replace() function may be required if you wish to preserve those extra characters in your string length checks.

Schematron: Nesting XPath Values Within an XPath Predicate

2009-11-23T08:45:00.001-08:00

In previous examples, we have seen the usage of a temporary variable or <let> tag to store a value which is later used in an XPath predicate (the square brackets surrounding the index of an element array). It is important to note that this is not required. A simple XPath statement can be used in the predicate for any other XPath statement. For example see the following:

<pattern id="wEmptyMetadataComment">
  <title>Ensure person metadata comment is not blank.</title>
  <rule context="/ns:SomeDocument/nc:Person">
    <assert test="string-length(/ns:SomeDocument/nc:Metadata[@s:id=current()/@s:metadata]/nc:CommentText) &gt; 0">
      Comments regarding a person should not be blank.
    </assert>
  </rule>
</pattern>

In the above example, simply the attribute @s:id=/ns:SomeDocument/nc:Person/@s:metadata is used to identify which specific Metadata element should be examined. With the context defined as /ns:SomeDocument/nc:Person, the rule will loop through each nc:Person element and use the appropriate @s:metadata value in each subsequent pass.

[Updated: Corrected Syntax on 04-01-2010]

Schematron: Validating NIEM Documents Against Non-Conformant Code Lists

2009-11-13T14:59:00.001-08:00

Schematron rules and assertions are based upon XPath statements, which allow for a number of powerful XML querying capabilities. Two XPath capabilities leveraged and outlined in this section are doc() and XPath predicates which allow us to validate data captured in an NIEM XML instance against external code list of any kind.

Lets assume a scenario where we would like to validate an exchange document’s category against a predefined list of enumerated values. This list is maintained by an outside party in a format other than NIEM and changes on a fairly regular basis.

Traditionally, a NIEM practitioner would take this list and define an enumeration within an extension schema to enforce this code list. Each time the third party makes a change to that code list, an updated NIEM extension schema would be created and redistributed. This maintenance-intensive process could become overwhelming therefore the team chose instead to simply adopt the third-party list and keep it in the following non-conformant format relying instead on Schematron to perform the validation:

<?xml version="1.0" encoding="UTF-8"?>
<!-- List of Valid code Values -->
<CategoryList>
  <Category>a</Category>
  <Category>b</Category>
</CategoryList>

As shown in the above, the valid categories include the values “a” and “b”. An example of a NIEM-conformant XML payload would look something like the following:

<ns:SomeDocument 
    xmlns:nc="http://niem.gov/niem/niem-core/2.0"    
    xmlns:ns="http://www.niematron.org/SchematronTestbed"
    schemaLocation"http://www.niematron.org/SchematronTestbed  ./SomeDocument.xsd">
  <nc:DocumentCategoryText>A</nc:DocumentCategoryText>
  <!-- Remaining Document Elements Omitted -->
</ns:SomeDocument>

In this example, the developers would like to perform the validation ignoring case, therefore the Schematron rule to validate the nc:DocumentCategoryText against the third-party-provided list would look something like the following:

<pattern id="eDocumentCategory">
  <title>Verify the document category matches the external list of valid categories.</title>
  <rule context="/ns:SomeDocument">
    <let name="sText" value="lower-case(nc:DocumentCategoryText)"/>
    <assert test="count(doc('./CategoryList.xml')/CategoryList/Category[. = $sText]) &gt; 0">
      Invalid document category.
    </assert>
  </rule>
</pattern>

Lets look at some of the key statements in the above Schematron example breaking it into individual parts.

lower-case(nc:DocumentCategoryText) – This statement encapsulated in a <let> tag converts the text in the NIEM payload to lower case thereby ignoring deviations from the code list due to case. It is then stored in a temporary variable named $sText.
doc('.CategoryList.xml')/… – This effectively points the parser at the third-party provided file (in this example assumed to be in the same directory as the .sch file) so that elements from that file can be referenced using the XPath in addition to elements in the source payload document.
…/Category[. = $sText] – The usage of the square brackets ([ and ]) in an XPath statement is considered a predicate. Any number of predicate statements can be made to help filter values contained within an XPath, but in this case, the expression tells the parser to select all of the Category elements with the value contained in the variable $sText.
count(…) > 0 – The XQuery count function returns the number of elements contained in the XPath. If no match to the category existed, the count would return a value of zero, therefore we want to ensure the value is greater than zero meaning a match existed in the external code list.

Schematron: Enforce String Patterns in Schematron

2009-11-06T14:21:00.001-08:00

In the general area of XML schemas, XSD “patterns” are commonly used to enforce special string formatting constraints. This is a very powerful tool when a document recipient wishes to ensure that the sender provides string data in a consistent format. A common example is the usage of a string constraint is to validate the structure of a Social Security Number (SSN). This would be expressed in a typical schema in the following manner:

<xsd:simpleType name="SsnSimpleType">
    <xsd:restriction base="xsd:string">
        <xsd:pattern value="[0-9]{3}[\-][0-9]{2}[\-][0-9]{4}" />
    </xsd:restriction>
</xsd:simpleType>

As with most parts of NIEM, much of the model is based on inheritance which makes enforce of simple data types, such as that shown above, cumbersome and awkward. Semantically, the correct element for an SSN would be under:

nc:Person/nc:PersonSSNIdentification/ nc:IdentificationID

Since nc:PersonSSNIdentification is an nc:IdentificationType, if one were to enforce SSN formatting on nc:IdentificationID, any other part of the schema that is derived from nc:IdentificationType would also need to abide by the same rules (e.g. Driver License Number, State ID Number, Document Identification, etc.). In the past this situation led to one thing. . . extension.

With Schematron, extension for this purpose could be avoided. Rather than enforcing the string constraints in the XSD file, instead the IEPD publisher could enforce this constraint within the Schematron rules document instead. The following is an example of what code would be required in Schematron to accomplish this purpose:

<pattern id="ePersonSSN">
  <title>Verify person social security number is in the correct format.</title>
  <rule context="/ns:SomeDocument/nc:Person/nc:PersonSSNIdentification">
    <assert test=
      "count(tokenize(nc:IdentificationID,'[0-9]{3}-[0-9]{2}-[0-9]{4}')) 
      - 1 = 1">
       Social security number must be in the proper format (e.g. 11-222-3333).
    </assert>
  </rule>
</pattern>

By using the Schematron approach, the semantically equivalent element is preserved in the schema and only the appropriate identifier is subjected to the constraint.

This approach can be further extended to address any number of string constraints. Another example would be ensuring an identification number only contains digits and has a string length of 5 or more. This could be done by using the following XQuery count() query instead:

count(tokenize(nc:IdentificationID, '\d')) > 5

This very powerful approach to constraining strings is yet another reason to take a real good look at Schematron in conjunction with your NIEM IEPDs.

Schematron: Correct nc:DateRepresentation Usage

2009-11-04T12:17:00.001-08:00

The inherent flexibility of NIEM proves to be an incredibly beneficial when used correctly, however this benefit can also be one of its largest banes. Sometimes this flexibility can lead to confusion when implementers attempt to deploy a NIEM exchange which is “valid” according to the XSD, yet not what the recipient is expecting.

One such example is NIEM’s usage of substitution groups where a variety of data elements are legal according to the schema, but rarely are all of these legal options accounted for by the recipient’s adapter. Take NIEM’s DateType as an example. It employs the explicit substitution group (abstract data element) of nc:DateRepresentation which can be one of several different data types. This representation can be replaced with a date (2009-01-01), a date/time (2009-01-01T12:00:00), a month and a year (01-2009), etc.

Lets assume for a minute that a document has two different dates: a document filed date, and a person’s birth date. The publisher’s intention is that filed date be a “timestamp” which includes both a date and a time, while the birth date is simply a date including a month, day and year. A valid sample XML payload would look something like the following:

<?xml version="1.0" encoding="UTF-8"?>
<ns:SomeDocument>
  <nc:DocumentFiledDate>
    <nc:DateTime>2009-01-01T01:00:00</nc:DateTime>
  </nc:DocumentFiledDate>
  <nc:Person>
    <nc:PersonBirthDate>
      <nc:Date>1970-01-01</nc:Date>
    </nc:PersonBirthDate>
  </nc:Person>
</ns:SomeDocument>

The Schematron code to enforce the publisher’s intentions could appear as the following:

<pattern id="eDocumentDateTime">
  <title>Verify the document filed date includes a date/Time</title>
  <rule context="ns:SomeDocument/nc:DocumentFiledDate">
    <assert test="nc:DateTime">
      A date and a time must be provided as the document filed date.
    </assert>
  </rule>
</pattern>
<pattern id="ePersonBirthDate">
  <title>Ensure the person's birth date is an nc:Date.</title>
  <rule context="ns:SomeDocument/nc:Person/nc:PersonBirthDate">
    <assert test="nc:Date">
      A person's birth date must be a full date.
    </assert>
  </rule>
</pattern>

This is a great example of how Schematron can help clarify a publisher’s intent as NEIM-conformant services are developed and deployed.

Schematron: License Plate State is Required when a Number Exists

2009-10-21T13:26:00.001-07:00

A common practice in transportation and law enforcement is to document a vehicle’s license plate number. In many situations, this plate number must be accompanied by the state which issued the license plate.

In NIEM, a vehicle’s license plate is contained within the nc:ConveyanceRegistrationPlateIdentification element which is an nc:IdentificationType. Using schema cardinality, one could make a the state required by simply assigning a minOccurs=”1” to the nc:IdentificationJurisdiction element, however this can often cause more problems than it solves for two key reasons:

Making jurisdiction required through schema cardinality makes it required globally throughout the exchange even if it doesn’t apply in those scenarios as many other elements in a typical NIEM exchange are also nc:IdentificationType data types.
nc:IdentificationJurisdiction is an abstract data element that can be replaced with any number of elements, not all of which are enumerated state values. Some are country codes, some are province codes for other countries and others are simply free-text.

This presents another ideal use case for Schematron. The following example code segment ensures a NCIC plate issuing state is included any time a Plate Identification exists:

<pattern id="eVehiclePlateState">
  <title>Ensure a plate state is included with a plate number.</title>
  <rule context="ns:MyDocument/nc:Vehicle/nc:ConveyanceRegistrationPlateIdentification">
    <assert test="j:IdentificationJurisdictionNCICLISCode">
      A plate state must be included with vehicle license plate.
    </assert>
  </rule>
</pattern>

The same segment can be modified to enforce any of the available jurisdiction code lists. For example, an exchange in Canada may wish to check for the existence of j:IdentificationJurisdictionCanadianProvinceCode instead of j:IdentificationJurisdictionNCICLISCode.

Link: Schematron Tutorials

2009-10-14T16:15:00.001-07:00

There are a number of solid Schematron tutorials posted on the internet. Two of most comprehensive ones available are:

The Oasis-Hosted Coverpages.com Schematron Article.
Roger Costello’s Tutorial

We won’t plagiarize their incredible wealth of Schematron material here, so it is highly recommend developers jump to and use these online tutorials.

Schematron: Use Phase for Errors and Warnings

2009-10-12T15:57:00.001-07:00

Schematron allows for grouping of rules not only by XPath, but also through association using the “Phase” element. While this has long been recommended as an approach for improving validation performance and unit testing, Phase also serves as an excellent way to group together and differentiate between critical errors and simple warnings.

For example, an agency might choose to have some minimum data restrictions surrounding Officers and Agencies on a electronic citation that can not be overlooked, and at the same time have warnings surrounding statue codes that do not match the known state code values. In Schematron the following would the the code matching this scenario:

<!-- Rules resulting in just warnings (should not prevent submission) -->
<phase id="Warnings">
  <active pattern="validStatute"/>
</phase>
<!-- Rules resulting in errors (must prevent submission) -->
<phase id="Errors">
  <active pattern="minOfficerData"/>
  <active pattern="minAgencyData"/>
</phase>

The pattern attribute is an IDREF to a related pattern ID somewhere in the document. A developer can then create code to prevent the submission of any validation errors resulting from only one of the above phases. Schematron validation engines typically have command line switches or parameters to specify which phase should be run. For example, in Xerces’ implementation of Saxon, the parameter “phase=x” is used where x is one of the phase id’s listed above or “#ALL” if all phases should be processed.

Schematron: Recommended Reading

2009-10-09T14:43:00.000-07:00

What books exists out there that detail Schematron and how to use it?

Best one I've seen to date is "Schematron" from O'Reilly Media available in electronic (Adobe PDF) format for $9.99 USD. Provides a great overview and set of examples using Schematron, albeit exclusively in hierarchically linked XML documents. It does assume the reader already has a strong understanding of XML and XSLT. If you are a bit rusty on XSLT or XQuery, you'll want to pick up a book on that as well.

Schematron: Officer Has a Last Name

2009-10-07T08:31:00.000-07:00

This is the first in a series of code example articles that will be posted to give NIEM developers a head start in using Schematron. This example will show how to perform a test across multiple branches or nodes of a typical NIEM schema as law enforcement officer is a role played by a person in NIEM schemas. Take the following example XML code:

<ns:SomeDocument>
<j:Citation>  
   <j:CitationIssuingOfficial>
     <nc:RoleOfPersonReference s:ref="P1"/>  
   </j:CitationIssuingOfficial>
</j:Citation>
<nc:Person s:id="P1">
   <nc:PersonName><nc:PersonSurName>Smith</nc:PersonSurName></nc:PersonName>
</nc:Person>
</ns:SomeDocument>

One way in which to test for the existence of a last name is to match the ID with the officer's REF and test to be sure the string length is greater than 1 as shown in the following example (using XSLT2 & ISO Schematron):

<pattern id="eOfficerData"> 
    <let name="sOfficerRef" value="ns:SomeDocument/j:Citation/ j:CitationIssuingOfficial/nc:RoleOfPersonReference/@s:ref"/> 
    <rule context="ns:SomeDocument/nc:Person"> 
        <report test="@s:id = $sOfficerRef and string-length(nc:PersonName/nc:PersonSurName) < 1"> 
            Officers last name must be provided. 
        </report> 
    </rule> 
</pattern>

In theory the same test can be done using the XQuery id() function however use of the id function is HIGHLY dependent on the parser's capabilities.

To ISO or Not to ISO

2009-10-05T19:31:00.000-07:00

Schematron comes in two common flavors, Schemtron v1.5 (older) and ISO Schematron (newer). While there are a number of differences that can be read about on the official Schematron Website, it can sometimes be confusing about which version is "best". The following table presents some of the differences between the two:

Feature	v1.5	ISO
Varriables	Not supported	`let` element available.
Query Language	XSLT 1.0/XPath 1.0	XSLT 1.0/XPath 1.0, XSLT 2.0/XPath 2.0, EXSLT, STX, XSLT 1.1, etc.
Abstract Patterns & Inheritance	Not Supported	Supported
`value-of` Element (helpful in debugging and error messaging)	Not Supported	Supported
`xsl:key` Element	Supported	Not Supported (Workaround Exists)
`flag` Attribute	Not Supported	Supported
SVRL	Not Supported	Supported
`include` element	Not Supported	Supported

My suggestion would be to go with ISO Schematron, for the following reasons:

Support for variables is important when working with ID and IDREF (more about this in a later blog).
XQuery 2.0 functions provide a number of goodies that would be hard to pass up.
ISO Schematron is a recognized ISO standard. . . which should count for something in a standards-based community.

Feel free to comment if you know any other key differences or disagree with my suggestion. Update: 2009-10-05 - Correct typo.

XML Editor Support for Schematron

2009-10-05T06:11:00.000-07:00

XML editors are one of the primary tools in a practitioner's toolbox and as such, are the first place many of us look for assistance using a complex validation engine such as Schematron. Currently today editors on the market have different levels of support for Schematron; as it is at the core a multi-pass validation engine. The following table should provide you with links to where tutorials for some of the most common XML editors are located:

Editor	Native Schematron Validation?	Native Schematron Editing?	Tutorial
oXygen	Yes	Yes	http://www.oxygenxml.com/validation.html
XMLSpy	No	Yes	http://xml.sys-con.com/node/40656
Editix	Yes	Yes	None available

Update: 2009-10-05 - Adding Hyperlink to Editor websites.

Welcome!

2009-10-05T05:59:00.001-07:00

Welcome to the initial post of the NIEM-Schematron (aka NIEMatron) community. At the 2009 NIEM National Training Event (see #NIEMNTE on Twitter) it became apparent that the next big task for NIEM was ensuring that data was self-describing. As it currently stands today, the XML Schema Documents (XSD) used can only enforce and describe a small portion of the data in each Information Exchange Package Document (IEPD) developed. Expressing complex business rules has and continues to be a large struggle for many implementers of the NIEM standard. Most implementers have chosen to use validation logic built using whatever programming language fits their need (e.g. Java, VB.net, C++, C#, etc.) however once this is done, there is very little that one of their information exchange partners can use if the partner leverages a different technology or programming language. This problem is one of the key reasons the XML community developed the ISO standard Schematron. Schematron is a simple XML/XSLT-based script that allows implementers to describe highly complex business rules that would otherwise be impossible to describe using XSD alone. This website and blog will be updated on a regular basis to provide NIEM practitioners a place to learn about, ask questions, and discuss Schematron. We will attempt to provide as much information to NIEM developers as possible about tricks and traps we've encountered while using this powerful scripting language and will also be providing further executive-level explanations surrounding the technology and the value proposition it brings. Please be patient as like many of you, we travel quite often and may have difficulty responding immediately to questions you pose. Please follow us on Twitter (user: NIEMatron) if you would like to be appraised of new messages as they are posted. Thanks, and welcome to the future!