Wednesday, October 21, 2009

Schematron: License Plate State is Required when a Number Exists

A common practice in transportation and law enforcement is to document a vehicle’s license plate number.  In many situations, this plate number must be accompanied by the state which issued the license plate. 

In NIEM, a vehicle’s license plate is contained within the nc:ConveyanceRegistrationPlateIdentification element which is an nc:IdentificationType.  Using schema cardinality, one could make a the state required by simply assigning a minOccurs=”1” to the nc:IdentificationJurisdiction element, however this can often cause more problems than it solves for two key reasons:

  1. Making jurisdiction required through schema cardinality makes it required globally throughout the exchange even if it doesn’t apply in those scenarios as many other elements in a typical NIEM exchange are also nc:IdentificationType data types.
  2. nc:IdentificationJurisdiction is an abstract data element that can be replaced with any number of elements, not all of which are enumerated state values.  Some are country codes, some are province codes for other countries and others are simply free-text. 

This presents another ideal use case for Schematron.  The following example code segment ensures a NCIC plate issuing state is included any time a Plate Identification exists:

<pattern id="eVehiclePlateState">
  <title>Ensure a plate state is included with a plate number.</title>
  <rule context="ns:MyDocument/nc:Vehicle/nc:ConveyanceRegistrationPlateIdentification">
    <assert test="j:IdentificationJurisdictionNCICLISCode">
      A plate state must be included with vehicle license plate.
    </assert>
  </rule>
</pattern>

The same segment can be modified to enforce any of the available jurisdiction code lists.  For example, an exchange in Canada may wish to check for the existence of j:IdentificationJurisdictionCanadianProvinceCode instead of j:IdentificationJurisdictionNCICLISCode

Wednesday, October 14, 2009

Link: Schematron Tutorials

There are a number of solid Schematron tutorials posted on the internet.  Two of most comprehensive ones available are:

We won’t plagiarize their incredible wealth of Schematron material here, so it is highly recommend developers jump to and use these online tutorials.

Monday, October 12, 2009

Schematron: Use Phase for Errors and Warnings

Schematron allows for grouping of rules not only by XPath, but also through association using the “Phase” element. While this has long been recommended as an approach for improving validation performance and unit testing, Phase also serves as an excellent way to group together and differentiate between critical errors and simple warnings.

For example, an agency might choose to have some minimum data restrictions surrounding Officers and Agencies on a electronic citation that can not be overlooked, and at the same time have warnings surrounding statue codes that do not match the known state code values. In Schematron the following would the the code matching this scenario:

<!-- Rules resulting in just warnings (should not prevent submission) -->
<phase id="Warnings">
  <active pattern="validStatute"/>
</phase>
<!-- Rules resulting in errors (must prevent submission) -->
<phase id="Errors">
  <active pattern="minOfficerData"/>
  <active pattern="minAgencyData"/>
</phase>

The pattern attribute is an IDREF to a related pattern ID somewhere in the document. A developer can then create code to prevent the submission of any validation errors resulting from only one of the above phases. Schematron validation engines typically have command line switches or parameters to specify which phase should be run. For example, in Xerces’ implementation of Saxon, the parameter “phase=x” is used where x is one of the phase id’s listed above or “#ALL” if all phases should be processed.

Friday, October 9, 2009

Schematron: Recommended Reading

What books exists out there that detail Schematron and how to use it?
Best one I've seen to date is "Schematron" from O'Reilly Media available in electronic (Adobe PDF) format for $9.99 USD. Provides a great overview and set of examples using Schematron, albeit exclusively in hierarchically linked XML documents. It does assume the reader already has a strong understanding of XML and XSLT. If you are a bit rusty on XSLT or XQuery, you'll want to pick up a book on that as well.

Wednesday, October 7, 2009

Schematron: Officer Has a Last Name

This is the first in a series of code example articles that will be posted to give NIEM developers a head start in using Schematron. This example will show how to perform a test across multiple branches or nodes of a typical NIEM schema as law enforcement officer is a role played by a person in NIEM schemas. Take the following example XML code:

<ns:SomeDocument>
<j:Citation>  
   <j:CitationIssuingOfficial>
     <nc:RoleOfPersonReference s:ref="P1"/>  
   </j:CitationIssuingOfficial>
</j:Citation>
<nc:Person s:id="P1">
   <nc:PersonName><nc:PersonSurName>Smith</nc:PersonSurName></nc:PersonName>
</nc:Person>
</ns:SomeDocument>

One way in which to test for the existence of a last name is to match the ID with the officer's REF and test to be sure the string length is greater than 1 as shown in the following example (using XSLT2 & ISO Schematron):

<pattern id="eOfficerData"> 
    <let name="sOfficerRef" value="ns:SomeDocument/j:Citation/ j:CitationIssuingOfficial/nc:RoleOfPersonReference/@s:ref"/> 
    <rule context="ns:SomeDocument/nc:Person"> 
        <report test="@s:id = $sOfficerRef and string-length(nc:PersonName/nc:PersonSurName) < 1"> 
            Officers last name must be provided. 
        </report> 
    </rule> 
</pattern>

In theory the same test can be done using the XQuery id() function however use of the id function is HIGHLY dependent on the parser's capabilities.

Monday, October 5, 2009

To ISO or Not to ISO

Schematron comes in two common flavors, Schemtron v1.5 (older) and ISO Schematron (newer). While there are a number of differences that can be read about on the official Schematron Website, it can sometimes be confusing about which version is "best". The following table presents some of the differences between the two:
Featurev1.5ISO
VarriablesNot supportedlet element available.
Query LanguageXSLT 1.0/XPath 1.0XSLT 1.0/XPath 1.0, XSLT 2.0/XPath 2.0, EXSLT, STX, XSLT 1.1, etc.
Abstract Patterns & InheritanceNot SupportedSupported
value-of Element
(helpful in debugging and error messaging)
Not SupportedSupported
xsl:key ElementSupportedNot Supported
(Workaround Exists)
flag AttributeNot SupportedSupported
SVRLNot SupportedSupported
include elementNot SupportedSupported
My suggestion would be to go with ISO Schematron, for the following reasons:
  • Support for variables is important when working with ID and IDREF (more about this in a later blog).
  • XQuery 2.0 functions provide a number of goodies that would be hard to pass up.
  • ISO Schematron is a recognized ISO standard. . . which should count for something in a standards-based community.
Feel free to comment if you know any other key differences or disagree with my suggestion. Update: 2009-10-05 - Correct typo.

XML Editor Support for Schematron

XML editors are one of the primary tools in a practitioner's toolbox and as such, are the first place many of us look for assistance using a complex validation engine such as Schematron. Currently today editors on the market have different levels of support for Schematron; as it is at the core a multi-pass validation engine. The following table should provide you with links to where tutorials for some of the most common XML editors are located:
EditorNative Schematron Validation?Native Schematron Editing?Tutorial
oXygenYesYeshttp://www.oxygenxml.com/validation.html
XMLSpyNoYeshttp://xml.sys-con.com/node/40656
EditixYesYesNone available
Update: 2009-10-05 - Adding Hyperlink to Editor websites.

Welcome!

Welcome to the initial post of the NIEM-Schematron (aka NIEMatron) community. At the 2009 NIEM National Training Event (see #NIEMNTE on Twitter) it became apparent that the next big task for NIEM was ensuring that data was self-describing. As it currently stands today, the XML Schema Documents (XSD) used can only enforce and describe a small portion of the data in each Information Exchange Package Document (IEPD) developed. Expressing complex business rules has and continues to be a large struggle for many implementers of the NIEM standard. Most implementers have chosen to use validation logic built using whatever programming language fits their need (e.g. Java, VB.net, C++, C#, etc.) however once this is done, there is very little that one of their information exchange partners can use if the partner leverages a different technology or programming language. This problem is one of the key reasons the XML community developed the ISO standard Schematron. Schematron is a simple XML/XSLT-based script that allows implementers to describe highly complex business rules that would otherwise be impossible to describe using XSD alone. This website and blog will be updated on a regular basis to provide NIEM practitioners a place to learn about, ask questions, and discuss Schematron. We will attempt to provide as much information to NIEM developers as possible about tricks and traps we've encountered while using this powerful scripting language and will also be providing further executive-level explanations surrounding the technology and the value proposition it brings. Please be patient as like many of you, we travel quite often and may have difficulty responding immediately to questions you pose. Please follow us on Twitter (user: NIEMatron) if you would like to be appraised of new messages as they are posted. Thanks, and welcome to the future!