Library Journal "Digital Libraries" Columns 1997-2007, Roy Tennant
Please note: these columns are an archive of my Library Journal column from 1997-2007. They have not been altered in content, so please keep in mind some of this content will be out of date.
The Engine of Interoperability
12/15/2003
We don't think about standards when we replace a light bulb or use an electrical outlet. But we are surrounded by standards, and their invisibility is a testimony to their effectiveness. Without them we would be faced with an array of tools that couldn't work together in expected ways. On the other hand, where development is in ferment, creating standards too soon can have a chilling effect. It's a fine line between interoperability and innovation. Today is a fertile time for standards development in digital libraries. Many of these standards are still in their infancy and some are only at the idea stage. As an example, at the end of October the National Information Standards Organization (NISO) hosted a workshop in Washington, DC, on OpenURL and Metasearching (also known as federated searching). Although "metasearch" products have not been on the market for long, increased use of these tools--and their impact on database providers--make metasearching a prime area for standards development. At this meeting there was a call for an "0.1" standard for sending a search to a database and receiving back a structured result (as opposed to an HTML page of search results). This would help database providers know when they were being metasearched and format a response that would take fewer resources. From the metasearch side, a structured result set (instead of "screen-scraping" an HTML page) would be a huge advantage. Emerging metadata standards Dublin Core is an emerging metadata standard in development since 1995. The simple set of Dublin Core elements serves as either a basic element set for simple metadata purposes (e.g., adding metadata to web pages) or as a common meeting ground for more complex metadata schemes (e.g., MARC). Earlier this year the standard was established as an International Standards Organization standard (DIS 15836). Dublin Core serves an important role for relatively simple uses, but we need more complex and powerful metadata standards as well. Others are gaining ground. The Metadata Object Description Schema (MODS) is being developed to carry selected data from existing MARC 21 records and to enable the creation of original resource description records. The Metadata Encoding and Transmission Standard (METS) is an XML schema for describing digital objects or collections of digital objects with one or more metadata packages, including virtually any descriptive metadata schema (e.g., MODS, Dublin Core, etc.), administrative metadata, and structural metadata. It's considered the most useful way to encapsulate digital objects with the related metadata for those objects. Protocols are formal definitions of the rules that software must follow to exchange information effectively. These definitions often include how messages must be packaged and sent, appropriate responses to all possible situations, and even specific schemas to transmit metadata. One of the best known is the HyperText Transfer Protocol (HTTP) upon which the web is based. Emerging protocols & players A new protocol that is already providing useful services is the Open Archives Initiative (OAI) Protocol for Metadata Harvesting. It defines a method by which digital repositories can "expose" their metadata for harvesting and indexing in centralized discovery services such as OAIster at the University of Michigan. As was pointed out at the NISO workshop, we may also need a protocol for metasearch software to query dynamically databases and repositories that cannot be harvested. Not surprisingly, major professional institutions lead most standards development efforts. The Library of Congress is managing the development of MODS and comanaging the development of METS with the Digital Library Federation; OCLC leads the Dublin Core effort; and NISO is at work on standards in areas like networked reference services. The World Wide Web Consortium (W3C) develops Internet-related protocols and standards, such as XML. In a departure from the above, the OAI--which was initiated to improve access to scholarly e-print archives--is formed from a group of interested individuals who came together to solve a particular problem. Standards and us Individuals representing particular constituencies develop standards and protocols, but we can all help in the process. Most developing standards have discussion lists where people can pose questions and make suggestions. While voting on a draft standard is typically restricted to members, nonmembers can participate in dialog leading to the vote. To get involved takes as little as understanding the problem and thinking about how a standard can solve that problem (for more information, see the online version of this column at libraryjournal.com). To comprehend the power of standards in a library context, just think where we would be if MARC hadn't been created some 35 years ago.