Features - Garbage In, Garbage Out: Working Around Document Profiling to Improve Work Product Retrieval Efforts

Angie Turner, Director of Knowledge Management, Polsinelli Shalton Welte Suelthaus PC
Peter J. Ozolin, Vice President, Enterprise Solutions Practice Technologies, Inc.

As the Chief Knowledge & Technology Officer at Paul Hastings Janofsky & Walker (950 attorneys, 15 offices), Peter Ozolin recalls a partner telling him, “what you put into a system is what you get out of the system – that is, ‘garbage in, garbage out.’” Peter admits that his initial reaction was to write off the criticism; after all, the firm had invested so much effort in profiling data in the various information systems (CRM, DMS, Elite, etc.) However, as more thought was given to the matter, there was a realization that this partner was right on point. Was it possible that too much emphasis was being placed on what they hoped the attorneys would do, rather than what they would actually do? The success of several important information management objectives depended heavily on the efforts of the attorneys.

Take Client Relationship Systems (CRMs) as an example. CRMs are well designed to handle thousands of contacts, mailing lists and profiled information about a prospective or existing client. If deployed with a thoughtful strategy, the use of such a system can yield great dividends. This said, over the years I have heard several of my colleagues discuss the challenges associated with keeping data current and/or getting attorneys to contribute and provide data about their contacts. Distilling the problem to the basics, it’s simple – the system is only as good as the information that goes in - therein lies the problem. Many attorneys are unwilling or unable to take the time to provide coveted client information on an ongoing basis. Why should they? It is difficult to ascertain the value of such an effort for the individual attorney and it takes valuable time that could be spent on billable client matters.

A similar issue extends to sharing attorney work product, as exemplified by the experiences of the KM efforts at Polsinelli Shalton Welte Suelthaus PC.

Assembling a Pragmatic Work Product Retrieval Strategy

In the1990’s Polsinelli Shalton Welte Suelthaus (PSWS) (210 attorneys, 6 offices), like many other firms of similar size, invested in a document management system in an effort to aggregate the firm’s work product. An important offshoot of this effort has been an attempt to collect work product for specific practice areas. If done properly a search of the DMS should contain a more finite set of work product that has been augmented or “contextualized” so as to provide the practitioner with the Firm’s most exemplary form and/or experience with a particular issue.

Regrettably, both initiatives come up short in terms of meeting the attorney’s most important information retrieval objective—getting access to the right document(s) in an expedient manner. Why have these initiatives fallen short? You guessed it-- “Garbage in, Garbage Out.”

When you consider the make-up of document management systems, this is not surprising. DMS systems suffer from the same challenge as CRM systems, they rely on the quality and accuracy of the person completing the profile to operate in a fully realized manner. For years, those charged with implementing DMS systems have spent time tinkering with the document profile template in hopes of creating that perfect form for capturing data about a lawyer’s document. Inevitably, results have been the same – the majority of work product is classified under document types such as “memo,” “form,” “agreement,” “pleading” or “letter.” The problem is only compounded as the DMS accumulates multiple versions of documents, administrative documents and several years’ worth of ancillary materials.

Attempts to codify exemplary work product (otherwise referred to as “best practice collections”) have met with similarly limited success. If managed with the appropriate care and attention (translation – professional support lawyers and staff), these projects have a definitive chance at adding value. The challenge of course is getting attorneys to consistently contribute to a system as part of their daily routines.

Taking the “Profile” Out of the Process

After several years of analyzing various offerings in the market (WestKM and most recently Lexis Total Search) with limited enthusiasm from the attorneys, the managing partner at PSWS came across a company called Practice Technologies. Founded by a group of attorneys, PTI and their enterprise product—RealPractice--offered a fresh new perspective on information retrieval. This perspective begins with the premise that attorneys will do minimal document profiling in the DMS. RealPractice captures the DMS profile but will build its own “profile” based on the content of the document without any further coding, flagging or interaction by the attorney. The attorneys identified with the solution right away for three reasons:

  • this approach asked them to do nothing different than what they currently do –save documents into the DMS;
  • information searches are quite easy to accomplish using a browser-based interface which offers more precise work product searching since the documents have been automatically profiled; and
  • little or no training is required to learn the system.

RealPractice In Action

As alluded to above, the foundation of the RealPractice approach to work product retrieval rests on the system’s ability to programmatically analyze each document in the firm’s information base (DMS and beyond). “Knowing” what is contained in each document makes searches infinitely more on-point. This profiling or abstracting occurs during initial system implementation, and then transpires automatically on an ongoing basis as new documents are added to the DMS. The abstracts include substantive information such as legal topics and governing law. Documents are further organized by the type of document, practice area, and substantive area of law it represents. Where other solutions required users to “back in” to document retrieval through case links (which were often absent in many documents or did not relate to the point of research at all) or to utilize full-text searching with DMS profiles, RealPractice lets users search the system according to their specific needs and situations. The system in turn combines categorical searching – by substantive area, document type, practice area or transaction type – with full text searching.

Peter has illustrated the process through the following example: let’s say an attorney was interested in locating a real estate agreement that considered construction loans in Orange County, California. The system would make a distinction between a general real estate lease agreement and one having to do with construction loans, without the attorney having to type in an elaborate search query. Likewise, the system would isolate documents pertaining to Orange County (if nothing turns up, the user can broaden the search to adjoining counties or go statewide). The results are accompanied by meaningful abstracts, such as governing law, that enable users to assess what they’ve found without opening and reading the documents.


PSWS selected RealPractice because its approach is consistent with how attorneys look for information (by topic or issue.) Building a search in RealPractice that is as easy as “googling” makes it seem plausible that attorneys will actually use the system to find on point work-product and expertise across the firm’s knowledge-base! While it’s a bit early to say if a system like RealPractice will prove to be a panacea to law firm work product retrieval efforts, it seems to be conceptually on the right track. With a September roll-out date, the interest in the program is growing throughout the firm.