Thanks to Twitter, I found (or refound - this paper sounds very familiar) a paper from 1988 by Jonathan Grudin on Why CSCW Applications Fail: Problems in the Design and Evaluation of Organizational Interfaces (pdf). I am particularly interested in the realization that multiple points-of-view are a critical element of good design for collaboration tools. This will definitely enhance the way I think about these kinds of efforts. From the abstract:
Many systems, applications, and features that support cooperative work share two characteristics: A significant investment has been made in their development, and their successes have consistently fallen far short of expecta- tions. Examination of several application areas reveals a common dynamic: 1) A factor contributing to the application’s failure is the disparity between those who will benefit from an application and those who must do additional work to support it. 2) A factor contributing to the decision-making failure that leads to ill-fated development efforts is the unique lack of management intuition for CSCW applications. 3) A factor contributing to the failure to learn from experience is the extreme difficulty of evaluating these applications.
The lessons described in this paper, and recounted above, are very familiar. Grudin describes a number of facets as to how these failure dynamics arise. And the paper goes through a number of examples to describe how and where the dynamics arise. Under failure mode 1, there is the simple matter of asking people to do something new without there being a clear benefit to them. Another has to do with training on the new tools (or intuitive design), so people become comfortable quickly. Grudin makes a point that new "systems" are often accompanied by intensive training and even organizational changes, whereas new "tools" or "applications" will not get such intensive focus. While Grudin doesn't mention this specifically, there is also the issue of asking people to do new things without removing old tasks from their plate - creating task overload.
The discussion of point 2 is narrower, but makes the important point that for collaboration systems, the intuitions of just one or two people aren't sufficient for evaluating and designing the system. Intuitions about how work is happening are not sufficient to understand how a system will work. There are simply too many types of work that will be affected for people to fully understand the impact without some more rigorous approaches. Managers, for example, overvalue benefits that they will see in the system while undervaluing additional workload for others. Conversely, people undervalue such systems if they are the ones who will be asked to do more work, even if that the benefit to the larger population is commensurately greater.
And the final point takes into account the idea that collaborative tools and processes by their very nature require many people to participate. Evaluating them from only one or few points of view is almost sure to miss the big picture. Similarly, running small pilots will not usually capture the benefits anticipated to be created when a larger network of people are actively engaged in the new way of doing business.
The way I usually look at these questions is inspired by a set of "Rules for Technology" embedded in Eli Goldratt's Necessary But Not Sufficient (I've written about these a number of times):
What is the power of the technology?
What limitation is being overcome (limited capacity, for example)?
What old rules were followed because of the limitation (and need to be eliminated)?
What new rules need to be created?
Taking the thoughts in Grudin's paper into account, these questions have to be considered from many perspectives. What does management think are the limitations? Are those the same limitations seen by executives? How does staff see things? Depending on the type of change being implemented, these limitations are going to look different from the various dimensions of the business too: finance, production, research, sales, etc.
Great stuff.