.pdf available from ACM Digital Library - SIGIR99
Author's Abstract
Although there have been many prototypes of visualization in support of information retrieval, there has been little systematic evaluation that distinguishes the benefits of the visualization per se from that of various accompanying features. The current study focuses on such an evaluation of NIRVE, a tool that supports visualization of search results. Insofar as possible, functionally equivalent 3D, 2D, and text versions of NIRVE were implemented. Nine novices and six professional users completed a series of information seeking tasks on a set of retrieved documents. There were high interface costs for the 3D visualization, although these costs decreased substantially with experience. Performance was best when the tool's properties matched task demands; only under the right combination of task, user, and interface did 3D visualization result in performance comparable to functionally matched 2D and textual tools.
Additional Comments
This study is very similar to Swan and Allan study from SIGIR '98. The text interface was color-coded, with scrolling features. The 3D interface was sets of color-bar-chart cluster icons arranged on the surface of a sphere. The thickness of the box is proportional to the number of contained documents. Colored arcs connect clusters that differ by a single concept. An "opened" cluster displays an (illegible) 2d document rectangle projected outward from the cluster icon in the 3D space. The 2D interface is a flattened version of the 3D one, with a map displaying clusters connected by straight lines. A small bar under the cluster icon shows the number of documents. The three systems were user-tested against each other. It almost seems like they purposely sabotaged their novice users, giving them a slow machine with response time delays that added additional problems in 2D and 3D navigation.
The user tests seem valid, although the testers observed: "3D visualization cannot be adequately evaluated using only short-term studies of novice users." Why are novice users chosen to test expert systems? Interestingly, the color-coding throughout the three systems greatly improved user navigation. Problems with visibility/legibility were also reported for the 2D and 3D displays. While the designers try to explain the problem in terms of visual real-estate trade-offs, perhaps they should have realized that their system displays were illegible before they started user testing. Users of the 3D interface reportedly opened up the full text of documents just to be able to read a legible title and determine relevance. The authors recommend that more work be done on this topic, with " a more principled approach to design and evaluation" being needed.
NIRVE would be an interesting system to examine in more detail.
.pdf available from ACM Digital Library - SIGIR'98 or
http://hobart.cs.umass.edu/~allan/Papers/sigir98b.ps (recommended)
Author's Abstract
We built two Information Retrieval systems that were targeted for the TREC-6 "aspect oriented" retrieval track. The systems were built to test the usefulness of different visualizations in an interactive IR setting - in particular, an "aspect window" for the chosen task, and a 3-D visualization of document inter-relationships. We studied 24 users of the system in order to investigate: whether the systems were more effective than a control system, whether experienced users outperformed novices, whether spatial reasoning ability was a good predictor of effective use of 3-D, and whether the systems could be compared indirectly via a control system. Our results show substantial differences in user performance are related to spatial reasoning ability and to a lesser degree other traits. We also obtained markedly different results from the direct and indirect comparisons.
Additional Comments
The purpose of the Aspect window is to assist the user in categorizing the information as it is discovered and to keep an overview of the information discovered so far. "Aspect" oriented IR searching is concerned with finding a set of documents that contain all the relevant information about the topic, not all the relevant documents. The "aspect" clusters are also displayed in a 3-D display of clustered blocks - documents that are nearby share information content. The 3-D display provides the user with information about whether the document is worth investigating further. A result list of retrieved documents is also always available on the screen. The Aspect window must be used to mark relevant documents but the use of the 3-D window is optional. All three windows are tightly integrated (i.e. each reflects changes made in the other windows). Carefully constructed user tests were run to determine statistically significant comparisons between 2D and 3D visualizations.
The experiment's shocking conclusion: "We found no evidence of usefulness for the 3-D visualization. We found that the use of the visualization is better predicted by the users past experience with GUI's and mouse based interfaces than it is by spatial ability." This study uses a standard IR interface as a control, an "aspect" and a "3D" interface as experimental variables. The study seems well done, and we can learn much from their surprising results -- that experience is a better predictor than spatial ability. The Aspect 3D interface was not interesting looking, but it was integrated into an IR system in a helpful and supportive way, which other designers often overlook in favor of fancier graphics. This article is important because it compares novel and 3D systems and reports the results honestly -- 3D didn't improve searching for first time users of the system.
http://www-ihm.lri.fr/~mountaz/Publi/chi98.pdf
Author's Abstract
In this paper, we propose a methodology for evaluating spatial display techniques. Our approach is based on two different types of evaluation: analytical experiments and user studies. This approach is important because it uncovers aspects not detected when either technique is used alone. We apply this approach to the comparison of two different layout techniques.
Additional Comments
The authors acknowledge that many spatial displays are claimed to be useful representations of information, but that very little evaluation of that usefulness exists. Instead of attempting to measure "document relevancy" for an information retrieval simulation, the experiment simply used the numbers 1 through 50 to test the usefulness of a spatial display in locating the icons of a close proximity. The user study and the analytical results differed for varied displays, and rather than conclusive evidence about effective spatial displays, the need for further study was identified.
John Stasko, Richard Catrambone, Mark Guzdial, Kevin McDonald
Graphics, Visualization, and Usability Center,
Georgia Institute of Technology
GVU Technical Report Number: GIT-GVU-00-03
Author's Abstract
A variety of information visualization tools have been developed recently,
but relatively little effort has been made to evaluate the effectiveness and
utility of the tools. This article describes results from two empirical studies
of two visualization tools for depicting hierarchies, in particular, computer
file and directory structures. The two systems examined implement space-filling methodologies, one rectangular, the Treemap method, and one circular, the Sunburst method. Participants performed typical file/directory search and analysis tasks using the two tools. In general, performance trends favored the Sunburst tool with respect to correct task performance, particularly on initial use. Performance with Treemap tended to improve over time and use, suggesting a greater learning cost that was partially recouped over time. Each tool afforded somewhat different search strategies, which also appeared to influence performance. Finally, participants strongly preferred the Sunburst tool, citing better ability to convey structure and hierarchy.
Additional Comments
Although the study focuses on hierarchical structures, the strategies and procedures used to test the usefulness and usability of graphical information systems provide valuable insight for non-hierarchical systems as well. The procedures and results are well-documented and large color screen shots of the two systems illustrate the report. The authors stress the importance of a "more thorough and rigorous evaluation methodology" for visualizations, and conclude: "New visualization techniques, no matter how innovative, are not valuable unless they provide true utility and assist people with real tasks."
Chaomei Chen and Mary Czerwinski
Author's Abstract
In this paper, we describe an empirical study of individuals' spatial navigation strategies and a number of performance and preference measures with
regard to the design of a novel 3D visualisation. The underlying semantic space of the user interface consists of a collection of papers from the three
most recent ACM SIGCHI conference proceedings, visualised as a virtual reality network. This network was automatically constructed based on
semantic similarities derived from latent semantic analysis. We studied the search strategies and general preferences of eleven subjects who used
this system to find papers on various topics. The study has led to a number of interesting findings, which should be valuable for designers and
evaluators of 3D user interfaces. The results highlight the importance of structural elements in the design of a semantically based user interface,
because search strategies of users relied heavily on these mechanisms in the design. The results of this study also demonstrate that we are able to
characterise and learn from users' search strategies in a visual environment strongly shaped by semantic relationships of the information content.
Implications for user interface design based on users’ psychological models of a semantic space are described.