Andy Edmonds
April, 2004. IRB Exempt MS Thesis Work
Clemson University, Advanced Reading Technology Group, Applied Psychology
Thesis Full Text (PDF)
Advisor: Dr. Lee Gugerty

Introduction: Visualizing Menu Mousing
First Steps
Making it Real
The Payoff: Understanding Behavior
To be continued
Related Links & References
1. Inbound Links
2. References

Introduction: Visualizing Menu Mousing

While the core purpose of this research is not to characterize the ways that people use menus, this result is a nice, planned-for bonus. In the visualizations below, users are making 4 clicks typically. The first starts the trial, hiding the target word, and reveals a top level category beneath the click. The user clicks the top level category starting the timer for subtrial 1. The second click is on a category in the second menu sequence, ending subtrial 1 and starting subtrial 2. Finally, the user clicks a terminal node in a third menu ending subtrial 2.

The menu was implemented with the Mozilla XUL toolkit and mouse movements logged with an "onMouseMove" listener yielding observations as rapid as one every 10 milliseconds.

Prior art has suggested binding pause duration to blob size (Lockerd & Meuller, 2000) and color range to absolute time (Gellner & Forbrig, 2003). This system goes beyond this in also conveying mouse movement speed more directly and distinctly from pauses.

First Steps

A multi-testcase was crafted to validate the drawing routine across multiple data files. The images below depict all the movements during a trial including a warm-up and homing behavior that's not of especial interst.

My first curiousity was time and acceleration/deceleration. The image below shows a solution I arrived at rapidly. By drawing one to 15ish pixel wide lines, with a touch of transparency, for each mouse movement observation based upon movement speed.

In this first iteration, line stroke width is inversely related to distance traveled. The level of acceleration for this image is greater than subsequent ones as this was recorded using a laptop trackpad, which due to it's small size, promotes (and defaults to) higher mouse acceleration settings than a typical desktop pc.

Making it Real

To really understand the mousing behavior one needs to see the UI that produced it. In the subsequent pictures featuring the "Leaders" options, the menu is representative but not the exact menu the participants used. The right hand pane displays global details about the trial and the results of some sequential data analysis routines not yet perfected (and hence blurred).

I eventually arrived at the formula line stroke-width equals (pixels traveled / milliseconds) * 50. Note, this reverses the earlier mapping of speed to width. In subsequent visualizations, wider lines indicate faster travel providing a clear view of acceleration and deceleration.

The yellow circles indicate pauses. More precision or at least a legend is still needed. Technically speaking, the width of a the circle is a multiplicative of the log of the pause duration in milliseconds.

The critical innovation here is mapping absolute color to time, with color ranges representing 250 milliseconds. There are only 8 colors, so the pattern is repeated for durations over 2 seconds. A fully color graded algorithm will facilitate a longer range colors and more precise timing assessment.

In the picture below the user moved from the first to second menu in about half a second. A pause of almost 3 seconds followed. This trace exhibits two interesting phenomena described in detail below: Think First, Move Second and Upward Arcing as an alternative to Shortcutting.

The Payoff: Understanding Behavior

Scanning & Moving

The menu below shows two interesting things. We see in the 2nd menu the user is slowly moving the cursor down the menu, presumably as they conduct a visual scan.

Please note: The menu is representative but the actual labels for the trial shown in this image were different.

The visualization does not include the visual feedback the user gets as the mouse over a target. The final click involves the user just barely reaching the menu-item for the final click. The reverse-video feedback makes this plausible, though the menu vs line positioning still has a potential 0 to +3 pixel margin of error.

Shortcutting

One of the errors most easily observed and introspectively recalled errors is the making a direct path from the start to the target. The menu system used in this study was somewhat forgiving of this situation, as long as the mouse path rapidly went past the n+1 menu item.

Upward Arcing

In a related situation, this user seems to use a micro-strategy to facilitate downward motion in subsequent menus by raising the mouse prior to making a ballistic or highest velocity motion to cross the menu barrier at a down-right angle.

In this user trial, the effect is particularly noticeable for the second menu but also present in the first. There are also signs of this strategy in first trial in under the "Making it Real" heading.

It should be noted that perfectly horizontal movements are particularly difficult with a mouse in a rapid fashion.

User Strategy: Think First

UIE (Ojakaar, n. d) has reported observing users "thinking first" and moving second in cascading menus. Here we see this behavior with movement initiating late/slow in the first quarter second and a long pause or two after clicking starting the 2nd subtrial.

Acceleration

A wide variety of acceleration/deceleration patterns can be seen. The image to the left shows a scenario that's easy to imagine, a rapid initial movement followed by short late stage corrections. This image is notable for a rapid acceleration and more gradual deceleration.

To be continued

The visualization will drive the development of data analysis mechanisms which locate incidents of particular patterns using techniques called sequential data analysis. This can then provide a map to visualize multiple instances for better characterization. In the long run, understanding the nature and frequency of different strategies and errors will provide insight into better constructing UI for selection from a set of meta-data enhanced alternatives.

There are also a number of polish issues including proper z-index stacking, positioning QA, and tooltips. A multi-trial render will also be needed, necessitating a zoomed out view. Slow-mo, realtime, and speeded playback are probably overkill, but who knows?

Scientifically, I expect to provide evidence of micro-strategies (Gray & Boehm, 2000) and am seeking to better understand the nature of mouse movements. Fitt's law describes mousing very generally, relating speed to distance and target size. Recent work on expanding targets calls into question the notion of an initial ballistic movement as the key method of mouse movement.

Feedback to andyed AT surfmind.