Efficient Usability Analyzer

Five years ago I learned about 4 basic concepts that dramatically changed my approach to usability. These were that it must be fast, efficient, simple, and focused.

Fast, efficient and simple are concepts that are based on old mathematical concepts.

Fast is connected to "GOMS keystroke model"
"The fewer times you click on something and the fewer times you move your hand between the keyboard and the mouse, the faster it is to use".
Efficient is connected to "Fitt's Law"
"A large target close to you, is easier to hit than a small target far away"
Simple is connected to "Hick's Law"
"The fewer choices you have, the easier it is to choose between them"

Sounds simple right?

Focus is equally simple
"Less distractions, enables you to better focus on the task at hand"

The best part of this approach is not that it is simple. It is that you can measure how usable your product is, before you show it to other people (like in user testing).

Prior to using this concept, I made a lot products that I believed was good. But the problem was, I didn't know until I had tested it with real people. Many times I tried making something, only to see people confused by it in a user testing. This was an unacceptable situation.

The major problem with user testing is that it comes after the work phase. It is a problem because it means that you are working blindfolded. You don't know the result of your work until it has been completed. This also means that you do not have control over your project plan (or that you have to plan everything to take longer than it should).

User testing is great for getting the finer details right, not as a general usability measurement.

Instead you need a tool that help you analyze the work ahead of you - before you start doing anything. This is one of the strong points of the "Fast, Efficient, Simple and Focused" concept.

The Analyzer

I have made an Excel analyzer (freebie) to help you do this. I have been using this successfully for the last 5 years - for both analyzing the best solution to a problem or when evaluated other people's work.

For me, it has been a tremendous time saver. It has allowed me to focus more on actually solving people goals and less time one guessing what approach is best.

Download

Usability Analyzer (Excel file - Metric)
Usability Analyzer (Excel file - Imperial)

How to use?

You use the analyzer by entering each step you need to take to perform a certain task. When doing comparison analysis it is important that your start and end points are the same. If you start with the mouse in one analysis, you must also start with the mouse in any subsequent analysis.

See also: "AJAX Type Ahead" example below

Selectors

Click: Indicates mouse clicks or use of the keyboard. Type in either the number of times you have to click on something (like "2" if you are double-clicking or "9" if you type the word "Usability"). If you only have to click once you can use "x" or "1"
Hold / Release: Used to indicate press and holding your mouse button or a key on the keyboard - and releasing it later. Using the shortcut for "copy" would be "Hold (CTRL) + Click (C) + Release (CTRL)"
Pointing: Indicates pointing your mouse cursor to a specific object (like a menu item, icon or form field etc.) You can either use pointing alone (uses GOMS) or add the distance you have to point using "Distance" and "target" (this uses FITT's and gives you a more accurate time).
Moving: Indicates moving an item from one point to another. Dragging an item is indicated by "Hold + Moving + Release". As with pointing you can define the exact distance using "Distance" and "target"
Distance: Used with either pointing or moving (cannot be used alone). Indicates the exact distance you move your mouse - measured in millimeters / inches (depending on which file you downloaded).
Note: This is how far you move your mouse on the table, not on screen.
Target: Used with either pointing or moving (cannot be used alone). Indicates the size of the target object (menu item, icon, object on screen etc.) - measured in millimeters / inches (depending on which file you downloaded).
Note: This is how far you move your mouse on the table, not on screen.
Switching: Indicates when you switch between input devices - like moving your hand from the mouse to the keyboard.
Deciding: Indicates when you have to decide on something. Is added every time you do something new. You can either type "x" to indicate a single decision (like when you decide to click "submit") - in which case it uses GOMS to calculate the time. Or you can add a specific number of decisions, as when you got more than one choice to choose from (like when you have to select an item in a drop-down list - then it uses Hick's which is more accurate. You add a decision point every time you do something different - or when the next task involves choosing between 1 or more items.
Waiting: Indicates the number of seconds you have to wait for the system to respond.

Example #1: AJAX Type Ahead in search

Type-ahead is popping up everywhere. We got Apple Spotlight, Google Suggest and Windows Vista's "Start Search" as well as a huge number of varies implementations in AJAX and Web 2.0 applications. But how does it actually in perform in terms of efficient usability?

Analysis parameters:

Interface elements include: Search box, submit button, and a area for the result.
Action start: the user start typing in the search input field
Action end: the user reaches the destination page
Complete input text: "GOMS Usability" (14 characters)
Result for "GOMS Usability" is 10 pages
The desired destination page is always the third result
Type-ahead results: the user needs to type in "goms usabi" to get a correct match (10 characters)
Distance from initial mouse pointer position to the third result (24mm - target: 8mm)

Base comparison, No type-ahead
Action: Type in the full search keyword, submit, activate the third result.

Instant type ahead: The results will appear immediately after each keystroke
Action: Start to type the full keyword, evaluate each result until we have a match, activate the third result

Instant type ahead - keyboard only: The results will appear immediately after each keystroke, and the result can be activated using the keyboard (key down x 3 + Enter)
Action: Start to type the full keyword, evaluate each result until we have a match, activate the third result

Delayed type-ahead: The results will appear 0.5 seconds after the user has stopped typing
Action: Type in the full search keyword, wait for the results, activate the third result.

Goggle Suggest type-ahead: The result is not the actual pages, but search terms.
Action: Start to type the full keyword, evaluate each result until we have a match, highlight the match, evaluate the results, activate the third result.

Comparison results: There isn't much to gain by using AJAX with search. All methods, except delayed type-ahead, is less efficient than a non-AJAX solution - with Google Suggest being the worst implementation of all.

Both Apple's Spotlight and Windows Vista's "Start Search" uses delayed type-ahead.

Notice: The analysis for both instant and Google Suggest has been simplified. The decision steps should have been repeated * the power of practice instead of simply grouping them together. The result is thus faster than in a correct analysis.

Example #2: AJAX Type Ahead in form fields

Another use for AJAX is to quick fill form fields, like when you add resources to tasks in a project.

Analysis parameters:

Interface elements include a single form field
Action start: When the user starts typing
Action end: When the user has move to the next field (using [TAB])
Input text is "thomas baekdal" (14 characters) or "tb" when using initials (2 characters)
Result for "thomas baekdal" is 3 items, of which the second item is the correct one
Type-ahead results: the user needs to type in "thoma" to get a correct match (5 characters)
Distance from initial mouse pointer position to the second result (9mm - target: 2mm)

Base comparison: No type-ahead - full name
Action: Type in the full name of the resource, move to the next field [TAB]

Automatic type-ahead: Type in initials and automatically convert them
Action: Type in the initials, move to the next field [TAB]

Instant type-ahead: The results will appear immediately after each keystroke
Action: Start to type the full keyword, evaluate each result until we have a match, activate the second result, move the to next field.

Instant type ahead - keyboard only: The results will appear immediately after each keystroke, and the result can be activated using the keyboard (key down x 2 + Enter)
Action: Start to type the full keyword, evaluate each result until we have a match, activate the second result, move to the next field.

Comparison results: Again we see an interesting pattern. Using AJAX to create an instant type-ahead feature is directly counter-productive. It is much faster to simply do without. But, using AJAX to automate can create a much more usable experience - in this case it speeds up the process by an amazing 500%

Notice: The analysis for instant type-ahead has again been simplified.

Comments

Marius Popescu - Jun. 7, 2006

The link of this article didn't work from your site's feed (in your feed it is http://www.baekdal.com/articles/na/analyzer/).

Just to let you know of this ...

Best regards.

Note: I tried to send this by email but the email address is not comprehensible from your contact page.

Thomas Baekdal - Jun. 7, 2006

Thanks for the heads up. I noticed it yesterday, when I published the report. The RSS feed has been fixed.

Folletto Malefico - Jun. 7, 2006

Yep, noticed and bookmarked. This page is a really good reference for me!

Christopher Fahey - Jun. 8, 2006

The "type ahead" feature is not necessarily intended to be used in the way that your test case uses it. At the very least, a user can use a "type ahead" style search form and ignore the suggestions/results completely, clicking submit and viewing results in the normal fashion. Simply having this feature doesn't necessarily slow the user down, as your conclusion seems to suggest. (In fact, I'd bet that *most* users ignore the suggestions and that most users simply type, submit, & view results like they always do.)

Also, type ahead suggestion features can have value by just giving the user a rough idea of what kinds of results their search term might generate before they click "submit". In fact, this "rough idea" is something that can be evaluated by the user entirely subconsciously, in their peripheral vision, without slowing them down on their path towards using the search in the normal fashion (type, submit, view results). In fact, it can be helpful even to those who pay no attention to it: if I type a character that causes my suggestion list to suddenly go blank, I might notice that change even if I wasn't paying attention to the suggestion list at all. This change might trigger me to realize that I've mistyped and that I should examine what I typed before submitting it.

Also, "type ahead" style forms allow some users to avoid leaving the page they are looking at before clicking submit or selecting the term or result. The "efficiency" of this value cannot be measured in seconds.

Again, users do not have to do what your test model does (reviewing the terms/results after typing each character and/or making a selection from the suggested terms/results), and I question the value of measuring usability in seconds anyway. If you compare two similar tasks, where it takes 5 seconds to do the task one way versus 15 seconds to do the same task but to have a sense of "hey, that was fun" afterwards, I'm not sure that we can say that the 5-second version was a better user experience.

Thomas Baekdal - Jun. 8, 2006

Christopher, Thank you for your views!

The examples are not meant to be an in-depth analysis of AJAX type-ahead functionality. Their purpose is merely to illustrate how to use the analyzer. And, as such you are absolutely right that there can be many other uses not illustrated here.

The purpose of the analyzer is to provide you with a tool to find the most efficient and painless way to get things done - to solve a specific goal. As such, you are also right that the analyzer does not help you to create explorative interfaces or to discover where people experience the "fun factor".

Take computer games. The analyzer can help you create an good interface for changing weapons in first person shooters or shifting gear in a NASCAR game. But it cannot help you make the game fun and exciting. It does not prevent it either, though.

The analyzer is like getting driving directions. It can tell you how to get from A to B as fast as possible, but not which roads that have the prettiest scenery.

I can recommend watching Will Wright's seminar on SimCity Online (Stanford University Video) or reading Don Normans Book "Emotional Design" to learn more about the fun factor of things

I do not agree with your other points. The Analyzer and GOMS, FITT's and HICK's in particular is designed to tell you the usefulness of different paths of interaction. You can very clearly see the difference in usability between using a page that displays the result without a page-reload vs. one that doesn't. And you can very clearly see how an interface performs where AJAX is used only as an extra layer of information.

These are the strong points of the tools.

Thomas Baekdal - Jun. 8, 2006

BTW: Jef Raskin's book "The Humane Interface" is a very good source for GOMS, FITT's and HICK's (it is old, but still highly relevant - and the book that originally introduced me to these concepts).

Christopher Fahey - Jun. 9, 2006

Good points, Thomas -- it's important to see these types of tools as only a small part of a usability evaluation toolkit. I've read Raskin and Norman's books, too.

I still am extremely skeptical of time- and formula-based UI evaluation tools, not just because of the "fun factor", but also because of other important intangible UI factors like confidence level (will this interaction lead me to where I want to go?), brand strength (does this UI feel like something this company would do?), trust (is this UI going to rip me off?), appropriateness (do I feel like I am being asked to do something I shouldn't have to do?) and others. The UI designer's job is a far, far bigger challenge than just maximizing efficiency, and I sometimes fear that a too-close focus on measurable efficiency will distract from other factors which in many cases should have a higher priority.

For example, imagine two different "Subscribe to Our Newsletter" interfaces. One of them takes 60 seconds to use, the other takes ten, but imagine that the first one clearly describes the newsletter's value and allows the user to be confident that they won't be spammed, while the second one does neither. I'd choose the slower design, but I could never come to this conclusion using a spreadsheet.

Thomas Baekdal - Jun. 10, 2006

Christopher, I don't share you skeptics but you do have valid points. I am currently writing an article about the difference between usability and user-experience, which I think is relevant to this discussion.

Folletto Malefico - Jun. 14, 2006

I assume this kind of analysis as relevant, but only as one tool between many others.

This kind of analysis doesn't give us any kind of reference on why the "60 seconds interface" in your hypothesis works better, sure, but still it isn't its target.

It simply tells you that the solution you're using "consumes" at least 60 seconds... if you could manage to do an interface equally appealing but that takes just 30 seconds, you've improved the interface: and this is what this test is about. ;)

Thomas Baekdal - Jun. 19, 2006

Folletto,

Exactly, I could not have put it better myself.

I have published my article about the difference between usability and user-experience