When to observe users. User observation should be an integral part of the design process---from the initial concept to the product's release. Software design that includes user observation is an iterative process; user feedback provides the data for making design modifications. As Figure 1 demonstrates, this iterative process assumes that preliminary human interface designs should exist prior to the development of underlying code. Interface designs should be tested frequently to determine which design should be implemented. Then, as the code develops, the entire product should be tested and revised several times.
Preparing for a user observation
Set an objective. Before you do any testing, you
should take time to figure out what you're testing and what you're not.
In other words, determine an objective for your test that focuses on a
specific aspect of the product. By limiting the scope of the test, you're
more likely to get information that helps you solve a specific problem.
Design the tasks. Your test participant will work through one or more specific tasks. These tasks should be real tasks that you expect most users will do when they use your product. The entire user observation should not run over an hour, so you should design tasks that focus on the part of the product you're studying. For example, if you want to know whether your menus are useful, you could design a task that requires the participant to access the menus frequently. After you determine which tasks to use, write them out as short, simple instructions.
Important: Your instructions must be clear and complete, but they should not explain how to do things you're trying to test. For example, if you want to find out whether users can navigate through your program easily, don't give them instructions for navigation. Or, if you want to know whether your interface is self-explanatory, don't describe how it works. This concept is extremely important to remember. If you teach your participants about something you're trying to test, your data will not be useful.Decide upon the use of videotape. Although you can observe users effectively without using special recording equipment, you may want to use videotape to capture the entire session. By videotaping the session, you collect an enormous amount of valuable information that you can review and analyze after the test is over. If video equipment is not available, a tape recorder can be helpful for recording what is said during the test.
Determine the setting. The ideal setting for user observation is a quiet, enclosed room with a desk, the appropriate hardware and software, a video camera, and two microphones (one for you and one for the participant). Of course, you may not have all these things available when you need to observe; therefore, you should try to approximate the ideal setting as closely as you can. If you have to conduct the observation in a regular office, ask the people around you to keep the noise level down during the observation. The key is to make the environment as interruption-free as possible. Get the participants out of their offices, away from phone calls and people who might drop by.
Find representative users. When looking for participants,
try to find people who have the same experience level as the typical user
for your product. Don't ask people you work with regularly to be participants
because they are probably familiar with your product or your opinions about
the product. Generally, you should look for people who are familiar with
the hardware you use but are not familiar with your product.
You may want to ask pairs of people to work together
on your tasks. You'll find that people working in pairs usually talk more
than people working alone, and they also tend to discuss features of the
product and explain things to each other.
- You're helping us by trying out this product in its early stages.
- We're looking for places where the product may be difficult to use.
- If you have trouble with some of the tasks, it's the product's fault, not yours. Don't feel bad; that's exactly what we're looking for.
- If we can locate the trouble spots, then we can go back and improve the product.
- Remember, we're testing the product, not you.
- Although I don't know of any reason for this to happen, if you should become uncomfortable or find this test objectionable in any way, you are free to quit at any time.
Unfortunately, most people feel awkward or self-conscious
about thinking aloud. Explain why you want participants to think aloud,
and demonstrate how to do it. For example, you could say:
Of course, there may be situations where you have
to step in and provide assistance, but you should decide what those situations
might be before you begin testing. For example, you may decide that you
can allow someone to flounder for at least 3 minutes before you provide
assistance. Or, you may decide that there is a distinct set of problems
with which you can provide help.
As a rule of thumb, try not to give your test participants any more information than the true users of your product will have. Following are some things you can say to the participant:
- As you're working through the exercises, I won't be able to provide help or answer questions. This is because we want to create the most realistic situation possible.
- Even though I won't be able to answer your questions, please ask them anyway. It's very important that I capture all your questions and comments on tape.
- When you've finished all the exercises, I'll answer any questions you still have.
Important: If you need to demonstrate
your product before the user observation begins, be sure you don't demonstrate
something you're trying to test. (For example, if you want to know whether
users can figure out how to use certain tools, don't show them how to use
the tools before the test.)
- explain what you were trying to find out during the test.
- answer any remaining questions the participant may have.
- discuss any interesting behaviors you would like the participant to explain.
To get the most out of your test results, review all
your data carefully and thoroughly (notes, the video tape or cassette tape,
the tasks, etc.). Look for places where participants had trouble, and see
if you can determine how your product could be changed to alleviate the
problems. Look for patterns in the participants' behavior that might tell
you whether the product was understood correctly.
It's a good idea to keep a record of what you found during the test. That way, you have documentation to support your design decisions and you can see trends in users' behavior. After you've examined the results and summarized the important findings, fix the problems you found and test the product again. By testing your product more than once, you can see how your changes affect users' performance.