Notes/Domino Usability Team
by
Laura
Rutherford
Level:
All
Works with:
All
Updated:
03-Mar-2000
[Editor's note: The Usability Team would like to know what you think about the usability of Notes and Domino. After reading this interview, visit the
Developer Spotlight Forum
to post your thoughts.]
The Notes/Domino Usability Team --
Kara Coyne, Eileen Driscoll, Rich Buttiglieri,
and
Sheri Branco
-- has the tricky but rewarding task of working with development to make the software easy to use for different types of users. Read on to learn what members of the Notes/Domino Usability Team have to say about everything from finding users to test the product to helping developers and designers implement test results. They also speak about the important role usability testing had in R5 and the role it will have in future releases.
Can you briefly explain usability and the role it plays in software development?
Kara Coyne
Usability is about trying to make a product transparent to whomever is using it. So, for example, if you walked up to a door, you should know what to do to open it. It’s much trickier with software because there are more components. Basically if somebody goes to a piece of software they should be able to use it, do their main tasks they want to do with it, and not be confused and have decreased productivity. The goal is not usually to get people to like the software so much – what we want them to do is not even notice the software. We want it to be transparent so it can help them get their work done.
People are becoming a lot more savvy about usability. A lot of the large companies now have their own design and usability groups. People notice more these days when software is not usable. And they know there are other options out there, so they don't buy unusable software because they know that it is going to reduce productivity.
Kara Coyne
What are the key attributes of a usable product?
Kara Coyne
A usable product is one that a person can work with and get the job done without noticing the software. For example, think of the software like a public speaker – a good public speaker is not one that is distracting or that you really notice. Instead, the person is a vehicle for the information. Software shouldn't be something you notice – it should just be a vehicle for getting your tasks done.
It is possible to design a product that is usable for all levels of users?
Rich Buttiglieri
It is very difficult to do, but there are techniques to make a product very easy to use to brand new, novice users that can also satisfy the advanced user. Some of techniques you might use are to bury advanced features. I am sure you have seen this technique in dialogs. For instance, you will have a pop-up dialogue that will have some very simple controls on it and it will have a button that says “advanced” or “configure.” That technique is basically hiding the complex stuff from the novice user. By doing that, it gives the novice user a feeling of comfort. It gives the power user enough control so that they can be satisfied with the usability of the product. It’s hard to satisfy everyone if you’ve got a range of novice users to super high level power users, but there are techniques to use.
Rich Buttiglieri
Is usability testing new to software development?
Kara Coyne
No, it has been happening for a long time. Some of the more renowned people like Jakob Nielson, Ben Schneiderman, and folks from Xerox PARC started doing usability years ago. I think in the last five years it has really become more mainstream and integrated as part of the development and design processes. Even start-ups are looking for people to work on usability. Where it used to be more of an afterthought, now usability is more integrated in the process. But people have been doing research about it for a really long time.
One thing that has broken a lot of boundaries is the Usability Professionals' Association [UPA], which started, I believe, in 1991. At a conference for SIGCHI, which is a computer software user interface design organization, they had a Birds of a Feather and a couple of people jumped off and created this organization.
Now the UPA has grown to thousands of people, and we meet every summer and talk about usability. Probably eighty percent of the people are software people, but there are some hardware people and people who work on VCRs and other consumer products.
What’s the process involved in deciding what area of the software to test?
Rich Buttiglieri
There are few ways we decide. For instance, in 5.0 first we reviewed the top task survey data to make sure our "bread and butter" tasks were covered and easy to use. Second, we monitored the [product] specifications – if we saw something that was being redesigned but looked complex and was not already on our list, we put it on our list. Third, some UI [user interface] designers or developers would come to us requesting feedback, because maybe they were stuck deciding on going a few different ways with the UI, or they've completed what they want to ship but want to know if it's easy to use.
What’s the process involved in finding people to test the product in the lab?
Rich Buttiglieri
I start by talking to the project leaders, project managers, and UI designers about who the real target audience is for this particular area of software. I then write a user profile describing the background for our test participants and draft some screening questions for our usability assistant, Sheri. She then goes and searches our database of test participants looking for those key words. When Sheri comes up with a list, she will call those folks and screen them further to make sure that they have or don’t have experience with whatever software we are testing. So it starts with the people who build the product to find out who their target audience is.
How do you contact potential users to test the product?
Sheri Branco
When a usability request is put together by a team, they generally have a user profile of the type of tester that they are looking for (like “surfs the web,” “uses Notes 5.0,” “has taken a CBT”). From there I can search on specific terms in the participant database and hopefully my search results will bring up documents of potential testers with the information I'm looking for. From there I can either e-mail them or call them to see if they are available.
Sheri Branco
What is the process involved in checking potential testers' backgrounds?
Sheri Branco
We keep as much relevant information as we can in our participant database about potential users’ software backgrounds, job responsibilities, and so on. When participants first sign up, everything that they fill in on their
participation form
[located on the Notes.net
Usability page
] is entered into our participant database. We've recently renovated this document, adding more questions relevant to the products we test, to help us get a broader background. If participants are contacted or come in for a test, we can also fill in any new information that was gathered as a result of the contact.
Do you find that people are willing or even excited about participating in testing?
Sheri Branco
I do find that people are very willing and excited to participate. For the most part, the biggest hindrance is their schedules not permitting them to participate. I've had people disappointed when I've had to tell them that their profile doesn't match what we're looking for in a particular test. I often keep those people on a next call list because I know that they are very interested.
What are some of the ways users can provide usability feedback on Notes and Domino?
Kara Coyne
One thing that a lot of people are already using is the [Notes.net Gold Discussion] Forum. We look in there. Even though we don’t respond all the time, that’s one place that we look to get a pulse of the usability issues people are talking about. Sometimes we respond, but we are not as active as we are lurking there.
Another way to provide feedback is through the questionnaires that we periodically post on the Notes.net site. If we want to know about, for example, bookmarks, we will formulate a questionnaire.
If you are in the Boston area, another way is to sign up on our
web site
[on Notes.net] to be a usability tester. Or, you can sign up to host a site visit. Sometimes we visit people in their work environment to get a better understanding of how they use Notes or a competitive product in their real, day-to-day job. Or people can sign up to be a design partner. We call on design partners when we need some quick feedback on a new design or one they are already using – we might do a quick e-mail, telephone survey, or study with design partners and ask them about specific things. This method has been extremely helpful in the past, especially in getting international feedback from our users.
How do you determine how many users to test the product in the lab?
Eileen Driscoll
That comes down to talking to the developers and the designers of the product to get a feel for the user profile. Usually we’ll do three or four sessions at two hours apiece and sometimes it helps a lot to have a co-discovery (or two people) in a session. The two people tend to talk to one another and are more at ease. And it may even be more like a real-life situation as opposed to sitting in a room all by themselves where sometimes they can become more self-conscious and not give as much feedback.
So we try to do about four sessions. Sometimes those four sessions are made up of two people each, or sometimes we have two sessions with two people each and two with one person – it all sort of depends on the test and what we are testing.
What happens in a “typical” lab test?
Eileen Driscoll
Usually we sit people down and give them a little informative session for two or three minutes before we start. We tell them how they are not being tested, it is the software that is being tested, that they should not feel self-conscious, that we would love to hear as much feedback as possible, so we encourage them to talk as openly and honestly as possible. Then we tell them a little about what the tasks are like. We tell them we won’t talk to them through the test because we want them to forget that we are here. The only time we might talk is if we need clarification on something they did. For example, if someone opens a window and then closes it to do something totally different, I might ask them why they closed the window, but we don’t ask leading questions.
They then go through the tasks and we film them. After they are through, we go in and ask some follow-up questions or maybe if they did something totally intriguing we might ask why they decided to do that. If there are designers or developers in the session, they might have specific questions and we might bring a couple of questions in from them.
Who typically observes the lab tests?
Kara Coyne
The usability specialist, the product designer, usually some of the developers, some of the QE people, some of the user assistance people, and sometimes course developers are there. We invite anyone who might be involved in a feature, and we post the test schedules in our internal usability database. If you have time and you want to stop in the lab even for a few minutes, you can learn from that – even if it is not a feature that you are working on.
How you do work with developers in creating the test?
Rich Buttiglieri
For R5, some tests the usability team initiated and some tests the developers or designers initiated. Sometimes we saw areas of the product that were top tasks that were not getting covered in the usability lab. Other times the designer or the product manager would come to one of the usability team members and say “There’s an area that is being redesigned and I think it is really important, can we get it usability tested?”
So from there we go talk to the designers and maybe some of the lead developers and figure out what some of the top tasks are and the particular focus of the product. For instance, for [the Domino Designer feature] Outliner they might say the big things users are going to want to do is create an outline, add elements to it, customize it this way or that way. So they would tell me what the top areas of working with that product are and they would tell me what they want to learn from the usability test – Is it easy to use? Is it easy to learn? Can users figure out this particular feature set?
From those sets of questions, I’ll understand what they want to learn about the product. Then we can design a test protocol that brings the test participant through those areas of the product. And through this fictitious scenario, this role playing, the users will touch different areas of the product that I want to get the feedback on.
How do you organize and present test feedback to developers?
Eileen Driscoll
I tend to take the tape of the sessions that I’ve just conducted and go back into the lab on my own and watch the tapes and make a whole series of notes. From those notes, I piece together a report noting trends. For instance, if I notice that three out of four people in a session didn’t quite know how to do this task or went to the wrong menu option, then maybe there is something there that needs to be changed or modified. I write a report and then meet with the developers and designer to go over my findings. Sometimes the meeting is in the lab so I can show them the video highlights from the usability test. Then if someone says “Gee I don’t understand why they don’t get this” I can show them the video clips of the users testing the software.
Are developers receptive to test feedback?
Rich Buttiglieri
Yes. They are very receptive. Early on it takes a little bit of salesmanship to get them to appreciate the value of the usability data or the feedback that we give, but it only takes them an hour to sit through one usability session or through a half a session to internalize that the data we are giving them is really valuable. So if they sat through any part of the session, they are usually very open to the feedback. If they have never sat through a usability test it takes a little more explanation of yes, we really did bring the users through the proper tasks and yes, we really did bring the proper set of users in so we did not create a bias in any way.
How do you work with developers in implementing test results?
Kara Coyne
We review the findings from the report and from our issues database. We try to present the findings as pure and keep our own opinions or recommendations as separate. People ask for our opinions too and that is fine, but usually you want to keep it a little more scientific about what you found in the lab. So sometimes we re-watch the tapes and figure out what are the issues and why and then we figure out what things work and why so we don’t change those things. Our goal is not to get our recommendations done but to get the problem addressed in some way, and the recommendations can sometime be used or at least help describe the problem better.
To track it all, we put the information in a database. We put the full report in there. In another database, every issue is logged and rated by severity and status and the problem is explained. We also note the build that the problem was last seen in so we can go back and say “OK is this going to be addressed or not?” and “What should I do with this? Should I go talk with the developer or not? What stage in the product cycle are we in?” At some stages it is better to go talk to the developer or designer, but at other stages things have to go through a triage process because the schedule is tighter. The onus is on us to make sure that the problems that we found are solved in the build. And we really install builds and go through them and say “Hey this was supposed to be done, what happened here?” and follow up on it, or “Great this is done,” and then mark it as done.
What kind of role did usability testing play in R5?
Rich Buttiglieri
From where I sit, usability played a significant role. I think the developers and designers relied on us to get the feedback they needed to validate their designs and sometimes to find out how to address an issue. Sometimes they did not know, they had too many different ways to design a particular area of the product and they just could not decide what the users would do in this situation. So they would come to us and say “Here, tell us which design is better.”
I think overall usability testing helped Notes to reach a broader audience more quickly – meaning people who have never used Notes before would have a much easier time learning and using Notes right out of the box. This can mean a higher degree of satisfaction for our customers because they don’t have to send people to more training and people are more efficient right away with our product.
How will usability play a role in the next major release?
Kara Coyne
It’s already playing an important role. The usability team has been involved in the project plans. The usability activities are in the schedule, and they are subtasks dependant on developers’ tasks so that developers know that there is something coming after what they do. There are also different times that we have chunked out, special usability times when a lot of features are going to be done, and we are really going to pound on those. So that is something that all the developers know is coming. That’s a little bit different from 5.0 in that usability was newer in 5.0 and now it is more expected and people are ready for it – it is part of the product cycle.
The Usability Team would like to know what you think about the usability of Notes and Domino. Visit the
Developer Spotlight Forum
to post your thoughts.
ABOUT EILEEN
Eileen Driscoll is a Usability Specialist with Iris in Westford, Massachusetts.
ABOUT KARA
Kara Pernice Coyne is a principal specialist at Iris Associates, where she manages the usability testing program for Lotus Notes and Domino and various other applications.
She has worked in the usability field for nine years, and has created usability departments from the ground up at three different organizations. Among her achievements, Kara has designed a state-of-the-art usability lab, and has refined various testing methods such as remote testing and creatively using customer conferences to collect usability information. She has also worked as a user interface design consultant for several companies.
Kara is an active member of the Usability Professionals' Association (UPA), was chosen as one of two conference co-chairs for the 2000 and 2001 UPA conferences, and was the presentations committee co-chair for the 1999 UPA conference. She is also an active member of SIGCHI (Special Interest Group on Computer-Human Interaction).
Kara has an MBA from Northeastern University in Boston, Massachusetts, and a BA from Simmons College in Boston.
ABOUT RICH
Rich Buttiglieri is a Principal Usability Engineer who received a BS in computer science from the University of Lowell and a graduate certificate in Human-Computer Interaction (HCI) from UMass Lowell. In a past life, he was a software developer (mostly developing user interfaces) for nearly 10 years and now evaluates and designs user interfaces for ease-of-use and intuitiveness. He is also an active member of ACM’s SIGCHI and the local chapter, Greater Boston SIGCHI.
ABOUT SHERI
Sheri Branco joined Lotus in June 1999 as an assistant for the Usability Team in Westford, MA, and has continued in that role at Iris Associates. She previously worked as a Customer Service Representative for Datawatch Corporation. Sheri graduated from Westfield State College with a BS in Business Administration.
ABOUT LAURA
Laura worked as a user assistance writer for Lotus until she had her daughter, Kate, in January 1999. Now she spends a good amount of time with some sort of baby food mush on her clothing, daydreaming about the days when "sleeping late" involved an hour later than 7:00 am. Some of Laura's favorite things include her daughter (of course), her husband, her two dogs, and taking long hikes with all of them. Other favorite things include frozen cappuccinos, cheese pizza, and margaritas (plain, on the rocks, and without salt) from Sierras restaurant in Sudbury.
