 |
Bonnie Fournier:
How a bug becomes a fix
Interview by
Lisa Daly and
Betsy Kosheff
 
Level: Beginner
Works with: All
Updated: 06/20/2001
Related links:
Notes/Domino Fix List Database
Download page
Lotus Knowledge Base
MR/MU FAQ
Get the PDF:
 (156 KB)
|  | 
| Editor's note: This interview was updated in April, 2001, to reflect changes in the Notes/Domino maintenance release process. |
They may not have the fanfare and media coverage of a major release, but Maintenance Releases and Maintenance Updates are a hot item in the "real world." These Notes and Domino point releases come out at a dizzying pace, and Bonnie Fournier is the Iris Program Manager that keeps a handle on it all. Here's the inside scoop on the development process and the exciting new Notes/Domino Maintenance Release Strategy that is targeted directly at improving code quality.
To start off with, what is a Maintenance Release or "MR" by today's definition?
Each time a new feature release is introduced to the market, a subsequent trail of regularly scheduled bug fix releases are typically provided.
A maintenance release is denoted by the use of a third digit in the release number. For example, during April 2001, we're working on R5.0.8, which is the latest maintenance release for the R5.0.x code stream.
What is the difference between a Maintenance Release (MR) and a Quarterly Maintenance Release (QMR)?
In April 2001, we made a change in our Notes/Domino Maintenance Strategy. We moved from a three month development cycle to a four month development cycle. Therefore, the "Quarterly" in Quarterly Maintenance Release no longer applied. We renamed the process to "Maintenance Release" or "MR."
Our releases will now be approximately 120 days apart. This new schedule allows our development and quality engineering testing teams more time to focus on quality assurance and client/server reliability.
What was the purpose of this change in schedule?
This change was made in direct support of increasing customer satisfaction and in particular around product quality. We believe that our new four month Maintenance Release schedule will align itself well with our customers' desire to receive regular maintenance on Notes/Domino, while assuring the highest quality level possible.
We have allocated an additional week toward development within the cycle and the remaining three additional weeks will be dedicated to product testing. We are also making changes to some of our internal deployment milestones. We will now have a Silver Candidate as well as a Gold Candidate. The Silver Candidate will have the same code quality as the Gold Candidate from the previous three month cycle. We will then use the three weeks between Silver and Gold to conduct additional enterprise-level testing and beta testing of the Maintenance Release.
The change in schedule is also an answer to many customer requests. We have been told that deploying new maintenance releases four times a year is too much for many customers. They understood that Lotus did not expect them to deploy every QMR that we shipped, but the fact that the QMR existed caused pressure from within their organizations. End users would hear of the existing QMRs and ask why they weren't yet deployed. Having fewer of them out there would help ease this a bit. Customers have requested this change for a long time.
Which releases will this affect?
We will be altering our 5.0.8 schedule to reflect this change as well as the remaining R5.0.x releases. This means that the target maintenance for R5.0.x for the rest of 2001 looks like this:
July 2, 2001 - R5.0.8 Web FCS
July 16, 2001 - R5.0.8 CD FCS
November 1, 2001 - R5.0.9 Web FCS
November 16, 2001 - R5.0.9 CD FCS
What does this mean to Lotus customers?
Reducing our schedule to three Maintenance Releases per year will allow customers more time to plan their business design goals and schedule infrastructure deployments.
This new schedule also allows customers more time to work with the MR beta. We typically did not have enough time in the QMR schedule to run a beta on these releases. We think that with the extra month in place we can accomplish this. Expect to hear more on that later in 2001. We hope to use the beta process to increase product reliability and stability.
Please note that we will continue to honor two Maintenance Releases past a major release or a minor point release. We reserve the right to change the schedule based on the market demand.
How does this change affect the current Quarterly Maintenance Update (QMU) strategy?
Currently, between Quarterly Maintenance Release cycles, unscheduled updates are sometimes posted. These are currently known as Quarterly Maintenance Updates or QMUs. QMUs are for the urgent, critical customer problems that are discovered between regularly scheduled releases.
The naming convention has also changed for QMUs, since "Quarterly" no longer applies. To reflect this change in our schedule, we have renamed our QMUs from Quarterly Maintenance Update to "Maintenance Update" or MUs.
We will continue to produce Maintenance Updates only on an as-needed basis. So, for example, we might release an MU to fix a security or interoperability issue. An MU is, therefore, a small release, with only a few fixes, or perhaps just one fix.
Also, an MU may not be for all platforms. This may be due to any number of reasons, including the fact that the problems fixed are not applicable to those platforms or the timing of the MU is such that some platforms can roll those fixes into their upcoming regular release.
How do I know when a release is an MU, versus an MR?
It's very easy—an MR is represented by a number and an MU has an additional letter. For example, the MU for R5.0.4 was R5.0.4a. These releases of Domino and Notes supersede the previous MR. They contain all of the fixes in the previous release, plus the additional fixes for the bugs being addressed in the MU. MUs are fully certified and supported by Lotus. Unlike the Hotfixes provided by Lotus during an MR development period, MUs do not expire upon the release of the next MR. They continue to be supported in the same manner that the superseded MR would have been.
How are the MR schedules determined?
Every 120 days, we release an MR for our active Domino and Notes code stream. So, for example, we released 5.0.7 in March 2001 and have targeted 5.0.8 for July 2001. We normally only have two MR code streams in development at a time. Our next set will be the R5.0.x and the Notes/Domino "Rnext" code streams. (Rnext is the code name for the next major release of Notes/Domino). There is typically a two- to three-week, or four- to six-week, difference due to the resource constraints of the release team.
Who makes up the MR release team? And, what is your role as the Program Manager?
It actually takes a wide range of people to put together an MR release. We have release engineers that do the actual building of the kits, and an MR quality engineering (QE) team for testing those kits. The MR QE team also focuses on all of the bug fixes and regression testing in each of the builds.
But, even before we get to that point, we determine what fixes should get built into the MR. We receive input on this decision from several different sources, including the Business Partner support group, Lotus Professional Services, Field Support Services, Support Engineering Team, Support Account Managers, our Critical Situation (CritSit) teams, and Product Management.
We then depend on the developers to code their fixes, and for QE to test the fixes. We also have international folks who prepare their teams for localization, an operations group that prepares distributions for our customers, product managers who work with the development team and others to incorporate certifications, marketing folks who help us to hit our market goals, and support people who help our customers communicate their needs in order to keep up with the product. Lastly, we're excited to report that we've recently added a technical editor to help publish the Fix List Database on Notes.net. I think that's everyone!
My role as the Program Manager is to manage the whole process, which involves doing a lot of different things. For example, it's my job to keep track of the project schedule and specific milestones, and then to communicate this information out to all the different groups involved in the release. A large part of this is knowing the right person to contact for disseminating the information. In addition, I do a lot of "fire-fighting" to solve the new issues that come up on a daily basis, and organizing of people to perform new tasks. I also coordinate the actual release process, including the escalation of critical bugs to development, when appropriate. One of the best things about working at Iris is the whole team atmosphere. Everyone wants to make the product a great product for the customer, which makes my job a lot easier. We have a great mix of expertise in all of the development areas, and they all complement each other in one way or another. Most of all, we also have a lot of fun in the process.
So, how does a bug qualify to get fixed in an MR? Can you tell us more about the process for developing MRs?
First, we have a "triage" to gather all of the Software Problem Reports (SPRs) requested from customers to be fixed. We rank them by importance, based on the severity of the problem, associated business cases, and the number of customers affected. Because of the schedule-driven nature of these releases, not all problems can be fixed immediately. And, as I just mentioned, we gather input from several different sources when making the decision of what goes into each release. Then, the developers do their coding and QE does their testing. Once we have a build that includes a particular fix, we may send out a Beta to customers requesting that they verify the fix. When they confirm the problem has been fixed, the code goes into the Silver Candidate build. We use three weeks between our Silver and Gold builds to conduct enterprise-level testing. We test our weekly builds on internal IBM, Lotus, and Iris servers to test stability. IBM and Lotus deploy the Gold build on our internal servers and then we post the Incremental Installer files on Notes.net. From there, the product goes to manufacturing for distribution to our customers.
What exactly is "First Customer Ship"? Is it different from distributing the MRs on Notes.net?
First Customer Ship (FCS) is the first drop of CDs that go out to our customers. For anyone familiar with IBM's terms, it's the equivalent of IBM General Availability (GA) when it first becomes available. The FCS software is the same exact software that we post on Notes.net. The only exception to this has been release 5.0.8, which was the first version of Domino to support iNotes Web Access. Because iNotes Web Access is a major feature enhancement, it was excluded from the Incremental Installers here on Notes.net and is only available to customers on Passport Advantage's software subscription and those customers acquiring new licenses of Notes or iNotes R5.0.8.
In fact, the first place we distribute MRs and MUs is on Notes.net (as Incremental Installers) to give our customers a head start at reviewing the software before we actually ship the CDs to their offices. We ship the CDs approximately three weeks after we post the files on the Notes.net site.
How do customers know when a new MR is available? Is there any notification?
We send out channel announcements to tell them about the estimated FCS time, and when they should expect to receive their CDs. We usually send out these announcements around 30 days before we ship an MR, and on average, they can usually expect their CDs to arrive in approximately one to two weeks time, or as soon as possible.
In addition, when we post the files to Notes.net, we add notices about the new release to the welcome page and you can register with Notes.net (or update your existing registration) to receive site announcement e-mail that will notify you when a new download is available.
Also, our sales force, technical support staff, business partners, and customers all eagerly await the Maintenance Releases, so they can see if their particular fix made it into the MR.
And, how can I find out what fixes are included in the MR?
Thanks to Will Raabe, General Manager of Support Engineering, there is now a powerful new tool available on Notes.net. It's called the Notes/Domino Fix List Database. From this database, you can see which fixes are shipped in MRs and MUs, beginning with the 5.0.x code stream—and, most importantly, which fixes will be in the next MR or MU.
This tool was created in response to our customers' requests for information on upcoming MRs. It's important for me to mention that fixes for upcoming MRs are posted as they are identified throughout the development cycle, and before the final product testing has been completed. Therefore, certain fix list entries may be pulled at any time, if regression or performance problems are introduced. Fix list entries for upcoming MRs are identified by a red exclamation point at the view level, and by a disclaimer on each fix list entry. Once the final MR testing has been completed and the MR has been officially released, the disclaimers are removed, indicating that the fix list entries are final.
You can sort the fix list entries in different ways—by release, SPR number, technical area—and the database is full-text searchable. You can read Lotus Customer Support Technotes for the associated entries and print individual entries. (Additional Technotes are located at Lotus Support Services.)
We're very excited about this new application and hope that you'll find it helpful. We're also excited about a new addition to the group, Lisa Daly, who as technical editor is responsible for publishing the Fix List entries to the Web. As SPRs are fixed, Lisa works with the developer and/or quality engineer to write an explanation of the fix.
How do I track a specific fix?
Each fix submitted to a particular release includes an SPR number, the releases where you can find the fix, the technical area for the fix, and a one-line description of the fix. For example, in the following Fix List entry, you can see that we have addressed the fix for this SPR in release 5.0.6.
How can I find out more information regarding a specific fix listed in the Fix List Database?
Customer-reported issues are documented once a workaround or solution has been identified. This document is found in the Lotus Knowledge Base. In addition to the Knowledge Base, the development team documents any "known issues" for an MR in the Release Notes before we ship. These known issues are for problems that we were not able to address before shipping the final MR. Typically an SPR is identified with a particular "release-noted" issue for ease of tracking by our customers.
What's the point of the Incremental Installer?
Well, the average Notes installation kit is 30 MB, so instead of having to get an entire Notes install kit for each build, you can download a smaller version. It uses a binary difference engine to apply only the required changes to each program file, making it an exact duplicate of the target program file. The size of the Incremental Installer depends on the scope of the changes between Notes releases.
You can get more details and technical information about the Incremental Installers from the MR/MU FAQ.
Do I need a password to download the Incremental Installer?
No, it's a free download. You just click on the Download icon to get to the Download page. Then, select "Notes/Domino Incremental Installers" and follow the links to the particular installer version you want to download. Before beginning the download, you can review the instructions for running the Incremental Installer, or access the Fix List Database to see the list of fixes that went into that particular release.
What kind of support is provided for the Incremental Installer?
Support is covered under existing Notes Support contracts with the exception of problems encountered accessing the Web, for which you should contact your Internet service provider.
Do other vendors have a similar approaches to the Incremental Installer—or is this unique to Notes?
Iris first came up with the idea to distribute code to our customers via the Web. Our customers love the idea of not having to wait for the CD to ship before they can find out if their SPR has been fixed.
Is it possible to find out the status of future MRs and MUs?
I'm glad you asked that question. We also have a new Notes/Domino MR/MU Status page on Notes.net. This page displays the status of each of our R5.0.x releases. For example, if you select 5.0.8 as the release number, as of today, you will see a red dot next to "Code Freeze", which means that our developers have reached a deadline for submitting fixes into the MR. This allows our quality engineering team time to start comprehensive product testing. The status will change to Silver Candidate, Gold Candidate, etcetera, as we reach each step of product development. This is helpful for customers to understand where we are in our development cycle so they can be proactive with their Notes/Domino deployments.
The new Status page sounds great! How can customers provide feedback on these new MR/MU Notes.net features?
We love to hear from our customers. You can use our feedback form to let us know what kinds of information you would like to see displayed on our site, what you like/dislike about our current site, etcetera.
What's in the future for MRs and MUs?
Our goal is to continue to make MRs and MUs available on Notes.net and to continue putting out the best product that we can. We're always investigating new ideas and tools to use for distributing the releases. We're lucky, though, in that our current methods work, and we have a great customer base that lets us know when problems do occur. We're happy with the new Fix List process and Status Page and have heard lots of positive feedback. And that was in response to our customer's requests. We hope to have more enhancements in the coming months. Please let us know your thoughts on the release process at MRFeedback@iris.com!
ABOUT BONNIE
Bonnie joined Iris in 1996 from Lotus, where she worked in the Customer Support area. During her four years at Lotus (1992-1996), Bonnie advanced from being a customer support analyst, to a senior support analyst, to a Product Line Consultant. Working for Lotus gave Bonnie a great background on the different areas within Lotus and how they work, who the people/team players are, and how to get the job done. Prior to working for Lotus, she worked for a company called Bolt Beranek and Newman for 10 years. Outside of work, Bonnie likes to spend all her free time with her 12-year-old daughter, Brittany Lea, and they have lots of fun together. She is an outgoing native New Englander with a variety of interests, such as music, travel, sports, and movies. Bonnie recently built a house in the Westford area; this effort required her to use her organizational and communication skills (along with her great sense of humor). |