 |
Early adoption of Notes/Domino 6 at IBM
Interview by
Tara Hall
Level: All
Works with: Notes/Domino 6
Updated: 10/01/2002
Related links:
Domino 6 Technical Overview
Notes 6 Technical Overview
Domino Designer 6 Technical Overview
Lessons learned from the Notes/Domino 6 Early Deployment program
What's in store for the Domino 6 database
Jim Rouleau on Domino 6 server availability
NotesBench Consortium
Get the PDF:
 (306 KB)
|
 |
James Grigsby and Cynthia Mamacos have been responsible for coordinating the internal Notes/Domino 6 Early Deployment program at IBM, the largest Notes/Domino deployment worldwide. We spoke with them to find out what early deployment at IBM has taught them about convincing users to become early adopters. For more about the Notes/Domino 6 Early Deployment program, see the LDD Today article "Lessons learned from the Notes/Domino 6 Early Deployment program."
What roles have each of you played in the Notes/Domino 6 internal deployment effort?
Cynthia Mamacos
I was assigned as the Development Relations Manager (DRM) on the IBM internal account six months ago. As the DRM, it's my role to make sure that deployment blocking issues IBM experiences are addressed and that the deployment can continue to roll. I field questions on all aspects of the project, making sure development is aware of the customer concerns and that development gets what it needs from the customer deployment, such as new feature sets that customers plan to deploy, feedback on areas of the product like C&S, Smart Upgrade, network compression, administrator client, and so on.
James Grigsby
As we looked at lessons learned from the Release 5 deployment, one of my challenges was to determine how we could validate solution readiness before product release. So we developed the specific metrics of 10,000 IBM users running Notes 6; 50,000 IBM users running Notes 5; and 100 IBM servers running Domino 6. IBM internally is the largest user of our products, so we felt we should prove the solutions were ready via an internal deployment. This would help us with external customers and also enable IBM to determine which areas were beneficial and which needed improvement. Most importantly, it gave each organization time to address the challenges before the product shipped.
In Notes/Domino 5, did we have the same Early Deployment program that we have today, or was it a very different program back then?
James Grigsby
Much different. During IBM's Notes/Domino 5 early deployment, we focused on the way IBM used Notes/Domino 4 and just leveraged the same functionality on top of Notes/Domino 5. In Notes/Domino 6, we made a deliberate attempt to exploit some of the new Notes/Domino 6 features, such as new transaction logging, network compression—those sort of things.
Internally, what efforts have been made to "sell" Notes/Domino 6 to IBMers?
James Grigsby
Many, but the main one is leveraging the brain trust that exists outside of the core development team. By working with teams such as IBM Research & Development, the IBM Chief Information Officer (CIO) office, and IBM Global Services (IGS), there was a wealth of ideas and excitement as individuals realized that they were on the forefront, that their suggestions would find their way into the product, and that in short order, new code versions were available with their suggestions. This empowerment enabled individuals to move the solutions out further within the infrastructure, not merely because of the functional job, but also because it was "cool" getting someone else to use a solution that they helped to implement. So it really was collaboration across IBM and leveraging the Notes/Domino 6 solution for the collaboration.
Cynthia Mamacos
IBM has an early adopter Web site where IBMers can download the latest Notes 6 client code. The site is very informative; the team has done a great job getting Notes/Domino 6 information out to users. The site extensively covers areas like installation, what's new, hints and tips, user education, discussions, known problems, problem reporting, deployment plans, and developer's corner.
How did you motivate internal users to adopt Notes/Domino 6?
James Grigsby
We really don't. I mean we tried, but messaging is so critical these days, you can't pragmatically dictate a change. You can't tell people to move to a new version, and now their calendar is screwed up, or their Palms don't synch. You have to motivate everyone through a peer-web of trust—that's really how the client adoption has ramped up. Yes, we put pressure on people to see who is on Notes 6, but in the end, people trust what their peers tell them—not the organizational dictates.
Cynthia Mamacos
Wherever and whenever possible, you show them cool new features, show them the new C&S user interface, and so on. I've personally put in my email signature file a link to the IBM early deployment Web site where internal IBM users can download Notes/Domino 6. The signature advertising campaign has caught on. People have added the same to their email signatures and are helping us to spread the word. I also ask everyone, "Are you using Notes 6?"
You both recently presented at a seminar on how to sell Notes and Dominos 6 internally to users. Can you tell me a little bit about what you presented in terms of content, and how well it was received?
James Grigsby
I explained to people how Notes and Domino have really changed—the focus on quality, wrapping up some of the more difficult feature areas early and getting them tested in early deployment. The other information that we provided was what were the things that had resonated with production customers? For example, things that were easy sales to the business executives, such as network efficiencies that minimize the capital outlay they had to make for expanding their networks. Storage savings, reliability improvements such that even if a server were to go down, it was actually able to come back much quicker. Those were key selling points that seemed to resonate extremely well with the audience.
Cindy Mamacos
My portion of the presentation was to give the audience the proof positive that we're successful in deploying Notes and Domino 6 in early deployment.
James Grigsby
The audience was surprised when Cindy mentioned and actually pointed out that we were global in the deployment. It was not North American-centric. Also that we had Notes 6 clients out there, thousands of them, that were running against a Release 5 infrastructure. That was quite surprising to them.
How have customers outside of North America received the new client, given that they don't have any language packs yet?
James Grigsby
For the users outside of North America who have experimented with the English-language version of the client, they are very interested in network compression. They like the improvements in the UI—the look-and-feel. And they like the new C&S features, so they have high expectations. But what's helping them to make decisions and to move forward is the fact that they can move directly from their Notes 4.x infrastructures to the Domino 6 infrastructure and avoid an entire upgrade to the Release 5 infrastructure. They're very excited about this. Based on a limited experience and their experience with the production servers, they seem to understand that we "get it" this time.
So, are the majority of clients outside the US still on a 4.x system?
James Grigsby
It's probably close to 40 percent or 45 percent of clients still on the Release 4 infrastructure, which was a big surprise to us.
When you gave your presentation about selling internally, what advice did you give?
James Grigsby
We broke it down to three levels. One is the business executive needing to make a business decision. The other was an administrator, and the third was the end user. From the end user perspective, it was features that really made life better, selfishly so. For example, Notes 6 is better at email management. There is color tagging of incoming email messages. The resizeable dialog boxes, copying items in a view that will automatically go into a table—it's those sorts of features that are selfish in nature, but really resonate with an individual user. And something as simple as faster replication can really make a difference for a remote user. For example, if you're a traveler, and you're in a hotel limited to a dial-up, then you really appreciate that improvement.
From an administrator point of view, it's the ability to upgrade the client incrementally, so that you don't have to push all the bytes down the wire just to migrate from Notes 6 to a maintenance release. It's also the ability to create policies, the ability to delegate control to a specific administrator without opening up the entire infrastructure to every administrator, and features like Tivoli Analyzer for Domino server health monitor, so that you know what your server's doing before it gets itself into trouble.
And then from the business executive point of view, it is knowing that Lotus itself has changed, that we're focused on quality, that we know that most of the enterprise customers are global in scope. We really focus on total cost of ownership—reducing the cost. We try to present a value proposition along those lines.
Inside of IBM, can you give me a breakdown in terms of numbers of Notes 6 clients and Domino 6 servers?
Cindy Mamacos
Currently, we have more than 6,000 internal Notes 6 users and 96 Domino 6 servers right now. Additionally, over 12,000 Notes 5 users are running against a Domino 6 server. We used a phased approach, so as we roll out servers and go forward, we have another 25 or 30 servers ready to go to Domino 6 once we meet our milestones for that particular rollout.
Cynthia Mamacos
Do you know roughly the number of bugs reported at this time, how many of them have been fixed or resolved in some way?
Cindy Mamacos
IBM has reported close to a thousand general bugs. For deployment blocking bugs that were very important to IBM—they couldn't continue the deployment without a fix—there were a 178 client deployment blockers and 87 server deployment blockers that we've addressed.
You have a release criteria for determining your milestones. Can you share those with us?
Cindy Mamacos
The way that the project began was with seven clustered servers. They clustered a Domino 6 server with a Domino 5 server, then waited until those servers met 99.5 percent availability. That was phase one. Phase two allowed us to deploy another, I would say, 13 servers with higher user loads and different server role coverage, for example, executive servers, application servers, and so on. These were not clustered, and we waited for 14 days of server reliability of 99.5 percent. It was late in phase two that we also began using non-debug (production) builds. Phase three includes Domino 6 mail servers for the general population.
How have you involved the IBM system administrators and application developers in developing release criteria?
James Grigsby
There is a cross-organizational team that Cindy works with. In fact, there's a weekly call with some of the key hands-on administrators so that they can have direct feedback into development. There are business administrators who are responsible and liable for policies and for making sure that we are adhering to IBM requirements. There are team meetings that initially were happening every month, in which everybody was checking in and giving us status. So we had representation from security, from the CIO office staff, and from the IT administrators as well.
So Cindy, it's been your responsibility to gather their feedback and to provide that to the development team. Do you log bugs just like the rest of us?
Cindy Mamacos
I let them—the administrators—log the bug. And basically that's how we make a deployment blocker chart. We, of course, think all their bugs are important, but we have them prioritize their blockers, and we basically take action on those and try to move those to completion as soon as possible so that everybody can just continue to roll out their plan.
All Domino 6 servers have been moved into production. Is there still an on-going pilot program inside of IBM right now?
Cindy Mamacos
The IBM Early Deployment plan has always been in production. The team is consistently upgrading the servers to newer builds, and the team has posted the Notes 6 client builds on the Web site for anyone who wants to try it.
It's obvious that because we're internal, IBM is a natural choice in terms of the Early Deployment Program. But are there other reasons why IBM makes a good candidate for early deployment?
Cindy Mamacos
They make a good candidate because of their size. They make a good candidate because they use a lot of features and do many different tasks. They're your broad spectrum Notes client users. A lot of customers want to know if we truly use the software that we sell.
In our Early Deployment Program, how much have we involved business partners who have created third-party products to work with Domino? Has that been a component of the program?
James Grigsby
That's a space that we need to improve on. This program helped us to identify some key business partners via the customers' early deployment environment, which is where we've gotten the interoperability testing so far. For example, we found that there was something that was different about a Solaris API, and along with the customer, Lotus, and Sun, we fixed that problem in a key customer environment. We did the same thing across IBM with several of their products as well as identify some problems like synchronization with PDAs.
As far as deliberately going out to the alliances, we haven't, but that is something that is falling under my new organization. We will bring some of those key partners into the lab and introduce them to the new technology.
And Cindy, is that part of any of your plans going forward with the Early Deployment program?
Cindy Mamacos
I would say it would have to be going forward because it's part of the Domino environment that really is affecting us. At the very first, we need to involve everybody that's part of the customer environment.
Has IBM encountered any problems with third-party software so far?
James Grigsby
There was a key issue early on in which IBM moved to a new virtual private network (VPN), and because they had debug code, we were getting an error when they tried to replicate. As it turns out, there was a VPN problem that had been dormant all along, so we chased down the problem and fixed it before IBM redeployed this VPN solution. It has been good in that sense, but again we're finding those out in production environments.
Is there internal testing going on with other Lotus products, like QuickPlace and Sametime?
Cindy Mamacos
We distinguish between IBM early deployment programs and product usage within IBM. Many Lotus solutions are used within IBM, but our Product Introduction team focuses on early deployments.
And is one of the goals of the early deployment programs for other products to have those products working with Notes/Domino 6?
James Grigsby
That's a big thrust because by the nature of being in deployment we have to get production interoperability. We found some problems with AIX's new 5.1 release, and our early Domino 6 code, and we fixed that in our code. We provided some feedback to Tivoli on some inconsistencies that we saw. We were able to understand how our Domino 6 servers, once they were consolidated, interacted with IBM Enterprise Storage servers (a.k.a "Shark"). So by the very nature of being in a production environment within a huge company like IBM, we get interoperability whether we want it or not.
Are we testing Lotus products with other IBM software products like WebSphere and DB2 to see how well we're playing with them?
James Grigsby
That is an area that's been identified as a focus area within Product Introduction Engineering. We're increasing the lab resources allocated for interoperability. For the most part, we have had a number of key level tests, and we have had some production interaction, but we haven't had the deliberate lab interoperability focus, and that's the lesson learned that we're addressing moving forward.
What have you learned from the Notes/Domino 6 Early Deployment program?
James Grigsby
There are definitely some positives and some negatives. On the positive side, we learned that we need to do this for all our products, so we created a new organization called Lotus Product Introduction to accomplish that. Another positive is that we were able to get nondevelopment people engaged in the development process, so everyone walked away feeling like they had contributed to this product, and rightly so. Some of the negatives in development, in a lot of cases, were not understanding the business ramifications as we moved out beyond development to a production environment. So I think that that was a large lesson learned. But, you know, there are certainly some technical lessons learned as well.
Cindy Mamacos
I think connecting the development group with some of these early deployment participants, at least for IBM, was important. We asked the early deployment participants to get out there and deploy the product, but we didn't give them a lot of up-front "here's how you do it, and here's how you use it" information. We got that under control, and we started connecting groups like the administration group and so forth, inviting them to the meeting, giving a little one-hour session on what to expect, how to use it, and then they can go off and run with it and figure it out. So I think that's important to the development cycle. And then participants can provide better feedback, not the "how do you use it" type of feedback, but "here's what I think it should do" feedback. It's a win for both sides. People are excited to be early adopters. The program is very popular, and when people—adopters—say to their co-workers, "Hey, why aren't you using that," that's a good thing.
Knowledge sharing is key. Ongoing and special sessions dedicated to specific topics has really helped direct focus on key areas. That's a win for both sides because it shows them how to reduce TCO. For us, we get validation of the feature.
James Grigsby
This program was a way for the IBM user population to feel like it was on the technology cutting edge. So it wasn't as hard as one might think to get the end users to adopt the product, especially when they saw their feedback being used. While Cindy does say correctly that they had 1,000 bugs that had been reported, they had probably submitted something like 4,000 pieces of information back to us that were distilled down into those 1,000 bugs. So everyone felt that they had a direct link with development.
How much emphasis would you put on training for end-users in this particular release?
James Grigsby
The UI is not that much different from Notes 5, so users can gradually discover the new features. It's a big win that we really haven't had to give that much training to the typical end user. Now, there was training for the administrative assistants that Cindy drove, so maybe she can give us a couple of pointers on that.
Cindy Mamacos
What James is talking about is that we took the Lotus executives and their administrative assistants (AAs) and gave them a hands-on tour of the new features around calendar management. We showed them how easy it was to manage your executive's calendar as well as your back-up executive AA's calendar and so forth. We gave them a little special instruction because we really wanted them to use it and to provide us their feedback early, so we had to give them a little bit of an up-front start.
Is there any training required for application developers and administrators?
James Grigsby
We haven't had much deployment of new Domino 6 applications. What we have been successful at is existing Domino 5 applications being hosted on a Domino 6 server. Our experiences so far show existing Release 5 applications can be hosted on the Domino 6 server without any modifications. We haven't had to focus that much on the new application developers, but again I think it's very minimal training that will be involved.
From a global perspective, if about 45 percent of our users are still using 4.x servers, how are those applications working on a Domino 6 server?
James Grigsby
We don't see the applications being upgraded. At least at IBM, we didn't see them being upgraded, and to be honest, these applications don't know that they are on a Domino 4.x server or on a Domino 6 server. We learned that lesson going from Release 4 to Release 5, where we had the large UniqueNameKey (UNK) table issue. That was one of the things that manifested itself pretty quickly in the Early Deployment program, where we had a Domino 4 server, a Domino 5 server, and a Domino 6 server, and it wasn't the UNK table that caused a problem, but the directory, which had a lot more names in it. Because we found the problem early, we were able to change directory names such that it would be consistent and exploitable across each of those versions. For example, one of the early deployment participants upgraded its directory designs to the Domino 6 design, so they had Domino 4, Domino 5, and Domino 6 servers in the environment, and it was totally transparent to the infrastructure.
Is it also transparent to the user?
James Grigsby
The users don't know. Users really don't know which server they're running against. So we have about 12,000 Notes 5 users hitting Domino 6 servers.
Cindy Mamacos
That's right. A good majority of software group servers in the South are Domino 6, and that covers 12,000 people. We know we don't have 12,000 people on the Notes 6 client, so they're using Notes 5, hitting a Domino 6 server, and they don't know the difference.
Do you have any tips or recommendations for upgrading?
James Grigsby
Yes, do it now! Don't wait to upgrade the server infrastructure. I know that we state that the upgrade model is servers first, but there's immediate value to end users from installing the Notes 6 binaries, even if they stay with their Release 5 template. Features like the improved UI, the drag-and-drop, edit attachment in-place, the faster searching—all offer benefits. The end user can see immediate value by going to the Notes 6 client even while remaining on a Notes 5 mail template.
A common misconception that customers have when upgrading is that they won't realize the benefits of a new release until they've upgraded their entire infrastructure. But customers will realize an incremental benefit to upgrading to Notes/Domino 6. For instance, if you have a Domino 6 server with 1,000 Notes 5 users and 100 Notes 6 users, those Notes 6 users will benefit from features like native network compression—highly appreciated if they are connecting using limited dial-ups or networks—so there's no reason not to implement that feature just because your Notes 5 users can't use it. An entire Notes/Domino 6 infrastructure isn't required to begin benefiting from the Notes/Domino 6 features.
One of the things that went over well in the seminar session was the fact that administrators can use the new Tivoli server health monitoring. They can use it against a Domino 6 server and a Release 5 infrastructure, so they can exploit that immediately. With the Domino 6 servers, you can target them for different functional groups. You don't have to wait to update your entire Domino infrastructure to start getting the benefits. For example, if you've got a highly mobile population, you can upgrade their servers to Domino 6, get them onto the Notes 6 client, and then they will immediately benefit from the improved replication speed, and still interoperate with the Release 5 infrastructure. So there's no reason now to wait to upgrade everything before you start seeing immediate benefits.
James Grigsby
Which areas of the product do you think have seen the most significant improvement?
Cindy Mamacos
I would say C&S from an end-user perspective. I would say it's a lot more intuitive with easier flow through the forms.
James Grigsby
I would agree that C&S has been a major focus with improvements in the interaction between the principal and the administrative assistant and with functionality. And that is a direct result of the feedback from early deployment customers.
There are scalability improvements in this release. Can one of you address what some of those improvements are?
James Grigsby
As we see the customer base exploiting IMAP, we've made a huge improvement in the IMAP server, about a 400 percent improvement in our lab benchmarks over Release 5 IMAP. We also see improvements in scalability in terms of the number of users connecting to a server, which raises the question of how much more reliable the server is.
We see scalability improvements in terms of how we're better corporate citizens with our storage, in that we have a new reply without attachment, so Notes 6 users can more easily avoid flinging around large attachments all over the corporation. Also, our compression of the attachments, running the LZ1 compression algorithm, reduces the amount of storage that we take for file attachments. And the one that I think that I love the most is the new edit in place. We're a better corporate citizen because we don't need two different versions of that attachment—one in Domino and another on the file system—but also the fact that, because I can edit in place, I now send out the right version in my reply!
Reliability is another issue for many customers, and I know many improvements were made in this area. Can you describe some of them?
James Grigsby
It starts at the lowest level and builds up. For example, the on-disk structure (ODS) has improved to detect and to avoid more data corruption. Because database transaction logging has proven so beneficial to reliability, it's been extended to transaction logging of views, but users shouldn't log every view on every database because this will have an impact on CPU and I/O. Administrators should use view logging for critical views in critical databases, such as directories or key application databases. There are also faster server restart features that enable the server to restart much, much quicker as well as improved problem data capturing with NSD.
You mentioned view logging. Do you have recommendations about view logging, for instance, should you turn it on for the views in which you already have transaction logging?
James Grigsby
We have view logging, but you can't view log a database that's not transaction logged. View logging for two key directory views, $ServerAccess and $Users, is enabled already in the directory template. Enable view logging on highly used and large views to reduce restart time, which is affected by view rebuilds after server crashes, and enable it for any views that need to be immediately available after media recovery.
When do I want to turn on transaction logging, when do I want to turn on view logging, and which databases do I want to turn it on for? Do I want view logging on my mail server?
James Grigsby
I would say for sure on your mail servers that have a large directory. And I think that criteria is whether or not the cost of the view logging is cheaper than the cost of having to rebuild a view in the middle of a server restart. In IBM's case, the directory is 2.1 gigabytes now. The cost of a delayed server restart is intolerable for their service level. So the cost of view logging is minimal compared to that. But any database or any view that gets a lot of updates would be a candidate for view logging, such that it would minimize the number of updates and hence decrease the view update time as well as the latency of writing the update to the actual database. Any of your key, critical databases or applications, like an expense application, would be a candidate.
For IBM, view logging was so important to them that they turned it on inside of the directory template, so they would never have an incident where the views weren't being logged. That's a tip: you can turn view logging on in a template, so that any database created from that template will have view logging turned on.
Would you recommend turning it on for mail file databases?
James Grigsby
I wouldn't on every mail file until we've done more performance testing. If I had 10,000 databases on a server, and I logged one view out of each one of those databases, I think it would be expensive in terms of server resources. Technically, you can, but I would not recommend that just yet.
You mentioned TCO a little earlier, and that's a huge emphasis in this release. Which areas do you think provide the most TCO benefit?
James Grigsby
Watching the early deployment customers, you get a good sense of the features that they think are real big for them. In one customer's case, it was a network bandwidth savings because they were having to make a significant capital investment in a network infrastructure outside of North America. And that was very expensive and problematic because in some regions they couldn't get the bandwidth even if they had the capital. So network bandwidth savings is huge. Outside of North America, another one was language consolidation—having multiple languages on a single server so that you don't have to create multiple servers just because you have multiple languages.
That leads right into server consolidation in general. The need to reduce the number of moving parts so that you can reallocate manpower to other projects. That's been really huge, and hand-in-hand with that has been the concern that if I consolidate, I have to have a reliable infrastructure to consolidate to.
One feature that's been well received is the ability to run multiple versions of Domino across different partitions. If you have an eight-partition server, you don't have to upgrade all eight servers to the new Domino 6 maintenance release. You can actually target the Domino 6 maintenance release for the particular partition that you're after. That's been huge.
Disk storage savings is another TCO benefit. And again, these are hard savings. Gartner Group projected that it costs $2,000 per two gigabytes to manage on a yearly basis, so anything that we can do to save on storage consumption—like improvement with archiving, reply without attachment, better file attachment compression, and edit in place—resonate with the early deployment customers. If you save 10 MB just through these features, and multiply that by 5,000 users, you're talking about 50 GB of disk that no longer needs to be scanned, backed up, or managed. Users can get more "data per allocated MB," and IT can slow the growth of disk demands. So that would be a lesson learned that these were the features that were important enough for the business executives to make a decision to move out of pilots into actual deployment.
Which new features are you most excited about?
James Grigsby
As a remote user, native network compression and streaming replication are my personal favorites. Many customers are looking forward to the improvements in C&S and new features like reply without attachment.
Cindy Mamacos
My favorite feature is Copy Selected as Table. This project requires much organization, and that feature has saved me many, many hours of work.
What's next for each of you?
Cindy Mamacos
With all the experience that I gained on this project, we want to basically make sure that as we move forward to the future releases of Notes and Domino that we take the experiences that we learned from here and make each consecutive early deployment project better. So I will be continuing on the Notes and Domino track for now with the point releases, but not the maintenance releases.
James Grigsby
It's been a huge success, and we want to continue that. And in fact, I'll be vectoring off to head up Product Introduction Engineering. We are doing more work in the lab around performance tool development, so that we can load stress other Lotus offerings in a lab environment before we introduce them to early deployment customers. We will also be expanding our usage-centric performance analysis, as well as introducing new benchmarks to the vendor space, such that the vendors can generate benchmarks on our new Horizon Two products—QuickPlace, Sametime, Discovery Server, and so on—as well as a deliberate focus on interoperability. Interoperability within the Lotus portfolio, interoperability across the key IBM portfolio, and interoperability around the key third-party products that we normally find ourselves having to work with inside the customer's environment.
ABOUT CYNTHIA MAMACOS
Cynthia is a Development Relations Manager in the Product Introduction organization. Her primary responsibility is the early deployment of Notes/Domino 6 for the IBM/Lotus account. She joined Lotus in 1991 and has worked in many parts of the organization ranging from Desktop Support to Domino Product Management before joining the newly formed Development Relations team.
ABOUT JAMES GRIGSBY
James leads the Product Introduction Engineering (p) team covering Lotus solutions. The team's primary responsibility is the development of load tools, performance analysis, and interoperability to increase the early production deployment by customers. He founded the NotesBench Consortium, and before becoming a civilian, he had a long career in the U.S. Air Force working in data and communications systems. |
|