 |
 Command Performance: The Domino Performance Team
Interview by
Betsy
Kosheff
 
Level: All
Works with: Domino 5.0
Updated: 08/02/1999
Inside this article:
Related links:
Optimizing server performance: Port encryption & Buffer Pool settings
Domino Server.Planner
NotesBench Web site
Domino Performance Zone
Get the PDF:
 (138Kb)
"Now, we're expanding our efforts with R5 performance to include messaging, calendar and scheduling, and authentication. This models many users concurrently accessing dynamically generated and newly delivered content, such as messages and meetings."
--James Grigsby
"We're looking for improvements in both local delivery and message transfer for native Notes and SMTP traffic. The improvements are based on code improvements, database refinement in mail.box attributes, and the ability to have multiple mail.box files on a server, as well as multiple threads for local delivery of messages."
--Brian Richards
"We are seeing significant performance improvements across all platforms. The coding of the UNIX versions of R5 takes advantage of some of the specific UNIX features that will help performance. We are seeing that the UNIX versions are equal or better in performance than the NT versions."
--Harry Murray
"We're working on tools to monitor real-time performance analysis and trend analysis, which began appearing in Domino R5 and will incrementally improve with future releases."
--Carol Zimmet |  |
Greater message throughput, higher server availability, bigger databases and directories, faster agent execution, and yes, the ability to support more concurrent users than ever before. These are just some of the areas where the Performance Team is continuing to tune the Domino server.
Also, during August 1999, the Performance team was our featured guest in the Developer Spotlight. You can go there to see their archived discussion.
What are the criteria you're using to evaluate server performance for Domino R5 and beyond?
James Grigsby
We're primarily focused on the widespread desire among customers to consolidate the number of servers that they use so they can reduce the cost of their networks. We're also looking for ways to increase server availability to improve end user access -- this is especially important as companies' Web sites become more critical to their business and these sites need to be constantly available. And, we're looking at leveraging some of the newer systems' capabilities, like being able to exploit faster processors, take advantage of more processors and more memory if it's available, and being able to exploit bigger and faster systems like the AS/400, OS390, and high-end UNIX systems like Solaris (Sparc), RS/6000, and HP-UX. With Domino R5, we have a much larger database file size capability, which leverages the larger file systems that these platforms can handle. At the low level, we're focusing on semaphores, cache hit rates, I/O queues, CPU utilization, memory utilization, and network utilization at a high load. We're learning a lot and providing feedback to kernel developers and hardware vendors so they can enhance or optimize their configurations.
What are your performance goals for Domino?
James Grigsby
We have a lot of goals, but just to give you an idea of what we're focusing on, we're looking to improve by a factor of two to five times, the number of users supported by a single server. We also aim at improving by two to ten times, directory size, message throughput, and agent execution. One of the main ways we're accomplishing these goals is by better supporting the Symmetric MultiProcessing (SMP) architecture for better scalability. The performance work is concurrent and an iterative process with R5.x development. This means important cross-platform performance decisions are made early in the feature design process. The result is a more reliable server load for customers across the various platforms.
The Domino Server Performance team. Front row: Michael Robinson, Nirmala Venkatraman, Carol Zimmet, Razeyah Stephen, George Demetriou, Ray Sambrano. Back row: Cameron Hildebran, David Robinson, Richard Kanosky, James Grigsby, Harry Murray.
What are the most noticeable changes for Domino R5 administrators?
James Grigsby
Administrators are going to see very noticeable improvements in the number of concurrent users that they can support, message throughput, and more efficient use of server assets. Also, they'll be able to better exploit additional server resources, like more memory, CPUs, and I/O subsystems with newer server configurations.
How has the Web affected your approach to performance tuning and capacity planning?
George Demetriou
Typically, models of performance follow models of usage. With the Web industry, everything is happening concurrently. For instance, some users are retrieving data from the file system, or data can come from Domino databases, as well as back-end relational database management systems, Web-based simple messaging, or messaging systems with calendar and scheduling and workflow applications. This is giving us a chance to define new models for performance tuning and capacity planning because you have such different types of Web "users" based on whatever the user happens to be doing.
How has your approach changed?
James Grigsby
The number of Web-centric usage profiles has increased. Our first efforts included WebWalker, which involves the HTTP read-only capabilities of a single .NSF file, and WebBuyer, which provides HTTP read-write of a single .NSF. The initial published performance data shows a configuration that supports 200 concurrent users. You can see that report on the NotesBench Consortium Web site. Now, we're expanding our efforts with R5 performance to include messaging, calendar and scheduling, and authentication. This models many users concurrently accessing dynamically generated and newly delivered content, such as messages and meetings.
What are some of the major things you would recommend people do to improve server performance?
Carol Zimmet
To see a list of 10 things we recommend you look at to improve server performance, see the "Iris Today" sidebar, "The top 10 ways to improve your server performance."
In general, people should be making use of their platform performance tools in addition to their Notes-specific performance information. This helps them to better understand the information returned and relate their performance to the published benchmarks.
Also, there are certain key areas to pay attention to -- for example, we've been recently observing that disk I/O is the bottleneck for many production environments. Specific statistics to look at are Average Disk Queue Length, Average Disk Sec/Read or Write, and paging. We're publishing suggestions about these key areas on the Notes.net site, as well as incorporating suggestions into the documentation.
How do you accurately gauge performance comparisons with competitive products when most Web servers are serving static content?
James Grigsby
That's a challenge. We focus on dynamic content, such as integrated business logic, or newly arrived mail, which is constantly changing. This is how our customers are using Domino, so this is where we focus performance tuning. Other vendors will have to move there as well to attain the scalability and performance required for constantly changing data and business workflows.
What kinds of tools for capacity planning are included with Domino R5?
Carol Zimmet
You can download Domino Server.Planner from Notes.net. This is a Notes application that we released with Domino R4.6 and enhanced for R4.61. We're also revamping it for R4.63. It's designed to query production environment loads against certified vendor data across a range of systems. Hardware vendors could not produce data for each load point, so we're enhancing Server.Planner to do extrapolation between data points. This means if the published data includes 750 and 1200 user datapoints, you can also obtain a 950 user extrapolation without an additional lab run. You can read more about Server.Planner in the "Iris Today" article "Simulating your Load Environment with Domino.Server.Planner."
There is also Server.Load, which provides an easy-to-use GUI to generate a load against a Domino server. It allows you to easily monitor Notes statistics and metrics under a simulated load. Server.Load is available in the Lotus Performance Zone.
What kinds of benchmarking tools are you using?
Rich Kanosky
We use both internal tools as well as public tools, such as NotesBench and Server.Load, to isolate bottlenecks on the back-end or Domino server. The public tools allow real-world workloads (like R5Mail, R5IMAP, WebMail, SMTP/POP3, and so on) to be run against the Domino server. By using these workloads, we are capable of determining the capacity and response times for a simulated number of users running a particular workload on the Domino server.
In addition to the above tools, we are also using OS-specific benchmark utilities, such as AIX Performance ToolBox and Windows NT Performance Monitor, to provide additional insight and assist in pinpointing bottlenecks (like CPU utilization, disk I/O, memory statistics, and so on) on the server's back-end.
What about the Notes Router -- what kind of performance improvements can we expect?
Brian Richards
We're looking for improvements in both local delivery and message transfer for native Notes and SMTP traffic. The improvements are based on code improvements, database refinement in MAIL.BOX attributes, and the ability to have multiple MAIL.BOX files on a server, as well as multiple threads for local delivery of messages. We are able to deliver tens of thousands of messages daily with R4. We want to move this to hundreds of thousands of messages, without a backlog. This is one example of the convergence of performance measured in terms of the load, with availability measured by the reliable and timely delivery of messages.
What can you tell us about performance of the different Internet protocols, such as IMAP?
James Grigsby
We're working aggressively on these. In general, the user/server metric will double. The protocols are better able to leverage other Domino server capabilities, such as the new data store and server framework. Because the protocols are more tightly integrated and able to leverage other server infrastructure code, we are already seeing capacity and performance improvements.
What about POP3 performance vs. Exchange?
James Grigsby
While our latest benchmark results are in the 2000+ range for a departmental class server (without leveraging the flexibility of moving the MTA to a different server which enhances the number even more), we feel we're competitive with Exchange on NT now. We also have a customer deployment. The Bankinter bank, featured on the Lotus Performance Zone, uses a single POP3 domain on a single Domino server. POP3 allows users to connect, retrieve and send mail, and then disconnect. Thus, the number of users connected to the service at a single time is not high, but the bank can provide this service from one Domino server to over 55,000 users, including supporting names for each of them. There are over 30,000 messages on peak days. A significant advantage is that the maintenance of user accounts (that is, file changes) doesn't impact other users since each customer has their own mail file. This user/file granularity allows administrators to avoid the risk of a single Domino message file corruption impacting other users.
On Exchange, the message store file corruption can cause denial of messaging service to all server users. Domino message store flexibility allows administrators to overcome the most common hardware bottleneck, I/O, by distributing Domino databases across multiple I/O buses, controller channels, and disks to leverage the server hardware's capability. Exchange's single file architecture compounds I/O contention and increases the denial-of-service risk to all users when a message store corruption occurs. Even their clustering solution doesn't help with this type of storage corruption because it's not file access, but file content (message store data) that's the problem.
How are you measuring search performance -- what kinds of improvements can we expect?
James Grigsby
We're still working on this, but the bottom line is that you can expect improvements because of the new ODS (On Disk Structure), which is designed to make indexing and searching better.
What kind of performance should we expect as user loads increase?
Mike Robinson
Our goal is to increase performance two to five times, without using partitioning, which will allow us to add even more users. Partitioning of Domino instances, Domino tasks, and features within a single server and across multiple servers keeps Domino capacity from being limited by current operating system and hardware limits.
How does Domino R5 perform across various platforms?
Harry Murray
We are seeing significant performance improvements across all platforms. The coding of the UNIX versions of R5 takes advantage of some of the specific UNIX features that help performance. We are seeing that the UNIX versions are equal or better in performance than the NT versions. If you run Domino partitioned servers on UNIX, you should be able to support even higher loads, since 64-bit UNIX enterprise systems can address more memory (greater than 4GB) and leverage large file storage space. Domino is able to take full advantage of large, stable UNIX enterprise systems.
What about Sun platforms, in particular?
Nirmala Venkatraman
Sun machines have already proven their CPU scalability in supporting a high number of concurrent users with Domino server partitions. In R5, we are seeing Domino on Solaris scaling much better in terms of capacity, throughput, and response times, with a single server instance. This is possible due to the core Domino architectural changes and UNIX -specific performance improvements. With implementations of asynchronous I/O being worked on, we should see a reduction in system resource utilization by the Domino server, and hence a better overall scalability. Solaris 2.6 supports large file systems -- larger than 2GB -- which should help Domino R5 large database support, which is in excess of 4GB. We are already seeing good performance improvements on the IMAP server on Solaris going from R4.6x to R5. With more performance optimizations being done on the IMAP server, we should see a tremendous performance gain of the IMAP server across all platforms.
What about the AS/400 -- is that going to represent a real performance win for you?
Harry Murray
Yes, because of its support for 64-bit hardware, operating systems, and applications. There is a native version of Domino for the AS/400, which provides much faster messaging, database, and Web applications.
How do you stack up against Exchange 5.5 in general?
James Grigsby
We're really working on reliability at load. What this means is that as our load increases, databases aren't corrupted, and in the event of an error, the "penalty" is transparent to the administrators and users. For example, because the server can repair the database online while still under load, messages are delivered and available to the user. Load-balancing clusters, and the ability for online backups done from a replica, are other capabilities that make errors transparent. This means the "risk" of consolidating more users onto a single server is reduced.
Another advantage is that we're leveraging the standards and capabilities of the other platforms that Exchange doesn't support. Plus, we're expecting to be on par with Exchange on NT from a performance standpoint, but we'll be more reliable because with multiple mail databases, we don't have the single point of failure that their shared mail store has. Corruption in, or restoration of one user's account doesn't have an impact on other users. We already have active/active server cluster members, which improves Domino server availability. By not being limited to two servers, a cluster can be up constantly, even when one member is brought down for upgrades to Domino R5 or hardware.
We can also perform beyond limits imposed by NT because we support other enterprise platforms, such as Solaris, AIX, AS/400, HP-UX, and OS-390. These platforms measure availability in months-to-years of scheduled uptime.
What about monitoring tools -- anything new here?
Carol Zimmet
We're working on tools to monitor real-time performance analysis and trend analysis, which began appearing in Domino R5 and will incrementally improve with future releases. We're also working on methods to bring the platform-specific metrics closer to the Notes-specific metrics, so the information can be monitored and analyzed from one process. For example, you'll be able to display Domino statistics side-by-side with OS metrics like 600 users using a defined amount of memory, with a specified CPU and disk utilization.
What is the database size limit? How will this affect performance?
James Grigsby
Databases are now certified up to 64GB. Other IBM platforms may certify even higher per database size limits. While the individual certified file size is 64GB, development tests run with 50,000+ databases on a single server. The database size has a minimal impact on performance.
What kinds of view/update and view/rebuild enhancements can we expect?
James Grigsby
The view updates and rebuilds are now three times faster than for R4.6.
What kind of Iris support resources are there? Where can we go on the Web for information on Domino performance issues?
Carol Zimmet
Of course, you can use Notes.net, where there are several performance articles. We will also be answering questions again in the Iris Cafe during the month of August. If there are performance issues you want us to focus on, this is a great place to post requests.
In addition, there's also the NotesBench Consortium, which offers the Server.Planner datasets as well as published vendor reports, and a general exchange of information and questions between users. And there's a new section on the Lotus Web site called the Domino Performance Zone.
What about products from Lotus Business Partners -- are there specific companies we should look for?
Carol Zimmet
Yes, there are a number of companies that have introduced performance planning products, and we encourage people to look for them by going to the Business Partners section of the Lotus Web site.
What performance areas are you concentrating on for post-R5 -- what's your top hit list?
James Grigsby
I think that we'll see more 64-bit performance work, especially as the 64-bit Intel-based hardware, 64-bit compilers, and 64-bit add-ins become available. As always, new usage models will evolve like Knowledge Management, synchronous messaging, and cross-product workflow. That's really the fun part. We get to define and shape these new models that are more customer-friendly. Just look at the success of the NotesBench Consortium (with 10,000+ registered users in one year) and Ideas International (tracking our benchmarks with the industry standards). It says we're doing something right to help ease the administrator's decision load.
ABOUT GEORGE DEMETRIOU
George Demetriou started working at in 1997 from Eastman Software. He is a Performance Engineer who has spent his time working on Domino Web server performance measurement and evaluation.
ABOUT MARK DOWDY
Mark Dowdy is another new addition to the Performance team, but he joined Iris in 1997 from Lotus. He has been working with the Notes server since Release 3.1.2. After a stint on the Programmability QA team, Mark is now working on application performance and the Agent Manager.
ABOUT JAMES GRIGSBY
James is the project leader for the Domino Performance team. He came to Iris in 1997 from Lotus, where he worked in Product Management, covering areas such as competitive analysis, performance, and the Notes server. Previously, he developed IT outsourcing proposals with Computer Sciences Corp. and had a career as an Air Force Officer working with computer systems at bases worldwide.
ABOUT CAMERON HILDEBRAN
Cameron Hildebran has been a member of the Domino Performance team since 1998. He is involved primarily with performance testing the Webmail workload. Before coming to Iris, he worked as a performance engineer for Digital Equipment Corporation, focusing on technologies ranging from relational databases to network streaming protocols. Cameron believes computers are inherently good if treated with the proper respect.
ABOUT RICHARD KANOSKY
Richard Kanosky has been a Software Quality Engineer at Iris since 1997. He previously worked at Lotus on the Network Protocols Quality Engineering team, which assisted in the Notes R4.5 quality engineering effort on the XPC protocol. He also worked with George Sprott on X.PC/ TCP/IP Passthru performance, which was presented at Lotusphere '96. Since joining the Performance team, Richard has worked on Server.Planner, NotesBench, and Server.Load. In his free time, he enjoys long distance, open ocean swimming, and playing baseball with his 6-year-old daughter, Christina.
ABOUT HARRY MURRAY
Harry Murray joined the Domino Performance team in 1998. He is currently involved in the testing of Domino R5 on IBM AIX UNIX and NT systems. Prior to joining Iris, he worked for Digital Equipment Corp. in their performance group doing NotesBench testing of Domino on Digital servers. Before that, Harry was involved in the system management of many Digital production systems and was manager of System Technical Support in a number of Digital facilities.
ABOUT DAVID ROBINSON
David Robinson came to Iris in 1998, after studying physics at the Massachusetts Institute of Technology. He's focused on developing a cross-platform performance monitoring component for Domino, and enhancing NotesBench to use this capability.
ABOUT MICHAEL ROBINSON
Michael Robinson is currently a Senior Software Engineer in the Database Performance Group. While at Iris, he has written performance stress and characterization tools for Domino Web Server, Calendar and Scheduling, and the Notes Database Engine. He has added several of those stress workloads to the official Lotus Notes Benchmark tool - NotesBench. He is also the developer of Lotus Server.Load (a free GUI Notes load generation tool), which is available on the Domino R5.0.1 CD and on the Lotus Performance Zone Web site. Previously, Michael spent a year working in Lotus Product Management as the Lotus Notes UNIX Product Manager. Prior to that, he spent four and a half years in Hewlett Packard's Workstation group writing HP-UX bootstrap firmware, and he spent time working as a Digital Designer Engineer. Michael is currently pursuing an M.S. in Engineering, and has a B.S. from the University of Miami, FL in Computer Engineering.
ABOUT RAY SAMBRANO
Ray Sambrano is the Senior Product Manager for the Domino Performance Team. He joined Iris in Q1 1999 from Lotus. At Lotus, Ray was in charge of the System Consulting arm of the ISV Technical Consulting group within the Lotus Business Partner Program. In this role, he worked with Lotus Premier Business Partners building applications for Notes and Domino. Previously, he worked as a Product Manager for Lotus in the system management arena, where he orchestrated the release of Lotus NotesView.
ABOUT RAZEYAH STEPHEN
Razeyah Stephen is a Domino Performance Engineer, who has worked at Iris since October 1998. She came to Iris from Digital Equipment Corporation, now Compaq, where she worked for five years in their StorageWorks division. Razeyah's specialty is UNIX performance.
ABOUT NIRMALA VENKATRAMAN
Nirmala Venkatraman works for Iris as a contractor. She started in April 1998 and primarily works on UNIX performance. She previously worked at Sun Microsystems.
ABOUT CAROL ZIMMET
Carol Zimmet started working at Iris in 1994. She is responsible for evaluating performance and performance tool development, on the server team. Carol continues to search for the one-step solution to everyone's performance problems. She is also interested in a 'white box' approach towards improving the quality of the product. Carol enjoys bicycling with her kids in the trailer, and playing racquetball. She has a longing to return to stained glass! |