 |
by Joe Bostian,
Rich Buck,
Harry Murray,
Andy Nolet,
and Jim Powers
Level: All
Works with: Notes/Domino 6
Updated: 12/02/2002
Inside this article:
Solaris performance
z/OS performance
Windows 2000 performance
AIX performance
Related links:
Domino 6 performance features
The new Domino 6 NotesBench workloads: Heavier by request!
Domino Administration online Help
The Documentation Library
NotesBench home page
Get the PDF:
 (499 KB)
|
 |
In the November issue of LDD Today, we highlighted the many new performance features in Notes/Domino 6. (See "Domino 6 performance features.") This month, we examine the results of performance tests we ran comparing Domino 6 with R5.0.10 and R5.0.11. As you'll see, our numbers clearly indicate Domino 6 offers significantly improved performance. This can help you to support more users on the same number of servers, to run more applications on these servers, and in some cases, to reduce the total number of servers in your environment. All these possibilities revolve around a single important theme—you can now do more with less.
When conducting our performance testing, we used several NotesBench workloads to simulate various user environments. These included the new Domino 6 workloads, which were constructed by analyzing actual Notes/Domino user work habits. (See "The new Domino 6 NotesBench workloads: Heavier by request!" in the October issue of LDD Today.) These new, extensive Domino 6 workloads are decidedly heavier on system resources than R5 workloads. To help illustrate this difference, this article includes several tables that compare system resource utilization data from both R5 and Domino 6 workloads at "peak load" on the same hardware configurations. Our goal is to demonstrate the performance improvements you can expect with Domino 6 and how you can use our test results for deployment planning.
We've organized this article by platform. Each section describes the results we obtained testing Domino on a particular operating system (Solaris, z/OS, Windows 2000, and AIX). Although we didn't configure our test systems as state-of-the-art (to avoid possibly skewing results), we did do some optimizing for each test, removing CPU and disk bottlenecks where possible. We also limited Domino tasks to only those necessary for the test being run. This allowed us to focus solely on Domino and/or platform restrictions and thus, to provide a good basis for comparison. But as with any test results, our numbers don't necessarily reflect those you may experience in a real-world environment.
One final word before we begin: By no means do we intend to compare the relative merits of the platforms we tested or to recommend one platform over another. The configurations we used for each platform were sufficiently different to render such efforts an "apples-and-oranges" comparison.
Solaris
The performance of Domino 6 on Solaris offers significant improvements in CPU utilization and scalability when compared to R5.0.10. In our tests, we obtained CPU reductions of up to 50% for our R6Mail and R6iNotes workloads and up to 25% for R6IMAP. In addition, the maximum number of users supported with sub-second server response time has grown from 5,000 to 6,000 users with our R6IMAP workload and from 2,500 to 3,750 users with our R6iNotes workload. All this translates into additional headroom on your servers when you migrate to Domino 6.
Test configuration
All Solaris tests ran on a Sun Enterprise 4500 server equipped with 12 CPUs (400 MHz Ultra Sparc II) and 12 GB memory. This system was attached to the network with a single 100 MB ethernet adapter running in full duplex mode. The Solaris 2.8 OS and Domino executable files were stored on three single 9 GB internal SCSI disks. User databases were evenly spread across five A1000 arrays, each on a separate SCSI adapter. The A1000s each contained 12 drives (9 GB each) configured as Raid 0. A single instance of Domino (no partitions) was used in all cases.
During each test, the server was loaded by a group of driver workstations, all running the NotesBench test tool to simulate typical user actions. Eight to 12 driver workstations were used in each test. The number of virtual users per machine varied between 250 and 1,000. (We reduced the number of users as response time approached one second.)
CPU utilization
The R6Mail test used the Notes NRPC protocol to simulate requests from a standard Notes client. Domino 6 produced a major reduction in CPU busy time (only 38% busy at the 9,000 user level compared to 74% on R5.0.10). These savings become more pronounced as more users are added to the system. Also, the total number of users we ran while maintaining sub-second response time grew from 9,000 in R5.0.10 to 10,000 with Domino 6:
The R6iNotes benchmark simulates iNotes Web Access users connecting to the Domino server using HTTP. Again, we found that Domino 6 has cut in half the amount of CPU resource needed to support a given number of users. Also (largely due to the reduction in CPU), we were able to support 50% more users on this configuration (3,750 vs 2,500) before response time exceeded one second. In fact, 3,750 users on the Domino 6 server used less CPU than 2,500 users did on R5.0.10!
The R6IMAP workload simulates Internet-connected users accessing mail via IMAP. As with the other Solaris tests, we obtained reductions in CPU usage and higher numbers of users supported for Domino 6 compared to R5.0.10:
Peak load comparisons
The following tables compare Domino 6 versus R5.0.10 results when running each workload to "peak load" (the maximum number of users where response time remains below one second). For further comparison, we also include results obtained using R5 workloads on Domino 6 and R5.0.10.
Note that the Domino 6 workloads use increased message and attachment sizes and for iNotes, include Calendar and Scheduling actions. As a result, Domino 6 workloads place higher demands on the Domino server. Thus, test results were often lower than those obtained running commensurate R5 workloads.
First, let's look at Domino 6 workloads:
| R6Mail | R6iNotes | R6IMAP |
| Domino 6 | | | |
|
| 10,000 | 3,750 | 6,000 |
Average response time (sec)
| 0.64 | 0.94 | 0.6 |
|
| 44.4% | 62.6% | 45.1% |
Total network
(bytes/sec)
| 1,739,252 | 1,761,175 | 1,508,233 |
| Domino 5.0.10 | | | |
|
| 9,000 | 2,500 | 5,000 |
Average response time (sec)
| 0.11 | 0.67 | 0.08 |
|
| 74.2% | 82.2% | 40.2% |
Now let's compare R5 workloads:
| R5Mail | R5iNotes | R5IMAP |
| Domino 6 | | | |
|
| 10,000 | 5,500 | 5,000 |
Average response time (sec)
| 0.12 | 0.78 | 0.05 |
|
| 38.4% | 61.6% | 27% |
Total network
(bytes/sec)
| 1,457,827 | 1,739,252 | 1,690,480 |
| Domino 5.0.10 | | | |
|
| 9,000 | 4,500 | 5,000 |
Average response time (sec)
| 0.75 | 0.83 | 0.07 |
|
| 61% | 81.4% | 41.3% |
For these test runs, we set HTTP threads to 250, but otherwise Domino ran default parameters. We found response times degraded very rapidly after the server became saturated. In some cases, adding a few hundred users caused average response time to go from a quarter-second to several seconds. In general, however, response times were well below a half second when running below peak load. The previous tables show the maximum users our configuration could support while still maintaining sub-second response time. That's why we include response times higher than you might typically experience if you lowered the user population by a few hundred or so.
Overall, our testing on Solaris shows that Domino 6 supports more users and consumes less CPU than R5.0.10.
z/OS
Domino 6 represents the culmination of a multi-year effort to exploit the eXtra Performance Linkage (XPLink) architecture of z/OS. This architecture shortens path length, substantially reduces CPU consumption, and complements the inherent performance and scale advantages of Domino 6. Our lab measurements indicate CPU reductions of 25% to 30% for traditional Notes mail workloads and 30% to 35% for iNotes Web Access users. This reduced CPU footprint of the Domino 6 server translates into improved stability at high user workload levels. This allows you to support more users per server. More importantly, the lighter CPU requirements of Domino 6 can produce substantially lower total cost of ownership.
Test configuration
All tests described in this section ran on one logical partition (LPAR) of a zSeries z900 model 2064-116. The z900 was a 16 CPU machine that ran our test system and five other systems simultaneously for other performance and test work. Six of the 16 CPUs were dedicated to our test LPAR; the remaining 12 were shared among the other LPARs. Our test LPAR was populated with 12 GB of central storage memory.
There were four Open Systems Architecture (OSA) network cards installed on the machine to support all six LPARs. The one used by our system provided LAN connectivity via Gigabit ethernet. All disks were allocated from an Enterprise Storage Server (Shark I) array with each disk configured as a 3390 model 3. There were separate Hierarchical File Systems (HFSs) allocated on single volumes (disks) for the Domino executable, Notes data, and transaction log directories, as well as separate file systems to hold the Domino Directory and log.nsf. User mail databases were distributed evenly over 20 HFSs, each allocated across four volumes, providing 9.2 GB of useable space per file system. The operating system installed was z/OS version 1 release 2.
CPU utilization
Here is how the R5.0.11 and Domino 6 servers compared when running the NotesBench R6Mail user workload:
During this test, Domino 5.0.11 reached a 9,500 user workload level for a short time, but failed during ramp-up to 10,000 R6Mail users. Domino 6 remained stable throughout the duration of the test. Our last useful CPU measurements for R5.0.11 occurred at 7,000 users.
Tests with the R6iNotes workload indicated similar savings:
For this workload, the R5.0.11 server required a projected 115% of the available CPU to support 4,000 users. As a result, it was not possible to achieve steady-state at this user load for the R6iNotes benchmark.
Here’s how R5.0.11 and Domino 6 compare running the R6IMAP workload:
Both R5.0.11 and Domino 6 scaled very linearly and predictably on z/OS for Notes and iNotes Web Access users. Neither server appeared susceptible to typical bottlenecks that limit server scalability. If the server and platform are properly tuned, both Domino releases make efficient use of all available resources.
Running the IMAP workload, R5.0.11 is limited by the thread-per-user process model that requires a physical thread for each user supported. This effectively limits the ultimate scalability of the server on z/OS to approximately 1,500 users. At higher levels, the heavyweight thread implementation of z/OS that serves the platform well in most instances starts to become a bottleneck, affecting end user response. With Domino 6, the thread pooling model of IMAP allows physical threads to be dynamically allocated to user requests, similar to how the main server task has handled user requests since R5. In this processing model, the robust thread support of z/OS becomes an asset again, and IMAP scalability is substantially enhanced.
Regardless of the workload type presented to the server, it’s clear Domino 6 provides a substantial reduction in CPU usage. This means, for instance, that by simply migrating to Domino 6, it’s possible to expand a 7,000-user Notes mail environment to 9,500 users (or a 3,000-user iNotes Web Access population to more than 4,200) without adding CPU resources.
Peak load comparisons
The following tables compare Domino 6 and R5.0.11 peak load numbers (maximum number of users with sub-second response time). As we noted in the previous section, Domino 6 workloads have been updated to provide significantly heavier usage. For more information, see the LDD Today article "The new Domino 6 NotesBench workloads: Heavier by request!" Domino 6 workload results are as follows:
| R6Mail | R6iNotes | R6IMAP |
| Domino 6 | | | |
|
| 10,000 | 4,000 | 4,000 |
Average response time (sec)
| 0.45 | 0.29 | 0.07 |
|
| 52.9% | 76.7% | 19% |
Total network
(bytes/sec)
| 5314 | 2261 | -- |
| Domino 5.0.11 | | | |
|
| 7,000 | 3,000 | 1,000 |
Average response time (sec)
| 0.18 | 0.53 | 0.08 |
|
| 51% | 85.2% | 9.3% |
Total network
(bytes/sec)
| 3870 | 3694 | 766 |
And here's the R5 workload numbers:
| R5Mail | R5iNotes | R5IMAP |
| Domino 6 | | | |
|
| 10,000 | 4,000 | -- |
Average response time (sec)
| 0.15 | 0.16 | -- |
|
| 40.4% | 55.2% | -- |
Total network
(bytes/sec)
| 4,669 | 2,091 | -- |
| Domino 5.0.11 | | | |
|
| 10,000 | 4,000 | -- |
Average response time (sec)
| 0.23 | 0.23 | -- |
|
| 62% | 75.6% | -- |
Total network
(bytes/sec)
| 4,740 | 3,237 | -- |
Note that for the iNotes workloads, 4,000 users is not a limitation for Domino 6. In separate tests, we have successfully scaled a Domino 6 server on this 6-CPU LPAR to 6,000 R5iNotes users with 0.28 second response time and 81.6% CPU utilization. With additional CPU resources, the server would have scaled beyond 6,000 users.
The test results discussed in this section are further indication that Domino 6 provides a large step forward in terms of performance and scalability. On z/OS, your CPU usage may drop as much as one-third simply by migrating to Domino 6. Our results have been further confirmed by early Domino 6 production deployment to more than 3,000 users within IBM.
Windows 2000
As with other platforms, Domino 6 on Windows 2000 offers significant improvements in CPU utilization and scalability. The IMAP and iNotes protocols also show enhanced memory utilization. In our tests, we found reduced CPU usage of:
- 4% for NRPC mail
- 47% for IMAP
- 37% for iNotes
In addition, the maximum number of R6IMAP users supported with sub-second server response time is 9,000 users compared with 3,500 for R5.0.10. For the R6IMAP workload, the numbers are 2,500 users for R5.0.10 and 3,500 users for Domino 6. This should produce additional capacity on your existing servers when you migrate to Domino 6.
Test configuration
We conducted our Windows 2000 tests on a server running Domino with one partition and no transaction logging. Client PCs, running the benchmark software NotesBench, simulated client users running against the Domino server. The server ran with default settings, except where noted in the next section.
All Windows 2000 tests used Netfinity 8500Rs with eight 550 MHz PIII Xeon processors and 4 GB of memory. Windows 2000 and Domino executable files shared a 9 GB drive that was part of a raid array in a Raid 1 configuration. User mail databases were distributed evenly across four Raid 0 configured raid arrays that were divided among three SCSI adapters.The data disk configuration was large enough not to be a bottleneck in any test. Network access was through a single 100 MB ethernet adapter running in full duplex mode. The operating system was Windows 2000 Advanced Server with SP2 applied. The pagefile resided on a Raid 0 array.
The systems we used for testing were not state-of-the-art. Instead, we tried to represent typical computing power available to most Domino customers. The limiting resource for the iNotes test was CPU, and for IMAP and NRPC tests, it was memory. (Note that the maximum amount of memory available to Domino running under Windows 2000 is 2 GBs.) To help alleviate the load on these resources, we did the following:
- For Windows 2000 iNotes tests, we reduced the value of the "Number of active threads" field in the Internet Protocols - HTTP tab of the Server document from 200 to 64. In addition, we set the Maximum Cached Users field in the Internet Protocols - Domino Web Engine tab to 5050, and the Cached User Expiration Interval field on this tab to a number longer than the test duration.
- For IMAP and NRPC tests, we set the NSF Buffer Pool to 400 MB by adding the variable NSF_Buffer_Pool_Size_MB=400 to Notes.ini. Also, for the NRPC tests, we added the Notes.ini variables Server_Pool_Tasks=100 and Server_max_concurrent_trans=100 to ease a task processing bottleneck at the high end of the simulated users. (We could set these parameters higher without causing problems because we had plenty of excess CPU bandwidth.)
CPU utilization
The R6Mail workload used the Notes NRPC protocol to simulate requests made by actual Notes users. As you can see from the chart below, Domino 6 uses CPU more efficiently than R5.0.10, while maintaining sub-second response time with up to 10,000 simulated users:
The R6iNotes benchmark simulates iNotes Web Access users connecting to the Domino server using HTTP. The following graph shows Domino 6 substantially reduces CPU usage and increases the number of supported users for iNotes compared to R5.0.10. Domino 6 supports 3,500 users, and Domino 5.0.10 supports 2,500 users at the same CPU usage on the same hardware:
The R6IMAP workload simulates Internet-connected users accessing mail via IMAP. The following graph shows that Domino 6 not only uses less CPU than R5.0.10, but it also supports substantially more users (9,000 compared to 3,500 in R5.0.10):
With the growing popularity of Internet protocols, we made sure that Domino 6 offers improved performance in this area. For example, R5.0.10 could handle 2,500 iNotes users with sub-second server response time. By comparison, Domino 6 can provide this same level of response to 3,500 users on the identical test configuration—a 40% improvement. For IMAP, R5.0.10 supported 3,500 users; Domino 6 supports 9,000 users—a 157% improvement! In our Windows 2000 tests, we also observed better memory use for our iNotes and IMAP workloads. Average memory usage reduction at high user levels was approximately 40% for both protocols.
Peak load comparisons
The following tables list results we obtained running our workloads on Domino 6 and R5.0.10 on Windows 2000. As we explained in previous sections, peak load is the maximum user population the server can support while still maintaining sub-second response time. Again, we used both the R5 set of workloads and the heavier Domino 6 workloads. We'll look at the Domino 6 numbers first:
| R6Mail | R6iNotes | R6IMAP |
| Domino 6 | | | |
|
| 10,000 | 3,500 | 9,000 |
Average response time (sec)
| 0.18 | 0.52 | 0.63 |
|
| 52% | 76% | 87% |
Total network
(bytes/sec)
| 1,980,202 | 781,671 | 1,826,268 |
| Domino 5.0.10 | | | |
|
| 10,000 | 2,500 | 3,500 |
Average response time (sec)
| 0.11 | 0.57 | 0.1 |
|
| 54% | 78% | 34% |
And now the R5 figures:
| R5Mail | R5iNotes | R5IMAP |
| Domino 6 | | | |
|
| 10,000 | 5300 | 9600 |
Average response time (sec)
| 0.13 | 0.52 | 0.16 |
|
| 42% | 90% | 66% |
Total network
(bytes/sec)
| 1,183,448 | 1,168,124 | 318,559 |
| Domino 5.0.10 | | | |
|
| 10,000 | 4000 | 4900 |
Average response time (sec)
| 0.07 | 0.35 | 0.1 |
|
| 46% | 77% | 50% |
As you can see from our data, Domino 6 on Windows 2000 offers significant performance advantages over R5.0.10:
- Lower CPU usage for all protocols
- Improved memory use for iNotes and IMAP
- Increased numbers of supported users for Internet protocols
Bottom line: Expect to support more users on your existing servers, if you have adequate disk capacity.
AIX
The fourth platform we tested was Domino on AIX. And once again, our results were highly positive:
- Running the R6Mail workload at the 8,000-user level, Domino 6 consumed 42% less CPU than R5.0.10.
- Running the R6iNotes workload, Domino 6 supported twice the number of users as R5.0.10 and required 47% less CPU usage across the entire run.
- Running the R6IMAP workload, Domino 6 scaled to 10,000 users (compared to approximately 1,000 in R5.0.10), while using the 92% less CPU at the 1,000 user level.
The following sections explain how we arrived at these results.
Test configuration
For our Domino on AIX tests, we used an S80 computer with twelve 400 MHz CPUs and 16 GB of RAM. The operating system was AIX version 5.1 with patches at the Domino 6 published levels. Hard drives were configured as follows: a single 9 GB internal drive for the operating system and 64 drives (9 GB each) set up to support two Notes data directories. These drives were part of a Raid 0 array with two SSA Controllers using 16 MB Fast_Write cache and connected to the LAN via a 100 MB ethernet full-duplex connection.
We configured Domino as a single partition. The only modifications we made to the default settings were to lower the NSF buffer pool size and to increase the DB cache, server pool tasks, and server maximum current transactions. (We changed these values for benchmarking purposes only—don't modify them yourself unless instructed to do so by Lotus Support.) No transaction logging was performed during these tests (although we do plan to discuss transaction logging in an upcoming LDD Today article).
During each test, the server was loaded by a group of driver workstations, all using the NotesBench test tool to simulate typical user actions. Ten of these driver machines were used in each test with the number of virtual users per machine varying between 250 and 1,000. (We reduced the number of users as response time approached one second.)
CPU utilization
The following graphs show results we obtained comparing Domino 6 performance on AIX to R5.0.10. As with the other platforms, the only difference between these tests was the version of Domino we used. The configurations were otherwise identical.
Continuing our winning streak, we found that on Domino 6 on AIX, running the R6Mail workload supported 10,000 simulated users and required approximately 42% less CPU than R5.0.10:
The next graph shows the differences between running the R6iNotes workload on Domino 6 and R5.0.10. R5.0.10 supported approximately 3,000 users; Domino 6 supported twice as many users while requiring approximately 47% less CPU across the entire run:
Finally, let's look at the R6IMAP workload. R5.0.10 supported a practical maximum of 1,000 simulated users. For comparison, let's look at the percentage of CPU use required for this number of users. At the 1,000-user level, R5.0.10 consumed approximately 36% CPU. We then ran the same test with Domino 6 and achieved perhaps the most impressive results of all—10,000 simulated users! And at the 1,000-user level, Domino 6 used 92% less CPU than R5.0.10.
In our tests, Domino 6 on the AIX platform consistently scaled to higher simulated user levels and required less CPU than R5.0.10. Improvements in scalability ranged from 20% for R6Mail to 90% with R6IMAP.
Peak load comparisons
The following tables list tests results obtained running workloads on Domino 6 and R5.0.10 on AIX 5.1 This first table shows Domino 6 numbers:
| R6Mail | R6iNotes | R6IMAP |
| Domino 6 | | | |
|
| 10,000 | 6,100 | 10,000 |
Average response time (sec)
| 0.12 | 0.80 | 0.11 |
|
| 29% | 90% | 51% |
| Domino 5.0.10 | | | |
|
| 9,000 | 3,200 | -- |
Average response time (sec)
| 0.10 | 0.46 | -- |
|
| 42% | 79% | -- |
And the second displays R5 figures:
| R5Mail | R5iNotes | R5IMAP |
| Domino 6 | | | |
|
| 10,000 | 9,000 | 10,000 |
Average response time (sec)
| 0.08 | 0.33 | 0.06 |
|
| 21% | 98% | 50% |
| Domino 5.0.10 | | | |
|
| 9,000 | 6,750 | -- |
Average response time (sec)
| 0.16 | 0.57 | -- |
|
| 44% | 95% | -- |
The numbers presented in the previous tables focus on peak user loads, along with the associated average response times and CPU utilization percentages. In addition to these results, our tests showed that Domino 6 on AIX 5.1 scaled to significantly higher numbers of active users and required less CPU to perform the same amount of work as R5.0.10, across all the workloads we used. For example, in the R6IMAP test, R5.0.10 supported approximately 1,000 users; Domino 6 supported 8,000 while using the same percentage of CPU. Running the R6iNotes workload, R5.0.10 supported about 3,000 NotesBench users, and Domino 6 supported 5,000 (again, using the same percentage of CPU). The R6Mail workload results showed an increase in the number of supported users from 8,000 on R5.0.10 to 10,000 on Domino 6, while requiring an average 58% less CPU across the entire run!
Domino 6 means better performance!
Today, everyone needs to do more with less, a fact we kept foremost in mind when designing Domino 6. From a performance perspective, Domino 6 lets you support more users and requires less CPU than previous releases. Migrating to Domino 6 should significantly reduce the demands on your servers—even if you don't change anything else in your environment. This should be welcome news to all Domino administrators, especially in organizations looking to consolidate and/or reduce their server configurations.
For more information on Domino 6 performance features, see the Domino Administration online Help, which you can download from the Documentation Library.
ABOUT RICH BUCK
Rich Buck has had a long career working on performance within IBM. Rich works on the full spectrum of operating systems supported by Domino with a current concentration on Solaris.
ABOUT JOE BOSTIAN
Joe Bostian has worked at IBM for 18 years and spent the last six focusing on Domino performance as a part of the zSeries porting team. Prior to working on Domino, Joe developed and tested UNIX interfaces for OS/390 and OS/2.
ABOUT HARRY MURRAY
Harry Murray has been on the Domino Performance team for over four years. Prior to that, he worked on the Compaq Computer (now HP) Application Systems Engineering Performance team.
ABOUT ANDY NOLET
Andy Nolet has been working with customers on Notes performance-related issues for over seven years. Before joining the Domino Performance team, Andy worked for Lotus Support.
ABOUT JIM POWERS
Jim Powers is a member of the Domino Performance team. Previously, Jim led the performance team for the Lotus Domino support organization. His experience with computer systems goes back over 30 years, performing various hardware and software roles throughout his career. |
|