The New Domino 6 NotesBench workloads: Heavier by request!

The new Domino 6 NotesBench workloads: Heavier by request!

by Razeyah Stephen

Level: All
Works with: Domino 6
Updated: 01-Oct-2002

With each major release of Notes and Domino, you've told us how your use has changed and therefore, how our workloads should change. With R4, you wanted a Notes client workload; and with R5, you indicated you needed workloads not only for Notes clients but for browsers, IMAP, and POP3 clients as well. As always, we do our best to listen and deliver to you! Since R5, you've been telling us that you use the R5 workloads when conducting your own performance analysis but that you would really like us to make the workloads more reflective of where your users' behavior is headed. Also, because you are deploying R5 iNotes Web Access (iWA) and like the Calendar and Scheduling (C&S) features, your administrators want to know how to better estimate and size their deployment configurations for iWA usage with C&S. You also asked for larger message sizes because your users' mail messages are growing in size. With all this in mind, we created heavier Domino 6 workloads, making our user model match your needs more closely.

This article describes our Domino 6 workloads. We explain how we defined our workload usage pattern. We then compare Domino 6 workloads to R5 workloads, identifying all significant differences. This article assumes you're an experienced Domino administrator.

What is a NotesBench workload?
A NotesBench workload is a defined script used to simulate a user activity through specific applications. The workloads cover a variety of protocols such as IMAP, NRPC (Notes Remote Procedure call, which is Domino/Notes native mail), and HTTP. The workloads allow us to have a common tool to apply a consistent, repeatable load against the Domino server so that we can assess the effects of various operating systems, hardware, and configuration changes.

See the "NotesBench for capacity planning" sidebar for additional information.

The "users" shown in the tables later in this article are only instances of the benchmark script running and do not necessarily correlate to the numbers of actual users deployed on a server. But our goal with the improvements in the Domino 6 workloads is to bring our user model closer to a typical production user. It's difficult to represent every customer usage, so we must emphasize that for the best capacity planning and deployment advice for your configuration and environment, you should contact your hardware vendors. (See the LDD Today Performance Perspectives columns "Choosing a platform for Domino 6: Hardware platform vendors" and "Choosing a platform for Domino 6: Storage vendors" for helpful references and information.)

Defining a workload usage pattern
When we start the development cycle of a new major release of Domino, we review the current version of the workloads. Through discussions with customers, review of production data, industry trends, and input from members of the NotesBench Consortium and various Domino performance groups, we define new workloads that simulate how we expect users usage patterns will behave approximately 24 months in the future.

What do the new workloads provide?
After all the research (and keeping your comments and feedback in mind), we've developed the new workloads for Domino 6. These workloads:

Bring the profiled user model closer to an enterprise user by increasing the variety of features and usage patterns
Allow us to leverage the new profile for performance analysis throughout our development cycle
Give more realistic information for deployment planning than our prior workloads
Provide more realistic estimates of system requirements such as CPU, disk, memory, and network
Enable our hardware partners to determine boundary conditions in typical configurations
Enable you to better determine strategic configuration decisions

Workloads comparison
The most commonly used R5 NotesBench workloads are the R5Mail, WebMail, R5iNotes (introduced for R5.0.8), and R5IMAP workloads. R5Mail simulates full Notes NRPC client users. Both the R5iNotes and WebMail workloads simulate Internet-based users who access their Domino mail via HTTP. The R5IMAP workload simulates Internet-based users who access their Domino mail via the Web using an IMAP client. For all of these R5 workloads, the message size is fixed at 10 KB and they have no attachments.

For Domino 6, most of our HTTP performance analysis was done with the R6iNotes workload instead of WebMail.

New features of the Domino 6 workloads include the following:

Workload increases	How much
Use heavier message sizes	10 times
Message sizes vary	From 0.5 KB to 10 MB
Attachments added	From 50 KB to 10 MB
Calendar and Scheduling	Added to the R6iNotes workload

Remember that the R5 and Domino 6 workloads can be run against both R5 and Domino 6 servers because the workloads are independent of the release of Domino. Keep in mind, however, that the features tested must be implemented in that version of the Domino server. For example, you can use both the R5iNotes and R6iNotes workloads to measure the performance of Domino R5.0.8 (and later versions of R5) or Domino 6.0.

As always, there is a lot of information in the details, so let's look a little deeper. The following table breaks down the Domino 6 message size and indicates how frequently that sized message is selected.

Body size	Attachment size	Total message size	Percentage of time used
0.5 KB	0	0.5 KB	10
10 KB	0	10 KB	30
50 KB	0	50 KB	40
50 KB	50 KB	100 KB	10
150 KB	0	150 KB	9.5
1 KB	10 MB	10 MB	0.5

The next table compares the R5iNotes and R6iNotes workloads. It shows that the R6iNotes workload is much heavier than the R5iNotes workload, with the average size of new messages (including attachments) at 100 KB. Most importantly, the R6iNotes workload includes the commonly used C&S feature.

Actions every 90 minutes	R5iNotes Workload	R6iNotes Workload
Open Inbox	6	6
Read Message	30	30
Delete Message	6	12
Add Message to Inbox	None	2 (100 KB average)
Send Message to 3 Recipients	1 (10 KB)	1 (100 KB average)
Send Invitation to 3 Recipients	None	1
Send RSVP	None	1
Close Inbox	6	6

The table below compares R6Mail and R5Mail workload. Again, the Domino 6 version is much heavier than R5. R6Mail has new messages of 100 KB average size with attachments. (Note that the R5Mail workload already implemented C&S.)

Actions every 90 minutes	R5Mail Workload	R6Mail Workload
Open Inbox	6	6
Read Message	30	30
Delete Message	12	12
Add Message to Inbox	2 (10 KB)	2 (50 KB)
Send Message to 3 Recipients	1 (10 KB)	1 (100 KB average)
Send Invitation to 3 Recipients	1	1
Send RSVP	1	1
Close Inbox	6	6

IMAP workloads comparison
In the following table, we compare the R6IMAP and R5IMAP workloads. As you've probably caught on by now, the R6IMAP workload is heavier than the R5IMAP workload. R6IMAP has new messages of 100 KB average size with attachments. The R5IMAP and R6IMAP workloads do not have C&S, since C&S is not supported by the IMAP protocol.

Actions every 90 Minutes	R5IMAP Workload	R6IMAP Workload
Open Inbox	6	6
Read Message	30	30
Delete Message	12	12
Add Message to Inbox	1 (10 KB)	2 (50 KB)
Send Message to 3 Recipients	1 (10 KB)	1 (100 KB average)
Close Inbox	6	6

How much change should I see with the new workloads?
Below are some results we obtained in the lab when we compared the new and old workloads on a R5.0.8 server, using a more typical customer system configured as follows:

eServer xSeries (Netfinity) 5500r M20 server
4 X 500 MHz Pentium III Xeon processors
2.5 GB memory
Windows 2000 Advance server
Domino R5.0.8 server

Server response to the R6iNotes workload
The following table summarizes server response to the R6iNotes workload. It shows the much heavier R6iNotes workload does much more work, highlighted by the transactions per minute increased of 32%. As you can see, the increase in message size and attachments contributes to the 90% increase in data transferred to the data drives. Thus the R6iNotes user model should help bring these benchmark results closer to reality, helping you plan your deployment configurations now and for future growth:

	R5iNotes Workload	R6iNotes Workload	Percent Change
Users	1,000	1,000
Transactions/Min	597	788	32 % increase
CPU %	31	40	29 % increase
KB/Sec (data drives)	486	925	90 % increase
Available Memory	1,931	2,003	4 % decrease

Server response to R6Mail workload
The next table indicates the heavier R6Mail workload does more work than the R5Mail. Transactions per minute increased by 8% and the CPU utilization increased by 33%. The increase in message size and attachments contributes to the 19% increase in data transferred to the data drives. Therefore the R6Mail user model should be more useful to your deployment and growth planning. (Since C&S was already implemented in the R5Mail workload, there is less workload impact on server resources compared to the iWA set above.)

	R5Mail Workload	R6Mail Workload	Percent Change
Users	6,000	6,000
Transactions/Min	8,034	8,672	8 % increase
CPU %	42	56	33 % increase
KB/Sec (data drives)	10,547	12,521	19 % increase
Available Memory	691	639	8 % decrease

Server response to R6IMAP workload
Our final table shows the heavier R6IMAP workload does more work than the R5IMAP. Transactions per minute increased 49% and the CPU utilization increased by 12%. The increase in message size and attachments contributes to the 24% increase in data transferred to the data drives.

	R5IMAP Workload	R6IMAP Workload	Percent Change
Users	3,900	3,900
Transactions/Min	3,195	4,745	49 % increase
CPU %	60	67	12 % increase
KB/Sec (data drives)	5,427	6,755	24 % increase
Available Memory	660	599	9 % decrease

Conclusion
You asked us to improve the workloads by having larger message sizes and attachments. You also requested C&S to iNotes Web Access workload. We listened to you, and the results show that our heavier Domino 6 workloads help our user model more closely match your expectations. Of course, these are still only benchmark tools. But we believe these workloads, along with other data you obtain from sources such as Server.Load and your hardware vendors, can help better equip you to more accurately plan your current and future upgrades, purchases, and deployment decisions.