Does more memory really help? by Bret Swedeen

Level: Intermediate Works with: Domino 4.6 Updated: 01-Mar-99

[Editor's Note: The From the Field column is brought to you by Lotus Support, and features technical articles based on our experiences with helping customers to find solutions in the field. This article describes a performance analysis on the AIX platform. For performance information on the Windows NT platform, see "Optimizing server performance: Port encryption & Buffer Pool settings".]

Why is more memory a common cure for poor computer performance? Because it usually works. Of course, exactly how well this solution works is rarely ever known. For example, when was the last time you upgraded someone's desktop machine from 16MB to 32MB, and then told the user the exact percentage of performance of improvement to expect, and under what operations to expect it? Probably never. Desktop memory upgrades from 16MB to 32MB almost always produce noticeable performance improvements. Therefore, there's no need to explain what someone can easily see.

While the benefit of a desktop memory upgrade is easy to see, the same isn't always true for a server machine. People use servers differently than desktop machines. Therefore, the benefit of adding more memory to a server isn't always as apparent. Since the benefit isn't so obvious, the upgrade is approached with more thought and planning. In other words, administrators want to know what type of improvement they can expect from a memory upgrade before they do it.

Fortunately, there are people at Lotus attempting to address these concerns. These folks have taken a close look at the performance improvement of a Domino server when the memory is increased from 512 MB to 1GB. While the test results don't guarantee that all upgrades will experience the same performance boost, they do validate the claim that more memory does help. Exactly what this group tested, and their results, are the focus of this article From the Field.

Not your ordinary box
We decided not to perform the tests on the average heavy-duty Pentium machine running Windows NT. Instead, we opted to test a less popular hardware configuration with a different operating system focus for a simple reason -- we wanted to simulate a similar customer environment:

• Computer: IBM Thin3 Node on an SP frame powered by a single 120MHz PowerPC processor
• Memory: 512MB for the first set of tests and 1GB for the second set
• Network: TCP/IP over Ethernet
• Operating System: IBM AIX version 4.20

Using this hardware and operating system configuration, we installed Domino Release 4.53. While we could have adjusted various NOTES.INI variables to increase performance, we settled on a fairly "vanilla" environment and only adjusted the following settings (which remained constant across all tests and memory configurations):

• NOTES_SHARED_DPOOLSIZE = 250,000,000 (We altered this setting to simulate a similar customer environment.)
• SERVER_MAX_CONCURRENT_TRANS = 30 (We altered this setting to simulate a similar customer environment.)
• NSF_BUFFER_POOLSIZE = 160,000,000 (We used this setting as a memory control mechanism. Therefore, any throughput gains that occurred when we increased the memory were the result of changes in physical memory and not something else.)

Note: The NOTES.INI settings shown above are not recommendations for better performance. For more information about server performance tuning, see Optimizing server performance: Port encryption & Buffer Pool settings, and Optimizing server performance: CPU scalability. For additional performance-related information, check out the NotesBench Consortium and the Domino Performance Zone.

Simulating the load
Conducting Domino server performance tests depends largely on the ability to simulate a realistic user load. Since it's not easy to round up a couple hundred users, we simulated them using a variety of test scripts from a specially designed Domino database. The scripts in this database helped simulate three types of users:

• Casual: (75%): Light e-mail use. Sends mostly text-based messages periodically throughout the day.
• Active: (15%): Sends e-mail regularly throughout the day occasionally with attachments. Sometimes performs group scheduling.
• Intense: (10%): Heavy e-mail use. Lots of messages, normally with attachments, and lots of group scheduling for other users.

With this user mix, we divided the tests into user and memory amounts in the following manner:

• Test 1: 512MB for 350 users
• Test 2: 512MB for 500 users
• Test 3: 1GB for 350 users
• Test 4: 1GB for 500 users

During the simulation tests, we gathered statistical information with the following tools:

• Performance Toolbox for AIX (PTX): A comprehensive tool for monitoring and tuning system performance. This extremely flexible tool has the ability to record over 200 Domino and AIX level statistics. In short, this tool polls the server at a user-determined interval and collects specified statistics (refer to the IBM Web site for additional information about PTX).
• IPTrace: Built into the AIX operating system, IPTrace obtains detailed packet-by-packet descriptions of the LAN activity generated by a workload. Our "home-grown" scrubbing programs processed the information, extracting the IP traffic that contained meaningful Domino Remote Procedure Calls (RPC).

We then fed the variety of statistical information gathered by these tools into an IBM DB2 database for further analysis.

Reading the charts
OK, with the details of hardware, software, and testing methodology out of the way, it's time to get to the real meat -- the test results. The charts that follow graphically demonstrate the benefit of more memory with 350 and 500 user loads. Brief descriptions precede each group of charts; however, we believe the results shown in the charts speak for themselves.

Transaction testing
The first test measures the number of Domino RPCs between the client and server that occur each minute. Right out of the starting gates, we clearly see the positive performance impact of adding more memory.



Input and output testing
The next three tests measure the effect more memory has on disk input and output (I/O). The first test, "Disk I/O Operations Per Second," demonstrates a clear performance boost when the memory goes from 512MB to 1GB. The following two tests, "I/O in Bytes Per Second - 350 users" and "I/O in Bytes Per Second - 500 users," help confirm what the first test demonstrated -- more memory means fewer trips to the hard drive to retrieve information, which means better disk I/O performance. Also, notice in the last two charts, the greatest improvement appears with "total read." "Total written" doesn't realize the same gain. Why the difference? The answer is simple. A certain amount of disk data gets stored in memory. Therefore, read events that request data already stored in memory don't generate a trip to the hard disk. Disk writing events, however, always result in disk access.





CPU performance tests
In the previous tests, we saw dramatic improvement with disk I/O operations. Unfortunately, additional memory didn't have the same level of impact on CPU performance. The next two tests, "System Calls," which measures the number of times per second that a system service call is made, and "Run Queue Length," which counts the number of threads waiting for the CPU, demonstrate that even CPU-intensive operations experience some benefit from additional memory.




Mail database tests
The final tests measured elapsed time for three types of mail database operations:

• Open Collection: The operation of opening a database
• Read Entries: The event that occurs when a user scrolls up or down in the Notes client looking at the descriptions of mail entries in a view
• Find by Key: An event that is part of a name lookup on a sorted list of documents

These three charts illustrate specific user activities within a mail database. In the case of these three tests, additional memory improved event execution speed. Similar to the previous two tests, however, the overall impact was less dramatic then the I/O tests.





And the winner is...
More memory is better -- the charts prove it. It's never easy, however, to test Domino memory usage and declare the results universal. There are a number of variables that directly affect such tests:

• Server hardware configuration
• Server software configuration
• Operating system configuration
• Database size
• Database contention
• Various network variables such as transport protocol, bandwidth, and traffic

Despite the range of variables that affect the tests, the final results still manage to answer the simple, yet critical question -- Does more memory really help? Obviously, while some tests produced more noticeable results than others, the answer to the question remains -- more memory does help (your mileage may vary).

The question of more memory, however, is not an academic exercise. The answer to the question has "real world" implications. Administrators on the verge of similar user growth (350 to 500 users) must decide whether to increase the resources on the current server or buy a new server. Knowing whether more memory can help server performance directly impacts your hardware purchase decisions. In the case of these tests, the final result did impact a customer's decision -- they decided to install more memory and got more life out of their current hardware.