Configuring Domino 5 for AIX/pSeries in large physical memory environments

Configuring Domino 5 for AIX/pSeries in large physical memory environments

by
Scott
Hopper

Level: Advanced
Works with: Notes/Domino 5
Updated: 02-Jan-2003

What is a large memory environment? Those of us who recall the early days of the computer industry remember when 16 kilobytes of RAM was considered more space than anyone could possibly need. Nowadays, system administrators talk of gigabytes without batting an eye. So obviously the concept of large memory is constantly changing. Nevertheless, it's an important consideration for Domino installations because large memory environments share several important issues and concerns about how to best utilize this memory (and possibly reduce the total number of servers needed to support their user communities). Even over the lifetime of development and release of Lotus Domino R5, there has been a significant growth in the capabilities of operating systems to support large memories and a reduction in the cost of memory.

In this article, we define a large memory system as one which has over 3 GB of RAM. Memory sizes of this magnitude have significant impact on how default Domino installations use both shared and private resources. This is of particular interest when running multiple Domino partitions.

This article describes a study we conducted to determine how to best manage large memory systems. We discuss our findings and offer suggestions that you can use to help manage memory at your own Domino site. We used Domino 5.0.9, 5.0.10, and 5.0.11, running on IBM RS6000 or pSeries eServer systems with 4 GB RAM or greater. But we believe our results will help anyone running Domino 5.0.5 through 5.0.11 on AIX and may even be of some value to environments using other operating systems. Of course, we must caution that our numbers may not exactly match those you derive at your own site. Different Domino server configurations may have very different optimal configurations. But we feel confident most sites may benefit from this article.

We assume you're an experienced Domino administrator.

Controlling memory
So why are we concerned about large memory and tuning on AIX? Let's begin with a brief discussion of how Domino runs on AIX. (Much of this applies to other UNIX environments as well.) A Domino server is a group of related processes—let's call this group a "swarm." Each process in the swarm—server, HTTP, IMAP, router, and so on—has a 32-bit address space, or 4 GB of virtual addressing capability. In any given process at any given moment, some of this virtual address space corresponds to:

Actual physical memory (RAM)
Storage on disk (paging space)
No real memory at all

Further complicating the issue is that some of the RAM or physical disk memory may be shared with other processes. For instance, the read-only text portion (mainly the code) of libnotes_r.a is shared by all the Domino processes, so ten different virtual addresses in ten different processes in the swarm may point to a single physical block of memory containing the function OSInit().

The Domino swarm's footprint in real memory is composed of three parts:

Process private memory such as thread stacks and the "heap" managed by malloc and free memory
Shared memory
"Hidden" memory that is being used by the OS on behalf of the processes in the swarm—primarily I/O buffers

Some, but not all, of the kernel's use of memory on behalf of each process is mapped to the virtual address space of the process.

On current AIX systems (and other platforms, for example HP-UX), there is an additional complication. A process's virtual address space is divided into segments. Each segment is limited to a particular use or set of uses. For 32-bit processes on AIX, there are 16 segments of (at most) 256 MB of virtual address each. Each corresponds to the highest-order hex digit of a 32-bit address. For example, segment seven (of the 16 segment numbers 0x0..0xf) consists of addresses 0x70000000 through 0x7fffffff.

A number of segments is pre-committed to particular uses. The maximum number of segments "left over" for shared memory or mmap'ed files varies with the version of AIX, but is typically either ten or eleven.

However, for some processes, forcing all the stacks and the heap into 256 MB in segment two can cause problems. AIX provides a way to split the heap and the stacks into separate segments and to let the heap grow larger than one segment, at the expense of the segments available for shared memory. (Stacks are limited to a single segment in all cases.) The size and layout of the private (heap) data area is determined by AIX. Modifying the size and layout of the private area is commonly done by changing a value named o_maxdata stored in the header of the XCOFF-format executable file. We'll use the shorthand MaxData=0,1,...,8 to refer to o_maxdata values of 0x00000000, 0x10000000,...,0x80000000. These nine values are the only legal ones in a 32-bit AIX executable. The value MaxData=0 is a special case that combines private data with various other memory uses (thread stacks, kernel stack, and so on) in a single segment.

When MaxData=0 (the default), a process will get the maximum number of potential shared segments, but can have some amount less than 256 MB of heap. (How much less depends on how many threads and how big their stacks are.) When MaxData=1, segment two retains the thread stacks, while segment three is given entirely to the heap. As a result, the heap can grow to a full 256 MB, though normally this is a fairly modest gain over the default. MaxData can be as large as eight, so you can reserve as many as eight segments (2048 MB) for private data (taking them away from other possible uses). Note that in AIX 5 (and to a lesser extent AIX 4.3.x) there are some extended segment facilities. These are not useful for the problems presented by Domino, and so will not be discussed here.

Beginning in 5.0.9, Domino R5 reserves two segments (512 MB, MaxData=2) for private heap because the ~200 MB provided by the standard configuration proved not to be sufficient for some processes. So after Domino 5.0.8, the maximum possible number of shared data segments is typically either eight or nine.

In the Domino R5 memory system, all shared segments must be mappable by all processes in the swarm. Domino can generally deal with "running out" of shared memory as long as all processes run out at the same number of segments. But if process A can only attach seven segments while all other processes can attach eight, when the eighth segment is created, process A will detect it and Panic, stopping all Domino processes. Domino 5.0.9 through 5.0.11 performs a check to make sure that every process has the same value of MaxData as the first process.

However, there may be situations when this check is not appropriate. In such cases, the check can be suppressed via the Notes.ini variable Disable_Shm_Check=1. (Domino 6 and 5.0.12 don't need this variable because they use a different approach to guaranteeing that all processes can map all segments.) For example, if an addin thread to a Domino process or to an API process uses a shared memory segment that is not a Domino shared segment, you can set Disable_Shm_Check=1. (This topic is beyond the scope of this article. You would normally receive instructions from the addin's vendor if modification is needed.)

For more information about memory allocation see the IBM site, "General programming concepts: writing and debugging programs."

Domino has "knobs" that affect how much shared memory it creates. In R5 on AIX, we use these knobs to limit the amount of memory to the number of segments to what the operating system allows. But the optimum settings may not be to use as many segments as possible, even on a large memory machine. For instance, reducing Domino's shared memory may let the operating system have more I/O buffers in physical memory or reduce paging activity. We will now explore these trade-offs.

Before we talk about the results of our study, let's briefly review the controls in Domino R5 (through 5.0.11) for memory management. Two Notes.ini settings play crucial roles:

PercentAvailSysResources
NSF_Buffer_Pool_Size_MB

PercentAvailSysResources
The name of this variable is somewhat misleading. It was first introduced to permit a quick adjustment for multiple partitions and to deal with problems on small memory and mixed-use machines where Domino was taking more than its share of resources. It assigns the amount of physical memory that Domino uses for calculating how to size certain memory allocations. The initial expectation was that you would use it as in the following example: If you are running Domino on a server with two partitions with different workloads and want partition one to use 75 percent of the resources and partition two to use 25 percent, set PercentAvailSysResources=75 in partition one's Notes.ini file and set PercentAvailSysResources=25 in partition two's Notes.ini. Allowable values are 2 to 100.

In large memory systems, PercentAvailSysResources can play two additional roles:

Preventing Domino from oversizing its memory usage and running into other problems
Domino R5 code doesn't "understand" segment limitations very well and can attempt to give itself more memory than it can actually handle. Because Domino R5 on AIX cannot allocate more than about 2 GB of shared data memory for the segment, you should never let Domino "think" the system has more than about 3 GB total memory.
Performance tuning
For example, if you have two equal Domino partitions on a 6 GB server, PercentAvailSysResources=50 is not set automatically and may not be correct. Domino's memory allocations are only a guess on its part, and it makes sense to treat this parameter as a tuning parameter to see what happens to the performance of the servers as you change it.

PercentAvailSysResources determines the maximum value for the NSF buffer pool, the cache of data from files through which almost all Domino's large-scale data flows (see the following section). This size also determines the values for other substantial memory uses. (You can also set these sizes directly with these three variables: NSF_Buffer_Pool_Size_MB, NameLookupCache, and NSF_DBCache_MaxEntries in Notes.ini.) PercentAvailSysResources can also affect other Domino variables; just what it does is dependent on which MR version of R5 is involved. For more information about these and all other Notes.ini variables, see the Domino Administrator's Help.

NSF_Buffer_Pool_Size_MB
This setting specifies the maximum size in megabytes of the NSF buffer pool, a section of memory dedicated to buffering I/O transfers between Domino and disk storage.directly. Doing this only affects one other variable (the NSF_DBCache_MaxEntries). The primary benefit of using NSF_Buffer_Pool_Size_MB instead of PercentAvailSysResources is granularity. For example, on a system with 32 GB of RAM, increasing or decreasing PercentAvailSysResources results in a change of about 150 MB in the buffer pool size. NSF_Buffer_Pool_Size_MB lets you set memory in much finer increments (although note that Domino rounds off this value to the nearest multiple of 8 MB). NSF_Buffer_Pool_Size overrides PercentAvailSysResources if both are set.

Test hardware and software
In our study, we derived memory utilization figures from our own production environment and verified these by running additional tests on a large M80 system. The following table lists the hardware and software we used:

	SP/2 Winterhawk	M80	S80	P690
CPUs	4 x 375 Mhz	16 x 500 Mhz	18 x 450 Mhz	10 x 1100 Mhz
Memory	4 GB	32 GB	32 GB	24 GB
Network	1- SP Switch	100 MB Ethernet	1 - SP Switch	1 GB Ethernet
Domino Data Disks	2 Advanced Serial RaidSSA	Shark storage array via fiber	Shark storage array via fiber	Shark storage array via fiber
AIX version	4.3.3	4.3.3	4.3.3	5.1
Domino version	5.0.9	5.0.9, 5.0.10, 5.0.11	5.0.8, 5.0.9, 5.0.10	5.0.9, 5.0.10
Partitions	1	1	2	2
Defined users	Various	2500-5000 test users	5000 per partition	4000/3500

In the following sections, the most important factors are the amount of RAM and the number of users.

Results
Having nearly 2 GB available for shared memory in each AIX Domino process may seem like a lot. However, when using the default NSF_Buffer_Pool_Size_MB setting, an environment supporting 5,000 real mail users can "max out" shared memory. Such was the case in our IBM production environment, where we observed shared memory allocation utilizing all ten available segments, consuming 2480 MB. At the same time, private (heap) memory was nearly 600 MB indicating a total Domino allocation near 3.1 GB without considering any hidden memory usage.

While it may seem logical that making Domino's caches as large as possible would increase hit rates and therefore improve performance, this is only narrowly true. For example, if Domino is taking away physical memory that the operating system might otherwise use internally to cache I/O, moving memory from Domino to the operating system may actually lead to performance gains. Additional caching also adds overhead (the bigger the cache the longer it takes to search), so at some point, devoting more memory to caching is at best minimally useful and perhaps even counterproductive. How effective additional caching can be, and what type of cache is best, is a very tricky subject to which there are few definitive answers for real world environments.

Domino 5.0.9 increased the default maximum private data size from approximately 200 MB to 512 MB by devoting two more segments to private data. This was done to reduce the risk of a single process running out of private memory on large systems running several thousand users. The downside of this change is less shared memory available for the Domino swarm as a whole. The maximum shared memory available with MaxData=2 is reduced by about 500 MB to 1984 MB. However, on large memory systems, this creates a crunch of resources on the shared memory side when the default NSF_Buffer_Pool_Size_MB settings are used. For example, on a 4 GB system with as few as 1,000 users, eight shared memory segments are used during the course of a normal workload. Add an API to the system that initiates its own shared memory segments that use Domino keys, and you will run out of shared memory segments using NSF_Buffer_Pool_Size_MB defaults.

The following chart illustrates the scarcity of shared resources that this code change created when using a default buffer pool of 1515 MB on a 4 GB server:

Shared memory available

The following shows the default buffer pool size for 1 GB, 2 GB, 3 GB and 4 GB of physical RAM. Notice that at 3 GB and 4 GB, the default maximum buffer pool size significantly exceeds 1 GB. If you use PercentAvailSysResources, you need to convince the system that it's really somewhere between 2 GB and 3 GB of RAM to accommodate a 1 GB buffer pool size or less.

Physical RAM vs default buffer pool size

Physical RAM vs default buffer pool size

The next table shows how much shared memory is utilized at these levels to sustain just the buffer pool. It also displays the number of shared segments used by the operating system to achieve a given buffer pool size. The data points are based on defaults that would be generated on 1 GB, 2 GB, 3 GB, and 4 GB systems:

RAM	1 GB	2 GB	3 GB	4 GB
NSF buffer pool size	350 MB	750 MB	1,200 MB	1,500 MB
Shared memory required	500 MB	750 MB	1,300 MB	1,500 MB
Segments required	2	3	5	6

Recommendations
Based on our tests, we recommend the following to anyone running Domino 5.0.9 through 5.0.11 on AIX with a large memory system:

Set PercentAvailSysResources to result in an NSF buffer pool of approximately 850 MB. (Total physical RAM determines how close to 850 MB the buffer pool will end up using this variable; the larger the RAM, the coarser the control.)
For systems above 32 GB, set NSF_Buffer_Pool_Size_MB=850. (Above 32 GB, PercentAvailSysResources lacks granularity.)

If available, LPAR technology should be used to divide the box into multiple OS partitions on systems above 32 GB.

After setting PercentAvailSysResources and NSF_Buffer_Pool_Size_MB, evaluate whether you need to raise or lower them. This will give the system four or more shared segments (over 1150 MB) for non-buffer pool activity.

The following table will help you find an optimized value for PercentAvailSysResources given a value of total RAM in GB. The table reflects numbers we observed on our production and test servers. Percentages are of total physical memory. These figures are based on one Domino partition per 4 GB RAM:

4 GB RAM		8 GB RAM		16 GB RAM		24 GB RAM		32 GB RAM
Percent	Buffer pool (MB)	Percent	Buffer pool (MB)	Percent	Buffer pool (MB)	Percent	Buffer pool (MB)	Percent	Buffer pool (MB)
32	471	16	471	8	471	5	440	4	471
40	594	20	594	10	594	7	625	5	594
48	717	24	717	12	717	8	717	6	717
Successful operation (for up to 5,000 users)
56	840	28	840	14	840	9	809	7	840
64	963	32	963	16	963	10	901	8	963
Caution
72	1,085	36	1,085	18	1,085	11	993	9	1,085
80	1,208	40	1,208	20	1,208	12	1,085	10	1,208
						13	1,178
Problems recorded
88	1,331	44	1,331	22	1,331	14	1,270	11	1,331
96	1,454	48	1,454	24	1,454			12	1,454

At the level in which PercentAvailSysResources equates to a 840 MB buffer pool, a 5,000 user deployment showed significant free shared memory. This equates to maximum NSF buffer pool allocation of 20 percent of RAM on 4 GB systems.

The benefits of setting the NSF_Buffer_Pool_Size_MB using PercentAvailSysResources to the green values (or potentially lower) are:

More efficient utilization of process memory resources
More efficient allocation of shared memory resources
Less total memory allocation
Greater scalability
Greater flexibility managing shared and process resources

This is a classic case where less is actually more. As an administrator, you're probably asking, "How do I know when I’ve taken too much away from the NSF buffer pool?" All other things being equal, look for the following signs:

< 90 percent Database.Database.BufferPool.PerCentReadsInBuffer on a consistent basis, occasional dips slightly below this value are acceptable
Response time increases significantly
Mail routing characteristics drastically change (backlogs, delivery times)
CPU usage increases significantly

In the case of our IBM production server (supporting 10,000 real IBM users between two partitions), we used memory tuning to get a configuration in which multi-segmented (MaxData > 0) binaries could be applied to the code. In this case, "de-tuning" (giving fewer resources) proved beneficial to their specific scenario/configuration. Our site had originally used the default of NSF_Buffer_Pool_Size_MB=1515 on a 4+ GB RAM system. Prior default allocation rules of giving the buffer pool approximately one third of physical RAM were too aggressive on the 32 GB configuration (as it would be on most individual systems with more than 4 GB of RAM).

For a while, we were faced with process heap memory growth issues until we reduced buffer pool consumption. Additionally, we were very close to running out of shared memory. We reduced the PercentAvailSysResources setting by one percentage point each week, until we observed reduced resource usage on the system. With a PercentAvailSysResources value of seven percent (equating to an 840 MB NSF buffer pool), we enabled multi-segmented binaries, solving both process heap memory and shared memory concerns.

Conclusion
Tuning R5 memory management on AIX can provide both high single-server performance and high stability at memory settings, and in particular NSF buffer cache settings, well below the maximum values. "Pushing the envelope" of the address space is not highly productive in performance and can reduce stability. The best way to improve R5 performance and system scalabilty is through its modularity—partitioning at the Domino or OS level—not by pushing the memory addressing of individual Domino processes to their limits. On AIX, this means setting Notes.ini variables to control shared memory allocation, which in turn lets Domino make the best use of total memory resources available to it.

We hope you found this article useful. We plan to conduct a similar study with Domino 6 servers and to publish our results.

ABOUT THE AUTHOR
Scott Hopper has been at Lotus nearly 15 years, many of which involved working with UNIX-based products. Currently he provides field and technical support to the IBM internal Notes deployment. In his free time, Scott enjoys gardening, skiing, and playing soccer, as well as spending time with his wife and daughter.