Domino 6 Directory performance improvements by Craig Garfinkel and Harry Murray

Level: Intermediate Works with: Notes/Domino 6 Updated: 03-Mar-2003

Considerable effort has gone into improving the performance and functionality of the Domino 6 Directory. This article discusses these improvements and how to best set up your directory architecture to maximize performance and reliability.

We start by presenting data indicating that Domino 6 can improve the maximum number of name lookups per minute by a factor of 10 compared to R5. Next, we show how the maximum number of lookups scales with the CPU capacity of the Domino 6 server. We then discuss how the performance of the new name lookup cache may be the single most important directory performance improvement. (You can now size the name lookup cache to fit all names in your organization. This means very high performance lookups are possible.) Finally, we explain why you may want to use a central directory server as the primary Domino Directory on a remote server with mail servers using userless configuration directories.

This article assumes that you're an experienced Domino administrator. If you haven't already done so, we encourage you to read the Domino 6 Administrator help topics relating to directories. The amount of help information is extensive and can be a considerable aid in deciding how to best set up your directory topology. You should be familiar with all options before you decide which is best for you. The optimal setup varies considerably with the organization.

Domino 6 Directory performance features
In R5, if you are updating the directory on the Domino server (for example, editing, adding, or deleting users), directory lookups may be suspended. In Domino 6, you can now perform directory lookups while directory updates and reindexing of the names view are in progress. Performance slows down primarily because updating clears the cache, but still may be satisfactory for most user activity. (We present this performance data later in this article, along with ways to ensure good performance.)

In Domino 6, there has been a major improvement in the name lookup cache for Domino Directory lookups. We discuss this cache in detail later in this article. We have also added a new LDAP directory cache for organizations that use this feature. This should greatly improve the efficiency of LDAP lookups.

Central directory for Domino 6 is a new feature. We found in our testing that clients can do lookups through a local mail server with a userless configuration directory to a central directory server on a different part of the network almost as quickly and efficiently as to a local mail server with a full directory. This is made possible by new code that establishes as many as 40 very efficient links between your mail server and the central directory server to pass directory information. More importantly, after the mail server caches directory information in the name lookup cache, it seldom needs to go to the directory server for data. We talk about the central directory more extensively later in this article.

Scalability tests for a local Domino Directory
We tested three differently sized servers to determine the maximum number of Domino 6 name lookups per minute for each. We deliberately avoided using a "souped up" top-end server and instead used servers equivalent to what may be found in typical Domino installations. The lookup load was generated by four client driver PCs that randomly selected names to look up from a directory containing over 500,000 names. We created this directory from our own IBM directory; it, therefore, contained all the real fields normally found in an organization's directory. The client lookups were done as fast and as often as the server allowed. (Note that our results represent potential number of lookups per second. Most current customer environments will not attempt this many lookups, so don't be dismayed if you find your site is not producing similar numbers.)

The servers tested were as follows:

• IBM Netfinity 5500 with four PIII Xeon 500 MHz CPUs, 2.5 GB memory
• IBM Netfinity 7000R with four PIII Xeon 700 MHz CPUs, 2 GB memory
• IBM Netfinity 8500 with eight PIII Xeon 550 MHz CPUs, 4 GB memory (used less than 2 GB)
• Client PC drivers were 800 MHz PIII’s with 512 MB memory

Our results show that the primary resource usage for lookups is CPU. We also found that the CPU did not max out when the number of lookups per minute max’ed out.

The following graph shows how well our servers performed:



In conducting these tests, we set the name lookup cache high enough to contain all of the names looked up. (More on this later in this article). The number of lookups per minute was the maximum for each platform. We believe the number of lookups will scale with platform CPU capability.

Domino 5.0.10 versus Domino 6
The performance improvements in the Domino 6 Directory allow it to support almost 10 times as many lookups per minute compared to R5.0.10. This is extremely good news for overall performance because lookups are used during authentication, authorization, and mail routing. The following chart illustrates this improvement:



Name lookup cache testing
As mentioned earlier, the new name lookup cache helps improve performance more than any other Domino 6 Directory enhancement. You can now set the name lookup cache size with the following Notes.ini setting:

NLCache_Size=n

where n equals the name lookup cache size in bytes. The default is 16000000 (16 MB). The maximum value is limited only by your need to leave enough memory for other Domino tasks.

To understand the effect of name lookup cache, we ran two tests with identical hardware. The server was an 8-way Netfinity 550 MHz. This server contained a directory with 500,000 names, but in each test, random lookups were restricted to 40,000 of these names. The only difference between the two tests was the size of the cache. In the first test, we used the default value of 16 MB. In the second, we increased the value to 50 MB. As the chart below indicates, increasing the cache size boosted by a factor of eight the number of lookups per minute. With the 50 MB cache, all 40,000 names were cached within several minutes, which made subsequent lookups extremely fast. However, in the test where the cache was 16 MB, all names did not fit into the cache.



To determine if you need to increase or decrease the size of the cache, you can compare the following Domino statistics used in our 16 MB test to your own environment. (You can display these statistics by entering show stat or show nlcache at the server console.)

• Database.NAMELookupCacheCacheSize = 16,447,205 (current size of cache)
• Database.NAMELookupCacheLookups = 1,879,903
• Database.NAMELookupCacheMaxSize = 16,777,216 (default maximum size)
• Database.NAMELookupCacheMisses = 1,362,746

You can see from the ratio of the number of cache lookups to cache misses (which indicates a high percentage of the latter) the 16 MB cache in our test setup was too small.

Note that iNotes Web Access uses a different cache for names. You can set the number of expected users in the Web Engine tab of the Server document, and it will automatically set the proper Web names cache size.

Central directory server architecture
Domino 6 introduces the concept of the central directory server architecture. This is an environment in which a few directory servers in a domain have a full replica (called the central directory) of the primary Domino Directory. The remaining servers in the domain have a configuration directory (also called a userless directory). This is a small, selective replica of the Domino Directory that contains only documents used for Domino configuration. To perform name lookups and other directory functions, servers with a configuration directory use the full Domino Directory on one of the central directory servers.

Central directory architecture provides the following:

• Tighter administrative control over directory management
  Only a few directory replicas contain user and group information.
• Less replication and view indexing overhead
  Configuration directories have immediate access to the changes critical business applications and processes require because they don't have to wait for the changes to replicate locally. Because updates to the directory trigger reindexing, you should consider doing updates to an administration server which can in theory be as small as a PC. You can then schedule a replication to each central directory server in turn at low usage times. As mentioned earlier, this is not strictly necessary because the updates do not stop lookups. But it’s a good idea because lookup performance is much higher if the server is not doing updates. It should be mentioned that when reindexing occurs, the name cache becomes invalid and is cleared. Also, disk space is saved on the local mail server with the small configuration directory.
• Failover and load balancing with a pair of servers and Domino clustering or directory assistance failover (We talk more about this later in this article.)
  You should have at least two servers for fault tolerance. If you create a Domino cluster between two central directory servers, be sure to consider replication control. You should ensure both servers don’t have reduced performance due to reindexing caused by too-frequent replication.

Central directory server testing
For our central directory tests, our main server was a Compaq Proliant 8500 with eight 550 MHz CPUs and 4 GB memory. The central directory contained 500,000 names (derived from the IBM combined directory). This central directory was located in a different part of the same building as the mail servers, but on a different subnet.

Two mail servers were configured with configuration directories and relied on the central directory for name lookups. We created the server load using two client drivers per mail server that performed name lookups. We used the following hardware for the two mail servers:

• IBM Netfinity 5500 with four PIII Xeon 500 MHz CPUs
• IBM Netfinity 7000R with four PIII Xeon 700 MHz CPUs

The two mail servers produced the results shown in the following two graphs. First, let's look at looups per minute:



Next, let's examine CPU uage:



During this test, the central directory server had a small CPU load of five percent for the first few minutes of the test, which eventually dropped to nearly zero resource usage as all the names were entered into the mail server's cache.

Testing the impact of updating the directory
We then ran a test using exactly the same configuration as described in the previous test, except:

• Only one userless mail server (using a configuration directory) was used.
• We constantly performed updates (adding users) to the central directory.

As we previously mentioned, the results indicated that the number of name lookups per minute was considerably reduced during updating. The primary reason for this is that the cache is cleared during updating. The test revealed that lookups during updating were reduced by a factor of six. Results are as follows:



Comparison of performance of central directory versus local directory
We used an IBM Netfinity 8500 with eight PIII Xeon 550 MHz CPUs as a Domino mail server with a configuration directory. We used directory assistance to access the central directory on the previously described Compaq Proliant 8500 8-way 550 MHz Domino server. Using a central directory, we achieved 61,640 lookups per second with 40 percent CPU usage. With a similar configuration using a local directory, our server performed 87,232 lookups per minute with 39 percent CPU.

This graph shows CPU usage:



And this graph compares lookups per minute:



Bear in mind the results shown in the preceding graphs are based on laboratory test simulations (as is all other data discussed in this article). When assessing the benefits of central directory versus distributed directory for your own production environment, you should examine your actual lookups per minute statistics. Also consider the advantages gained by saving disk space on each distributed server for the directory, as well as reduction in total CPU usage across your site (because indexes of large directories aren't maintained on each server).

Redirecting client lookups to the central directory server
You can redirect client lookups to a central directory by entering the name of the central directory server in the Directory Server field in the user's Location document. For clients, name lookups are now redirected to the server containing the primary Domino Directory (because that server is listed in the Location document). Mail servers also see a benefit because the client is going directly to the central directory server, so the overhead does not affect the mail server. (Note that the mail server and routers still perform lookups when sending and routing mail; only client activity is redirected.)

Whether or not to use this feature at your site depends upon how much load each server (mail and central directory) can handle. You can determine this after you deploy the central directory and get an idea of the load on your mail servers versus central directory servers. (Preliminary tests indicate a small CPU reduction is realized on the mail server when you redirect client lookups through the Location document.)

Fault tolerance for central directories
Domino 6 includes several features to help ensure fault tolerance for your central directory servers. For detailed information on these features, consult the Domino 6 Administrator help.

Directory assistance
Directory assistance enables a server to look up information in a directory other than the local primary Domino Directory, for example the central directory. This provides directory failover for all directories listed in directory assistance (primary and any secondaries) via directory assistance failover functionality.

Cluster failover
You can set up directory assistance to use cluster failover and workload balancing. To do this, open the Replicas tab of the Directory Assistance document for the directory and specify only one directory replica within the cluster.

If you use a central directory, cluster failover is particularly useful. For instance, you may want to configure cluster failover for a central Domino Directory to allow servers with configuration directories to use cluster failover to find an available replica of the central directory.

Combined central and local directory architecture
You can also combine directory architectures, using central directories and local distributed directories within a single domain. This "hybrid" architecture is suited for domains that include a main office with multiple servers on a fast network as well as remote offices with limited network bandwidth. Servers in the main office can use the central directory model, while the remote servers can each store a full Domino Directory locally.

Note that if any part of your network is congested, at least one server on each side of the congestion should have a full Domino Directory.

Other directory performance considerations
Two other directory performance issues to consider involve type-ahead addressing and Condensed Directory Catalogs. For further information about these features, see the Domino 6 Administrator help.

Type-ahead addressing
You can use a setting in a Configuration Settings document to disable type-ahead addressing on a server to reduce network traffic and to improve server performance.

Condensed Directory Catalogs
Condensed Directory Catalogs (also known as Mobile Directory Catalogs) can improve the performance of Notes clients. For example, you can press F9 to quickly verify an address from the local Condensed Directory Catalog. In this case, type-ahead name resolution is instantaneous because Notes searches the local directory catalog instead of the server.

Conclusion
To summarize, important points covered in this article include:

• Performance improvements in the Domino 6 name lookup cache have improved the maximum number of lookups per minute by a factor of 10.
• Performance scales with the CPU capability of the server.
• A remote central directory server used for name lookups by a local mail server with a userless configuration directory may be a good option for medium to large enterprises.

We hope this article has helped you understand some of the performance and functionality enhancements available in Domino 6. Please let us know what you think!


ABOUT THE AUTHORS
Craig Garfinkel is the Product Manager for the IBM Lotus Product Introduction, Integration & Interoperability team. This team is responsible for ensuring that Lotus products integrate in a consistent manner with IBM and third party software products.

Harry Murray has been on the Domino Performance team for over four years. Prior to that, he worked on the Compaq Computer (now HP) Application Systems Engineering Performance team.