LDD Today

Performance perspectives

QuickPlace 3 performance: Authentication and MTBF

by Jack Williams,
Sami Shalabi,
and Dick McCarrick
Level: Intermediate
Works with: QuickPlace
Updated: 02-Dec-2002

In an earlier Performance Perspectives column, "Comparing QuickPlace 3.0 with QuickPlace 2.0.8," we described a detailed study that compared QuickPlace 3 performance to 2.0.8. In that study, we employed tools that simulated typical user workloads at real-world customer sites, performing routine QuickPlace tasks, such as logging in, reading documents, editing documents, and so on. In the end, we determined QuickPlace 3 performance is indeed significantly faster than QuickPlace 2.0.8.

One important area missing from that study, however, was data on authenticated users. Most Places include users who require authentication. Because authentication is a multi-step process, it can require significant system resources to complete. To measure the impact of authentication on a typical QuickPlace site, we performed a follow-up test that included authenticated users. This article describes the results of this test. (For those who can't stand suspense, QuickPlace 3 performed significantly better with authenticated users than QuickPlace 2.0.8 both in terms of how many concurrent users it could handle while maintaining optimal performance and overall response time for equal numbers of users.)

We also studied how reliable QuickPlace 3 is by measuring server uptime and Mean Time Between Failures (MTBF). The second part of this column examines the results of this test. This should give you a good idea of how reliable QuickPlace 3 will perform in your own environment.

As always, in these tests we made every effort to ensure our results were as realistic and meaningful as possible. But of course, we can't guarantee your numbers will exactly match ours.

This article assumes that you're an experienced QuickPlace administrator.

Authenticated users
As in our previous QuickPlace performance test, we used LoadRunner 7.0.2 to create the workload of a typical population of users. (For more information on LoadRunner, see the LoadRunner product page.) Using LoadRunner, we developed load scripts simulating various numbers of users performing typical QuickPlace tasks. The LoadRunner drivers executing these scripts resided on two separate PCs. To help ensure the results were as realistic as possible, we ran the scripts against a test set of 100 QuickPlaces. These QuickPlaces employed the full range of QuickPlace functionality.

Test setup
To compare QuickPlace performance, we placed this set of QuickPlaces onto two servers. One ran QuickPlace 2.0.8 on top of Domino 5.0.8. The other ran QuickPlace 3 on top of Domino 5.0.10. Both servers were configured as recommended by the QuickPlace documentation. Our test included a separate directory server, running a Domino LDAP directory. This was a large (2 GB) directory, modeled after one used by an actual customer. All computers involved in this test were Netfinity 7000 M10 machines running Windows NT and included the following specifications:

Processor typePIII
Number of processors4
Processor speed550 Mhz
Memory2.3 GB
Disk (onboard)4/18.2 GB

The following table displays the user actions our test population performed on our QuickPlaces:

Task
Percent
Login
80
Anonymous access
20
Read attachment
50
Read a simple page in TOC
90
Read an HTML page in the TOC
90
Read a simple page in the start of a folder
80
Read an HTML page in the end of a folder
60
Read last simple page in a folder
5
Read last HTML page in a folder
5
Open the calendar
60
Read a calendar event
20
Open the task list
5
Read a task
5
Open a page in a room
6
Open a page in an inner room
6
Create a new page
20

In the table, the first column shows the action performed, and the second lists the percentage of users who performed the action. One way to visualize what this second column represents is to consider a group of 100 of our mythical users. Of this group, 80 logged in to the QuickPlace, while the other 20 were anonymous users. Half (50) read an attachment, 90 read a simple page in the QuickPlace Table of Contents, and so on.

Our simulation produced a gradually growing population of users with four users added every 30 seconds until we reached 500 users. The 80 percent of users logged in required authentication, a key point in our test. Our goal in this simulation was to create an environment that matched a real-world QuickPlace site as closely as possible.

Results
The following chart shows QuickPlace 3 performance with authenticated users versus QuickPlace 2.0.8:

QuickPlace 3.0 vs 2.0.8 performance -- authenticated users

As you can see, the QuickPlace 3 average response time remained at or below five seconds for up to 375 concurrent users. Compare this with QuickPlace 2.0.8, which could support approximately 275 concurrent users with this level of response. The QuickPlace 3 workload is about 36 percent larger. For bigger simulated communities (up to 500 concurrent users), QuickPlace 3 average response exceeded five seconds. However, QuickPlace 3 response time was still about twice as fast as that of QuickPlace 2.0.8.

We feel this test clearly indicates that when compared to its predecessor, QuickPlace 3 supports larger numbers of concurrent users at optimal system performance and better overall response. Of course, all this improved performance is of little use if QuickPlace 3 servers are unreliable and crash frequently. Fortunately, we have good news to offer in this department as well, as discussed in the next section.

QuickPlace 3 reliability
Now let's turn our attention to QuickPlace 3 reliability, in particular Mean Time Between Failures (MTBF). When designing QuickPlace 3, our reliability goals included:
To determine whether QuickPlace 3 met these criteria, we conducted a study using test configurations and workloads similar to those described in the previous section. We also tested a production QuickPlace environment—and in the spirit of "there's no Place like home," the one we chose was used by our own QuickPlace development team.

Simulated test environment results
In the simulated test, we used similar hardware and software configurations and user actions used in the QuickPlace 3 performance test discussed earlier in this article. We created a population of approximately 250 users visiting 100 Places. To expedite the test, we assumed that a full workday could be encapsulated in ten hours.

Our reliability numbers literally could not have been better—there were no server failures reported during the ten days (100 hours) of testing! During this period, our QuickPlace server successfully performed 1,660,380 separate user transactions, maintaining an average response time that remained below five seconds (except for a few short spikes that occurred while the Java garbage collector ran). Needless to say, we were very encouraged that our test reported 100 percent server uptime over ten days, well beyond our reliability goals for QuickPlace 3—impressive, if we say so ourselves.

QuickPlace development team environment results
Lotus has always been a big believer in "eating our own dog food." Phrased in less gastronomically disturbing terms, this means we use our own products in production environments. So it should come as no surprise that the QuickPlace team uses a number of production Places for functional specifications, release information, developer notes, discussions, and so on.

In our QuickPlace environment:
To examine QuickPlace 3 reliability, we focused on two servers in particular. One is a heavily-used production server containing a number of active Places. The other contains nonproduction Places used for testing.

We observed both servers for several weeks and obtained reliability numbers nearly as good as in our simulated test. Both servers averaged 99.9 percewnt uptime with MTBF approaching three weeks. Again, these numbers are comfortably ahead of our design goals for QuickPlace 3.

QuickPlace 3: Fast and reliable
It's always great to have good news to report. And it should indeed be good news to all QuickPlace administrators that QuickPlace 3 offers significantly better performance than 2.0.8. As our latest tests show, this improvement extends to authenticated users. QuickPlace 3 can handle more of them while delivering better performance. Equally important, QuickPlace 3 is reliable, exceeding design goals in terms of server uptime and MTBF. These facts should make the decision of whether or not to deploy QuickPlace 3 at your site considerably easier.


ABOUT JACK WILLIAMS
Jack Williams has been an Advisory Software Engineer with Product Introduction Engineering for approximately three years, working on projects involving QuickPlace testing, IBM hardware testing in the AIX environment, and various other responsibilities. Previously, Jack spent a year and a half with Lotus Consulting on numerous on-site customer engagements. He spent earlier parts of his career developing database applications and designing wide area networks. Jack is an avid runner, having successfully completed his first marathon this past year, and is an emerging triathlete.

ABOUT SAMI SHALABI
Sami Shalabi, a Senior Software Engineer, has been a developer with Lotus QuickPlace since its pre-release days. He is the developer audience lead, responsible for architecture, APIs, and many other things. At Lotus, Sami had worked on the Domino Web Server and InterNotes WebPublisher. Prior to IBM, Sami worked in computer graphics, specializing in high-performance urban visualization and weathering. Sami holds both Bachelors and Masters degrees in Electrical Engineering and Computer Science from MIT. When not working, Sami enjoys driving, fine food, photography, and film.