 |
Sametime 3 vs 1.5.5 performance comparison
by Craig Garfinkel
and Dick McCarrick
Level: Intermediate
Works with: Sametime 3
Updated: 11/04/2002
Related link:
More Performance Perspectives
|
|
Lotus recently shipped Sametime 3, the latest version of our market-leading instant messaging and Web conferencing software. Sametime 3 offers client enhancements, new tools for system administrators, and new server features. For detailed information on these and all other Sametime 3 functionality, see the LDD Today article, "A preview of Sametime 3.0."
But as we've stated in a previous Performance Perspective column, making a product better isn't the whole story. New releases should also offer better performance than previous ones—and Sametime is no exception. To help determine whether Sametime 3 meets this criterion, the Lotus Product Introduction Engineering team conducted a detailed study in which we built a test environment and workload modeled on IBM's own existing Sametime 1.5.5 community. We then ran tests within this lab environment with both Sametime 1.5.5 and 3 and compared the results between these two releases. We employed tools that simulated typical user workloads, performing routine Sametime tasks, such as logging in, changing active status, sending a message, and so on. We concluded that Sametime 3 performance is indeed significantly better than Sametime 1.5.5, handling up to 50% more work while providing faster response time for most operations. This should be very welcome news for Sametime administrators because an improvement of this magnitude may well translate into reduced total cost of ownership, as fewer servers are required to support larger numbers of users.
This article describes our test setup, lists the results we obtained, and examines what they mean. Because we conducted these tests in an environment based on IBM's actual Sametime community, we're confident that our results are realistic and meaningful. At the same time, bear in mind that no test, even one modeled on a real Sametime site, can fully duplicate every possible customer configuration.
One final point before we begin: You may naturally be wondering why we're comparing Sametime 3 to 1.5.5, rather than 2.5 (a more recent release than 1.5.5, and the immediate predecessor to 3). This is because IBM initially deployed a customized version of Sametime 1.5.5. Later, while Sametime 3 was under development, several IBM groups were involved in early testing and piloting. Rather than upgrade to Sametime 2.0 or 2.5, IBM decided to go directly to Sametime 3 and become one of its earliest and biggest adopters. Thus, when we conducted our study, the two internal environments available to compare were Sametime 3 and 1.5.5. We recognize this may not be the typical upgrade path for all customers. Nevertheless, we feel our results are a strong indication of the continued improvement in Sametime performance and capacity from release to release.
Test setup...
In this test, we used four Stress Tester 3.15.0.8 robot drivers to each simulate 17,000 users (for a total of 68,000 simulated users) accessing one Sametime Community server. Each Stress Tester robot ran on a Windows 2000 1.7 GHz server. The Sametime Community server ran on Netfinity 7000 M10 4x550 MHz machines running Windows 2000 and equipped with 2.3 GB of memory. User authentication was performed through a Secureway LDAP server (an RS6000 H50 4x332 MHz computer running AIX 4.3.3 with 3 GB memory).
The drivers executed a script that performed the following common Sametime actions:
- Log in a subset of the available users.
- Pick a random user who is logged in.
- Change the active status of the second user a number of times (do not disturb, out of office, active).
- Send several instant messages between random pairs of users.
- Log some users off.
- Send instant messages between random pairs of users.
- Log some users on again.
- Log off all users that were logged in.
This script allowed us to measure:
- How long log in authentication took, both when scattered over time, and when they happened within a relatively short period
- How quickly log in and status change information propagated to buddy lists
- The relationship between the amount of instant message traffic (which varies) and performance of log in, buddy lists, and sending messages
...and results
The following two graphs compare Sametime 3 and 1.5.5 in terms of response time and total transactions handled. When reviewing these results, it's important to note that issues with Sametime 1.5.5 caused three of the four Stress Tester robots to terminate prematurely. As a result, in the Sametime 1.5.5 test, only 44,000 of the 68,000 intended users logged in.
First, let's look at the total number of transactions executed during our test:
As you can see, Sametime 3 performed significantly more transactions than Sametime 1.5.5, in some cases handling several times more. This clearly shows Sametime 3's superior capacity. Now let's examine response time:
The above graph shows Sametime 3 response times for these typical user actions were equal to or better than Sametime 1.5.5 response. In fact, for logging in, Sametime 3 was nearly three times faster!
Sametime 3: Significantly improved performance
Our study indicates that Sametime 3 provides more stability with improved performance over Sametime 1.5.5. In our tests, Sametime 3 completed over 50% more work than Sametime 1.5.5, while maintaining better response time in most operations. This is an important point to consider when planning the hardware/software needs for your own Sametime environment. It's possible the increased capacity and better performance of Sametime 3 may help reduce your overall equipment and maintenance expenses.
For more information about Sametime 3, see its product page on LDD. You can also download the Sametime 3 documentation from the Documentation Library.
ABOUT THE AUTHOR
Craig Garfinkel works for the IBM Lotus Product Introduction Engineering team. He leads test teams that gauge system scalability and performance for Lotus Software products to ensure they meet design goals. Craig and his team are currently involved in the internal deployment of Sametime 3 across IBM. Craig spends most of his free time with his 7-month old son, Jaden. |
 |