 |
Optimizing Lotus Workplace Messaging: Early experience
by Michael Gazda,
Titus Kao,
and Joe Wissman
Level: Intermediate
Works with: Lotus Workplace Messaging
Updated: 07/01/2003
Related link:
More Performance Perspectives
|
|
IBM Lotus Workplace Messaging 1.0 provides low-cost messaging that allows organizations to extend enterprise email to users currently without a dedicated workspace or email access. Lotus Workplace Messaging delivers a J2EE-based Web client for email and personal contacts, built on IBM's WebSphere Application Server, WebSphere Portal, and DB2 technologies. In addition, Lotus Workplace Messaging uses LDAP to provide directory services, allowing you to use your site's existing directories. For more information, consult the Lotus Workplace Messaging product page.
When developing Lotus Workplace Messaging, one of our primary goals was high scalability. To help achieve this, the Lotus Product Introduction Engineering team worked closely with development to perform extensive code validation and stress analysis. Results were continually sent to developers to identify and remove bottleneck areas as soon as possible in the development cycle.
And since Lotus Workplace Messaging incorporates several different IBM technologies, we decided to call upon IBM's High Volume Web Site and DB2 performance teams to provide additional tuning expertise. The High Volume Web Site team brought experience and perspective on how to effectively scale WebSphere Application Server. The DB2 performance team contributed towards ensuring that Lotus Workplace Messaging's DB2 back-end performed extremely well. The combination of WebSphere and DB2 experts, along with our own Lotus email and functionality expertise, helped ensure that Lotus Workplace Messaging 1.0 met our high expectations.
Along the way, we discovered a number of useful tips and techniques to improve overall Lotus Workplace Messaging operation, which we review in this article. We start by looking at a typical setup used in our analysis, including our hardware and software configurations and the workload we used to emulate users. We then discuss what we learned to help you optimize Lotus Workplace Messaging at your own site, particularly with system queue management. We assume that you're an experienced system administrator, familiar with Lotus Workplace Messaging, WebSphere Application Server, and DB2.
A sample test configuration
To ensure we looked at Lotus Workplace Messaging from multiple customer perspectives, we ran it on several different hardware configurations and workloads. These workloads varied from stressful, high transaction rate per test user scenarios to more "balanced" scenarios where many test users drove the servers with fewer transactions per user. We also created workloads that focused exclusively on the HTTP interface, workloads that focused exclusively on SMTP mail routing, and workloads that combined the two.
For example, here is a sample of a balanced configuration, using a hardware setup and HTTP workload that you could use at your own site. In this example Lotus Workplace Messaging environment, we installed the following servers, running on a 100Base-T network:
- WebSphere Application Server 5.0 with IBM HTTP server 1.3.26, running on an IBM xSeries 360 server with Windows 2000 server SP3, four 1.5 GHz Xeon processors (hyper-threading disabled), 3.5 GB of RAM, and three internal 36 GB Ultra 160 SCSI disk drives
- DB2 Enterprise Edition 7.2 Server with fixpack 7, running on an IBM xSeries 330 server with Windows 2000 server SP3, two 1.26 GHz Xeon processors, 2.5 GB of RAM, two internal 36 GB Ultra160 SCSI disk drives, and SSA controller with two 69.5 GB arrays in a RAID 5 configuration
- IBM Directory Server 4.1 with Windows 2000 server SP3, two 1.26 GHz Xeon processors, 2.5 GB of RAM, and internal 36 GB Ultra160 SCSI disk drive
We simulated 3,000 registered users. Of these, 1,500 were concurrently active during the test. Each mail file began the test containing approximately 95 pieces of mail at an average size of 32 KB. This included 10 message body sizes ranging from 1 KB to 100 KB with 30 percent of the messages also containing a 50 KB text attachment.
During this test, all messages that active users sent were delivered locally. Recipients were generated randomly from the pool of 3,000 users registered on the server (this includes the 1,500 active users). The messages that were sent during the testing differed from the existing messages in the mail files. The following table shows the sizes and distributions:
Body size (KB) | Attachment size (KB) | Total message size (KB) | Percentage used |
0.5 | 0 | 0.5 | 10 |
10 | 0 | 10 | 30 |
50 | 0 | 50 | 40 |
50 | 50 | 100 | 10 |
150 | 0 | 150 | 9.5 |
1 | 10 | 10 | 0.5 |
To simulate user activity, we used a scripted workload that did the following: At the beginning of the test, the 1,500 active users gradually logged into the Lotus Workplace Messaging server over a 50-minute period. After each user logged into the system, he began a loop of actions that lasted exactly 15 minutes and then continually repeated. During this loop, each user performed the following:
- Check for new mail.
- Wait one to three minutes.
- Read first message.
- Wait one minute.
- Read second message.
- Wait one minute.
- Read third message.
- Wait one minute.
- Read fourth message.
- Wait one minute.
- Read fifth message.
- Wait one minute.
- Delete a random message.
- Every sixth iteration (every 90 minutes), open the Compose Message window, take 40 to 80 seconds to compose message, and send the message to three random recipients (senders CC themselves as fourth recipient).
- Wait until 15 minutes have elapsed since beginning of loop (so each iteration is exactly 15 minutes long).
The test was run for three hours and ten minutes after all users were logged into the system, making this a four hour test in total.
We used Tivoli Performance Viewer to investigate the connection pool size, server thread usage, and Java memory size (discussed later in this article) while we ran the multiple user load. We also used Tivoli Performance Viewer to look for memory leaks. (We're happy to report we didn't find any.) We also consulted the InfoCenter for WebSphere Deployment Manager, which provided a great deal of performance tuning information. We recommend you visit this site when looking for information applicable to your own Lotus Workplace Messaging deployment.
What we learned
As mentioned previously, the purpose of our work was to help the Lotus Workplace Messaging development team identify areas of the product that could be potential bottlenecks and scalability impediments. The data proved invaluable; we observed major response and capacity improvements from one build to the next. We also examined various ways you can tune WebSphere Application Server settings to improve Lotus Workplace Messaging performance, particularly in the area of managing system queues.
A quick overview of system queues
Every request to a Web application results in queued requests at multiple layers of the WebSphere Application Server. These requests are serviced by components of the application serving platform. These include the network, Web server, Web container, Enterprise JavaBean (EJB) container, and data sources to back-end databases or other enterprise systems. Each resource services a queue of requests waiting to use that resource. Queues are load-dependent resources, and the average service time of a request depends on the number of concurrent clients and the average time spent processing a request. Thus, queues can be an important factor in how fast the system responds to users.
Queues are either open or closed. Most queues in the queuing network are closed queues. A closed queue places a limit on the maximum number of requests present in the queue; an open queue does not. All Web servers supported by WebSphere Application Server are closed queues. Web containers can be configured as either open or closed queues. In general, it is best to make them closed queues.
It is usually better for requests to wait in the network instead of the WebSphere Application Server. The size of a queue should decrease as a request propagates through the WebSphere Application server. That is, queues closer to the client should be larger than those further from the client. The following illustrates how this works:
The following sections discuss several important queues and log file settings that should be monitored in Lotus Workplace Messaging. These include:
- IBM HTTP server queues, including concurrent threads and HTTP server logging
- Web container queues, including server threads and HTTP sessions
- Database queues, including connection pool size, connections to DB2, and statement cache size
- Java memory
Note that the recommendations contained in this article are specific to the Lotus Workplace Messaging 1.0 release. Some of this information may not apply to future releases. And upcoming releases may offer new tuning features and tips. For more information, consult the Lotus Workplace Messaging documentation.
IBM HTTP server queues
Two settings, concurrent threads and HTTP server logging, can affect IBM HTTP server queue performance.
Concurrent threads
Ensure that the number of concurrent threads running within the IBM HTTP server is below the maximum allowed. Set this value high enough to avoid bottlenecks at the HTTP server but low enough to allow enough connections through to the application server. The recommended initial value is 60 for a single Lotus Workplace Messaging mail application server. Concurrent threads are controlled through the IBM HTTP server MaxClients (on AIX) and ThreadsPerChild (on Windows) settings.
HTTP server logging
You can disable HTTP logging to eliminate its impact on performance. (This is a common technique for optimizing scalability.)
Web container queues
For Web container queues, look at the server threads and HTTP sessions settings.
Server threads
WebSphere controls the number of container requests by allocating threads to service requests. The default settings in WebSphere are:
- Minimum Thread Pool Size: 10
- Maximum Thread Pool Size: 50
We have found these settings to be adequate, but they should be periodically monitored with the Tivoli Performance Viewer.
HTTP sessions
Configure the HTTP session timeout to be as low as possible. Idle HTTP sessions consume server resources that could be better spent on active user sessions.
Database queues
Database queue settings of interest include connection pool size, connections to DB2, and statement cache size.
Connection pool size
The connection pool is used for direct JDBC calls within Lotus Workplace Messaging. If you expand Lotus Workplace Messaging by adding application servers, a data source pool exists for each node. This is important when configuring the database server maximum connections. The default values are:
- Minimum connection pool size: 1
- Maximum connection pool size: 10
You can improve performance by allowing more database connections, but the number should not be excessive. Start with a minimum of 10 and a maximum of 30. If you have multiple Web applications horizontally deployed, be careful that the total number of allocated connections do not exceed the allotted DB2 connections (see the following section).
Connections to DB2
The database configuration parameter maxappls sets an upper limit to the number of applications that connect to a database. Be sure that the DB2 maxappls setting is greater than the number of connections configured through Lotus Workplace Messaging data sources. Also be sure to account for all nodes in the deployment, as well as any session management should you configure for failover.
The DB2 maxagents setting also influences how many applications can connect. Along with maxappls, maxagents determines how memory is allocated to connecting applications. As a rule of thumb, set maxagents greater than or equal to maxappls.
Statement cache size
The WebSphere Application Server data source optimizes the processing of prepared statements. A prepared statement is a precompiled SQL statement that is stored in a prepared statement object. This object is used to efficiently execute the given SQL statement multiple times. Recommended values are as follows:
- 200 if the mail service and messaging client are deployed on the same machine
- 150 if the messaging client is deployed on a separate machine
- 50 if the mail service is deployed on a separate machine
Java memory
The Java heap size should be set to facilitate efficient application execution, including garbage collection. It is normal for garbage collection to consume anywhere from 5 to 20 percent of the total execution time of a properly functioning application. If not managed, garbage collection can be one of the biggest bottlenecks for an application, especially when running on Lotus Workplace Messaging SMTP server machines. The default heap size is 0. We do not want the WebSphere Application Server dynamically allocating memory, so it is much better to set the heap size to its expected size. We have had success with the following settings:
- If the mail service and messaging client are deployed on the same machine, set the heap size to 578 MB
- If the messaging client is deployed on a separate machine, set the heap size to 512 MB
- If the mail service is deployed on a separate machine, set the heap size to 512 MB
In conclusion...
We hope you found this article helpful as a starting point for understanding and tuning Lotus Workplace Messaging 1.0. Ideally, our test setup may be useful to you if you decide to conduct Lotus Workplace Messaging capacity and response testing at your own site. (Many administrators have found such testing very useful in planning and configuring their enterprise-wide deployments.) And our quick overview of system tuning tips and techniques will help keep your Lotus Workplace Messaging environment humming smoothly.
ABOUT THE AUTHORS
Michael Gazda has worked in Product Introduction Engineering since 2001 on projects involving Network Compression, IMAP, iNotes Web Access, and Lotus Workplace Messaging. Previously, he spent three years as a Senior Technical Support Analyst in Lotus Notes/Domino Technical Support where he handled a variety of support issues including database design, mail routing, and server issues. When not at work, Michael attends Boston University part-time, earning his BA in Computer Science.
Titus Kao has worked for IBM since 1992 on various Lotus applications such as SmartSuite, Notes/Domino, and eSuite. Titus has worked in different capacities as a quality engineer and software developer. In recent years, he has been working as a performance engineer on portal products such as K-station and WebSphere Portal Server. His most recent contribution has been to improve performance for Lotus Workplace Messaging 1.0. Titus holds a bachelor of science degree in Computer and Systems Engineering from Rensselaer Polytechnic Institute.
Joe Wissmann is a Senior Engineer working on Lotus Workplace Messaging server development. He joined Lotus after previously working as a Senior Solution Architect for IBM AIM Services and as an Architect at Ariba. Joe has a PhD in mathematics from the University of Maryland. |
 |