LDD Today


The Iris Interview
Command Performance 2: The Iris Domino Performance Team

by
Lynda
Urgotis
Level: Intermediate
Works with: All
Updated: 01-Aug-2000


There's a new tradition in the making: the August interview with and spotlight on the Iris Domino Performance Team.

In this wide-ranging interview, you'll find tips, insights, and information about what the Performance Team's been doing since last August's spotlight, as well as some hints about what's coming up next. Kick off your flip-flops, pour yourself a lemonade, and curl up with the summer's best seller.

The team was also our guest in the August 2000 Developer Spotlight; you can visit there to read their archived discussion.

For the readers who took our advice and are stretched out in a beach chair, sipping a tall iced drink, and reading this on their laptop, what are some of your coolest server performance tips?
Razeyah Stephen
My first performance tip is that if you have not joined the crowd that upgraded to R5, do so now! You can take advantage of the cross-platform, Sun Solaris, RS/6000, NT, AS/400, and OS390 improvements in performance that resulted in increased scalability for NRPC, HTTP, IMAP, and POP3/SMTP. We have heard rave reviews from customers about how, as a result of upgrading to R5 ODS (On Disk Structure), semaphore timeouts have been virtually eliminated. Other features, like multiple MAIL.BOX databases, eliminating the extra overhead of unread marks, and transaction logging have also improved performance. We recommend transaction logging for high availability and failure recovery. You can also get an extra performance bonus with transaction logging enabled with improvements in response time and CPU utilization if you have or are close to an I/O bottleneck.

Carol Zimmet
Turning on transaction logging is not used just for the performance gains. Analyze your environment and try and organize the disks so that a disk is free or underutilized so you can benefit from the reliability and availability benefits offered by transaction logging. I predict that you will be happier in the long run by taking advantage of this feature.

One of our favorite questions is "Where do you want to spend your weekends?" Is it (A) at work overseeing your Domino server, (B) worrying about the reliability of your Notes databases, or (C) doing your own thing, as you choose. Transaction logging is a capability that has a good track record. It's been proven in customer environments, and it will benefit your environment.

Razeyah Stephen
My second performance tip is to use the new integrated platform statistics for the access and analysis of operating system and Domino statistics. This single point of information capture makes life much easier. You can now look at operating system statistics such as CPU, memory, disk, and Domino statistics all in one place through the Domino server console. It's already available on NT and Solaris, and coming soon to other platforms.

Lori Fucarile
One of our favorite server performance tips is "Monitor your server stats." Collect your server stat data and review it on a weekly basis. Forewarned is forearmed! We have many customers who are seeing great performance with R5 in part because they monitor their systems closely as they scale. What do we recommend? Collect statrep data by setting up Stats & Events collection, especially during peak times -- perhaps every Monday morning from 8:30 AM until 3 PM.

Remember to enable those platform stats. These are new with R5.0.2 and allow you to view Domino and OS stats from the server console or via statrep.

Review your data on a week-to-week basis and monitor stats that reflect CPU utilization, disk usage, mail delivery latency (if you are running mail), memory utilization, and network utilization. Many problems can be avoided if you monitor your server stats and notice trends in your utilization. For example, one customer was noticing that memory utilization was increasing, although the performance was still fine. But he was starting to worry. So by analyzing it before it became noticeable to the end users, he was able to add memory and alleviate any potential problem.

Harry Murray
Here's another tip: if you are using R5 and have a large number of users (say 1000 or more active concurrent users), you may benefit from setting the following .INI parameters:

NSF_DbCache_MaxEntries=3500 (gauge number is based on number of users and number of simultaneous active database connections)
Max_Concurrent_Trans=1000
Server_Pool_Tasks=100 (not useful for Internet protocols)

Ray Sambrano
Let me talk a little more about multiple MAIL.BOXes, because people are always asking about this. What is the optimal number that you need? If you are running Domino R5, start with two MAIL.BOXes enabled. Then monitor to see whether or not your message routing patterns are optimal. If you observe overtime message contention with two MAIL.BOXes active, you can always add MAIL.BOXes to eliminate this bottleneck.

Our general recommendation is to start with two MAIL.BOXes and add more as you need them. The maximum number recommended to date is 10. To determine if you have message contention, issue the server console command SHOW STAT MAIL.WAITING. This gives you information about how much mail is waiting to be routed over a period of time. If you see that number beginning to grow, then you know that one of the paths you might need to pursue is to add another MAIL.BOX to your server.

Photos of the Performance Team
Team Photo -- Front row (L to R): Harini Srivatsan, Ken Bell, Razeyah Stephen, Carol Zimmet, George Demetriou
Back row (L to R):Mark Dowdy, Louis Bradbard, Ray Sambrano, Rama Karedla, Harry Murray, Richard Kanosky, Rich Buck, Lori Fucarile
Separate photos -- L: Cameron Hildebran, R: Nirmala Venkatraman

Over the last year, many companies have moved to R5. What are some of the themes you're hearing about as real-life users make the transition?
Lori Fucarile
One major theme has been "migration and consolidation." Many customers are migrating their R4 servers to R5 and slowly growing the number of users per box. After monitoring their systems and finding good performance, it's easy to take the next step and consolidate two R4 servers into one R5 server. This effectively doubles their scalability per box and lowers the total cost of ownership.

Ray Sambrano
Increasing the number of registered users per server and reducing the number of servers in your environment makes a lot of sense. With R5, you have the opportunity to reduce the number of Domino servers, resulting in fewer "moving parts." That's another vote for lowering the administration and management costs.

Lori Fucarile
Another common theme is "upgrade to R5 and save on hardware costs." Many customers were seeing peak scalability with R4 and their current hardware. These customers were evaluating hardware upgrades and decided to first upgrade to R5. After upgrading to R5 on the same box, they saw better performance and scalability. They were able to save in the short-term by upgrading their software instead of hardware. As you know, with better performance comes more scalability and more users per box, so ultimately these customers did choose to upgrade to faster machines to take advantage of the better scalability with R5.

Carol Zimmet
Another R5 benefit is the result of the worker thread model that we are using. This is a core Domino component that the Performance Team tests with. We could not have reached our scalability goals without it. When you upgrade to R5, you too may exceed your expectations for your scalability goals based on this architecture enhancement. We're hearing customer success stories on all platforms: AIX, NT, and Solaris/SPARC. We're pleased to introduce this architecture improvement so that you can benefit.

Ray Sambrano
We are also hearing about better performance of the Domino Router. People are very happy with the SMTP Router. Using internal performance analysis, we validated greater messaging throughput. Over the past 12 months, our internal number has been validated. And now customers are reporting throughput numbers that exceed what we have simulated internally. In R5 we also did a lot of work with the ODS and we are seeing considerable performance gains as a result.

To see the ODS performance gains, you have to compact the Notes database so that it takes on the new R5 design. For example, one of our customers migrated to R5 and reported a reduction in system resources (CPU, memory) utilization. But their biggest performance gain came when they compacted the databases and put them in the R5 format. Making that change was crucial to seeing even greater performance gains.

Hardware obviously has an effect on performance. Do you have some examples?
Ray Sambrano
Let's take a look at rebuilding views, which is the Indexer task itself. The hardware component that affects it greatly is the processor availability. Therefore, the speed of the processor affects how fast you can rebuild views.The number of processors on the other hand impacts the number of concurrent database transactions and the number of HTTP threads that can run efficiently. Memory impacts the number of client connections and the size of the buffer pool and database cache allocated by the server. Finally, disk access speed impacts the number of database connections and the speed of database and view openings. One more thing, we also recommend using disk controllers with onboard cache -- this will help increase the speed of database access times.

Carol Zimmet
Also, don't always assume that Domino is the cause when systems are not performing as hoped. We understand that problems are highlighted through the application, but the analysis phase shouldn't stop there. There are many other areas of investigation, for example, hardware, disk I/O, configuration, topology, tuning parameters, OS, and patches, among others.

In the last interview in August, 1999, the Web was beginning to have a significant impact on the kind of work you were doing and planning to do. What's been the effect of the Web on the Performance Team's work over the most recent 12 months?
Carol Zimmet
Over the last year, Web mail has been getting higher visibility within Iris' development team. We're continuing to focus on Web mail because it's a higher priority for our customers. And, as always, performance and scalability remain high on our list. We are also broadening analysis of the Web model to include Web applications. We're taking a two-level approach to Web applications analysis -- the impact on the server and optimizing the application itself.

Razeyah Stephen
We are also continuing to focus on other Internet protocols such as IMAP.

Lori Fucarile
We have a good handle on mail application performance analysis and have helped customers over the last several years to improve their performance through our server and tool development work. With R5, customers are asking us more and more about Web application performance analysis. We are seeing more and more customers using Domino as a Web application server as they move their businesses over to the Web. Our team is focused on helping them by enhancing our current tools to better monitor and improve Web performance, as well as aid in more efficient and scalable Web applications. We are most interested in hearing from customers who want to work with us in this area and provide us with customer requirements.

It seems strange to be talking about penguins in the middle of summer, but there's one little penguin who's been getting a lot of attention. What is the Domino Performance Team doing with Linux?
Cameron Hildebran
The Domino Performance Team is committed to ensuring that Domino is not the bottleneck when it comes to Linux scalability. Towards that end we are constantly testing our builds on the latest Linux kernel versions available and tracking down the performance roadblocks. Since the initial Domino for Linux release (R5.0.2) our internal benchmarks have shown over a 3X increase in the number of concurrent NRPC users and a 2X increase in active Webmail users that a server can maintain. This is due to the ongoing evolution of both Domino and the Linux kernel itself. The next major kernel release for Linux (2.4) promises many generic scalability improvements that should benefit Domino as well.

Ray Sambrano
There are still some factors on Linux that prevent Domino from achieving the same level of scalability visible on other Domino supported platforms. We expect to see closer parity with the next major release of the Linux kernel.

Lori Fucarile
So now we're asking you, the customer, what aren't we doing with Linux. We want to hear what you are doing with Linux. Are you using it in production and for what applications. We want to hear about your experiences. Do you want us to focus on this platform? In your deployments, how important to you is Domino on Linux scalability. Are you planning to incorporate small or large system configurations.

"Feedback" often focuses more on problems than praise. But I see that you have quite a success story from a customer who is extremely pleased with their uptime. Tell us about that, and how can other people can share their success stories with you?
Lori Fucarile
We love to hear from customers, not only about successes with Domino performance and scalability, but also with uptime. This customer routinely monitors not only his performance data, but also his Domino server availability. His implementation of Domino is seeing much improved availability with Domino R5. His end users are very happy with the availability of their mail and Domino applications. These are non-clustered servers, too.

During August 2000, we're the featured guest in the Notes.net Developer Spotlight Forum, so you can give us feedback there. After that, you can always add your comments in the Notes/Domino Gold Release Forum under the Domino Server -- Performance category. Please include your e-mail address if you want us to respond to you directly.

Ray Sambrano
The feedback we're receiving is that customers are successfully migrating to R5. Another successful customer recently migrated more than 10,000 users to R5 over the weekend. We are talking about more than 1 terabyte of data movement here folks, no easy task, but, with R5, certainly achievable.

We've heard of Club Med and other vacation clubs. But lately we're starting to hear about Club 2000. Should we pack the sunscreen and resort wear?
Lori Fucarile
Club 2000 is for those customers who are seeing at least 2000 concurrent users with Domino R5. This year at Lotusphere, Deluxe Corporation was one of our first Club 2000 members. Bob Knieff from Deluxe presented his data with us, and we presented him with a Domino leather jacket. While we don't have enough jackets for everyone, there are certainly a lot of opportunities for Lotus gear, and opportunities to pack your resort wear and sunscreen and present with us at Lotusphere 2001.

There are many more advantages to participating in Club 2000. The premiere advantages include participating in discussion groups with Iris and other Club 2000 customers; having first-hand access to alpha performance tool code; and being able to give your performance and tool requirements directly to Iris for faster incorporation into future releases.

So customers are running greater than 2000 concurrent users. Any tips to make this happen?
Lori Fucarile
As we look at production numbers, we have customers now reporting over 2200 concurrent active users per Domino partition. They are seeing great performance as they continue to monitor their statrep data and are even scaling to larger numbers. We encourage customers to report in with their improved scalability stats, especially for protocols like HTTP and IMAP.

There are certainly performance gains to be had with Domino R5 and a well-tuned server. In general, customers are seeing better performance and scalability with R5 than they saw with R4. We publish a lot of information on the Web that helps customers tune their servers to run with R5. Check out the Lotus IT Central Performance Zone.

It would be great to see a course or class about Domino performance. Is there anything like that planned?
Terri Karpel (from Lotus Education)
Yes! Coming soon from Lotus Education is "Performance Tuning a Domino Infrastructure." This will be a two-day workshop. You'll learn basic tools and approaches to improving Domino performance on various operating system platforms. We'll have hands-on exercises using NT workload simulations. We're also planning a one-day seminar version.

Is this course for you? The answer is "yes" if you're an experienced (6 months or longer) Domino administrator who is responsible for improving performance on existing Domino Release 5 servers. You should know the basic hardware and operating system issues, and basic networking concepts. As a prerequisite, you should have attended either the "New R5 System Administrators" or "Upgrading R4 to R5 System Administrators" course.

You have an opportunity to learn a lot. At this time, the scheduled topics are:
To find out more, stay tuned for availability and further information at the Lotus Education web site.

Lori Fucarile
The Domino Server Performance Team is very involved in this Domino Performance curriculum. Not only have we included tips and techniques from our articles, but we have also included answers to the most frequently asked performance issues. We highly recommend that you sign up for this course.

Carol Zimmet
The course also achieves a nice balance between addressing individual platform architectures and design considerations, as well as educating and discussing important Domino server performance strategies.

Can I grill you about the NotesBench workloads? What's current for Release 5? What's the main focus of the NotesBench workloads these days?
Carol Zimmet
Going forward, the workloads are evolving. We are trying for more "real life" workloads. We've always tried to be on parity -- it's just that the user model continues to change: there's more activities, more data, and more collaboration. So for each release, we re-evaluate our scalability goals and the workload requirements. This makes it tougher on us -- bigger workloads are like putting ankle weights on a person and telling that person to run the same distance just as fast, or worse, even faster! But we are rising to the challenge -- greater scalability and beefing up the workloads.

Also associated with the workloads is the work that the vendors are doing. Check out the NotesBench Consortium for new reports that are being published by the vendors on different platforms and different workloads. They also continue to do work in their performance labs on individual situations. Even though Domino R5 is out, the vendors continue to release new hardware upgrades, new scalability strategy techniques, continuously improved I/O configuration options, and new benchmark reports.

Lori Fucarile
While no benchmark ever completely reflects real-world production systems, and since all production systems are unique, we are enhancing our R5 workloads to better reflect the workloads that our customers are saying will be in effect in 2001 and beyond. These workloads will include mixed sizes for messages and larger total message sizes.

George Demetriou
As Carol and Lori pointed out, the messaging workloads for Release 5 have evolved to reflect the increasing complexity of a "typical" message. Along with the additions already mentioned, we've incorporated true Calendar and Scheduling features into our messaging workloads in order to evaluate the effect of sending and responding to invitations. This is one area that differentiates NotesBench workloads from the competition.

What's coming next for the Performance Team? What topics will we be following up on a year from now?
Ray Sambrano
In general, we're focused on three areas. First is application performance. What that really means is that we need a way to determine how Domino applications are performing. Today an application can consist of a single database or multiple databases. It is very challenging today to determine how the application is behaving and what resources it is consuming. We are investigating ways to make this less challenging to customers and business partners.

The second area we're focused on is your "out of box" experience with Domino. We want to provide tuning recommendations based on server profiles and hardware types employed. The goal is to provide a friendlier experience for first-timers and experienced users.

The third area is providing some real-time performance information to help you visibly determine how your Domino server is performing, enabling analysis of utilized systems and potentially stressed situations.

Carol Zimmet
We're still climbing a taller mountain to achieve higher scalability for our different user types (IMAP, POP3, Webmail, NRPC.) This is in line with our customer profile that's shifting to include these additional protocol paths.

The Performance Team made a splash at this year's Admin2000, sponsored by The View. What were some of the highlights of that conference?
Cameron Hildebran
I received a lot of great feedback from the presentation entitled "Is Linux the next Domino platform for you." It really highlighted just how much interest surrounds this (relatively) new operating system. There were plenty of performance-related questions in general, and Linux-related specific ones, that gave me new directions to think in. That is the best part of getting feedback, it opens up new areas to consider and also reminds you sometimes to return to problems that are still incomplete. I enjoyed hearing the real-world perspective of the way Domino is employed and especially how it is performing. When you are part of the development cycle, you don't necessarily realize all the things Domino is being used for or how many people rely on its performance and stability.

Lori Fucarile
One of the favorite topics of our presentation on performance, benchmarks, and capacity planning was performance monitoring and bottleneck analysis. We discussed important Domino statistics to watch for, and how to use Server.Load to perform performance analysis.

In addition to our presentations, we were available in the Vendor Performance lab where many of our hardware and software vendors were showcasing Domino scalability on their latest hardware. There was a lot of interest in our benchmark information as well as our capacity planning tips. We shared some of our real-world customer production numbers. We always encourage people to send us their production numbers so we can continue to share this information.

Ray Sambrano
The best thing about this conference is that it really focuses in on administration. One thing I noticed was that there are a lot of customers migrating to R5 and they are looking for recommendations on topology and sizing.

A lot of people I talked with are now looking at the configuration information that is in NotesBench reports. These are the hidden jewels. If you want to understand how to configure a scalable server, take a look at the configuration parameters used by the hardware vendors when they run their reports. They expose not only the system configuration, but they also indicate the NOTES.INI parameters they change, and any special system parameters they may have changed. Look for more information on the NotesBench Consortium web site.

As we head into the fall, the Performance Team will also be a presence at Lotusphere Berlin. Along with ein glas bier and wurst, what else is on the menu for the premiere European Domino event?
Lori Fucarile
The presentation in Berlin will be focused even more on customer R5 scalability and performance feedback to date, and on performance tuning tips and techniques. One other point: we love hearing from our international customers. Send us your scalability numbers with R5 and tell us if you are interested in presenting with us at Lotusphere Berlin.

Ray Sambrano
My presentation is about reaching a successful scalable Domino solution. I am basing my talk on customer experiences. I'll discuss what you need to know to successfully design, plan, and strategize a scalable Domino solution. The focus will be on high-performing Domino R5 implementations. Best of all you will learn about proven performance techniques. You will be able to take away information that will help you validate your server performance. I'm looking forward to it!

Performance always seems to be a moving target as new machines are introduced, faster processors appear, and usage increases. How does the Performance Team handle the testing demands?
Lori Fucarile
The Domino Server Performance Team has long-standing relationships with our hardware partners. Compaq, IBM NetFinity, IBM RS6000, IBM AS400, Network Appliance, and Sun graciously provide us with new machines as our Domino performance testing needs change. We know customers are looking at these new machines with faster hardware and new OS. It is important for us to stay ahead of the curve by providing customers with the scalability information and the Domino performance improvements they need.

Razeyah Stephen
Also, we are continuing working closely with our vendor partners to ensure that Domino continues to scale on the operating systems of various platforms.

Harry Murray
To give you some specifics, both IBM and Compaq have supplied us with the latest 8-way Intel servers. Our testing has indicated that these servers scale very well running Windows 2000. We are using many new 4-way Intel systems donated by IBM. This allows us to realistically test how scalable our new software is. Some of the servers have large memory configurations which will give us the ability to test the usefulness of more memory.

Ray Sambrano
We're noticing a trend towards using more storage area network (SAN) solutions. We are now focusing our attention in that area. We pay attention to the market and we do pay attention to trends. And then of course, like everyone else has said, we do rely on our hardware partnerships to test out the latest and greatest. We're always getting new equipment. We can't do it all, but we do try.

Carol Zimmet
Testing demands are not just limited to server level analysis. To successfully meet the high server scalability goals, we have to make sure our client benchmark systems are sufficient to support many, many users. The next planned release, in particular, has put a lot of pressure on us in this area.

An interview with the Performance Team wouldn't be complete without hearing about what's new with performance tools
Lori Fucarile
We are working on some new and exciting tools for the next major Domino release, one of which is so secret we are patenting it! Let's just say that a couple of lucky customers will be involved in this Performance beta program and will get the first chance to better predict and tune the health of their Domino servers.

Carol Zimmet
There are also some things that I can tell you about. Internally, we're working on NotesBench workloads. We've adjusted the mail delivery size to a larger average size. This is based on customer trends, such as feedback from Lotusphere 2000 in Florida and our observations from customer production data.

We're also doing some work on Server.Load. This is an end-user tool, and we're trying to keep in step with our internal benchmarking tool enhancements.

In terms of platform statistics, we're supporting more platforms and adding more statistics. For example, we're representing network utilization and error rates in the default set. The result is that more information is revealed and the end-user, analysts, and support efforts all benefit.

We're also working with the development teams on enhancing the Domino stats that are collected and maintained for different feature areas. We're always looking for good candidates: by company, by platform, or by application.

George Demetriou
Another improvement in the NotesBench tool is its increased scalability. With improvements in hardware (and, of course, in Domino), a Domino server can support more and more users. With these "bigger numbers" comes the requirement of running benchmarks to simulate these increased user levels. NotesBench has been enhanced to handle more users, thereby minimizing the number of drivers needed to generate a high-user workload.

Cameron Hildebran
As the small but vocal Linux contingent, I am always vigilant in my quest to see existing and future tools ported to this platform. There are many unique challenges involved with open source development that tend to keep this process a little behind other operating systems (but not forgotten).

Along with the new, are there are some tried-and-true performance tools and procedures that people might want to know about?
Harry Murray
I have a few comments about transaction logging in R5, partitioning, and clustering, and how they affect performance.

Recent testing for R5 shows that transaction logging may not help performance when the I/O is low to moderate, but there is evidence it may lower the percentage of disk utilization when there's high disk usage. But when the system I/O is maxed out, the only solution is probably adding more disks and RAID. Transaction logging will, of course, improve reliability and restart time.

Partitioning can improve scalability on some systems that use large memory configurations with many CPUs. Running multiple partitions can be helpful. Ray says that if you're consolidating to a small number of systems that can support large memory configurations, you should consider partitioning to achieve even greater scalability.

Iris Today has a two-part series on clustering performance that are important to review if you are running or are planning to run Domino clusters. For example, one discusses how to set a couple of NOTES.INI parameters to provide a better failover when a server gets overloaded.

[Editor's note: You can read the clustering articles here: Optimizing server performance: Domino clusters Part 1 and Part 2.]

Lori Fucarile
A sampling of Lotusphere 2000 presentations can point you in the right direction. For instance, did you know that you can use Server.Load not only for looking at scaling loads on your Domino server, but also for monitoring server performance. Or, are you experiencing database application performance bottlenecks. Try analyzing the data from the R5 console database SH DBS. The Performance Team routinely uses this information in their performance bottleneck analysis. If you want more information about either of these topics, take a look at the Lotusphere presentation Tools and Techniques for Assessing Domino R5 Performance.

Are you looking for capacity planning assistance? The hardware vendors are the best source of this. We documented all of these vendors and their information in the presentation Deploying Domino R5 for Performance and Scalability, The Sequel.

Prior to Release 4, the "Performance Team" was one person. Since then, the team has grown significantly, and now includes many experienced Domino people, as well as performance engineers and developers who have come from other companies. So, what is it that attracts and keeps people on the Domino Performance Team?
Harini Srivatsan
What attracted me the most towards the Performance Team was their enthusiasm and pride about their work at Iris. The Performance Team is a highly energetic one and is full of very talented people. They are very dedicated to their work and I am very excited to be a part of the team.

I have always worked with business applications using Java and Oracle prior to joining Iris Associates. For the past couple of months I have been busy coming up to speed with LotusScript and understanding the Domino Server as a product. I have been having a great time with it. I am currently involved in designing user interfaces for some of the performance tools developed by the team.

Lou Bradbard
My responsibilities on the Performance Team include testing products like Server.Load, platform statistics, NotesBench, Notes Object for NT Perfmon, and exciting new stuff coming up for the next release. Being on the Performance Team is being in the center of the action. We are involved with all of the major issues of the product, like large site performance, e-commerce, clustering, and Web applications.

Ken Bell
I was attracted to Iris and the Performance Team because I wanted to gain a better understanding of Domino and the development process. I've been working with Domino for about 6 years.

Lori Fucarile
Performance encompasses a need to acquire knowledge about the whole server, the client, the network, the application -- everything. And it is a great way to see the whole picture. As a product manager it means I'm acquiring a great understanding of Domino as a customer solution including the systems and applications for both internal and Web applications across all protocols.


ABOUT THE IRIS DOMINO PERFORMANCE TEAM
Ken Bell is a six-year veteran of working with Domino. After a short stint at Lotus in the Notes Support department, he joined Digital (now Compaq) as a Domino consultant. While working at Compaq he provided technical assistance to Compaq's customers regarding performance, storage, and high availability solutions for Domino.

Lou Bradbard has been with Iris for five years. He has worked with the Server and Programmability QA teams before joining the Performance Team last December. When he is not testing Domino he can be found on his mountain bike or studying Macintosh programming. He has a secret ambition to do cartoon voices.

Rich Buck joined the Domino Performance Team at the end of 1999. In many ways, Rich's career in performance has mirrored industry trends. He began doing performance work on mainframes for IBM in the early 1980's. From there, he moved into client/server performance and eventually wound up at Lotus working primarily on client side performance for SmartSuite and eSuite. Joining the Domino team has been a bit of a homecoming for Rich. Once again he is thinking about how to support many thousands of users, and how to minimize server bottlenecks. When not doing something computer related (a somewhat rare occurrence...), Rich enjoys woodworking, and exploring some of the wonderful bicycling opportunities to be found in New England.

George Demetriou started working at Iris in 1997. His current areas of responsibility include performance tool development, including NotesBench and Server.Load. He also was involved in the effort to improve Web Mail performance for Notes R5. George is an avid runner and has completed several marathons.

Mark Dowdy joined Iris in 1997 from Lotus. He has been working with the Notes server since Release 3.1.2. After a stint on the Programmability QA team, Mark is now working on application performance and the Agent Manager.

Lori Fucarile joined the Domino Performance Team in 1999 and is the Senior Iris Product Manager for the Domino Server Performance Team. She is currently involved in delivering a new server health monitoring tool that will help customers better predict and tune their servers. Lori has been a Product Manager for Domino for 4 1/2 years and as a Lotus product manager delivered Domino R4.6 and R5 Enterprise Server, Domino Clustering for R4.6 and R5, and the Lotus Calendar Connector for OfficeVision R4.X. Before becoming a product manager, Lori managed technical alliances with the (then) Big Six in the Lotus Strategic Alliances group. Before joining Lotus, Lori worked at IBM as a consultant for enterprise customers focusing on decision support applications and as a systems engineer for insurance and telecommunications customers.

Cameron Hildebran has been a thorn in the side of the Domino Performance Team since 1998. Before that he took up space #2341 in the vast cubicle expanse of Digital Equipment Corporation. He has been interested in all things computer since watching "Plan 9 From Outer Space," subsequently coming to the realization that computers would one day rule the world, and therefore deeming it prudent to be on the winning team. When not analyzing Dilbert cartoons for the hidden meaning, Cameron teaches dyslexic geese to fly south for the winter.

Rama Karedla realized that the planet is not yet ready for telecommuting from mountain tops, came back to the world again, and joined Iris in May. Prior to Iris, he worked at Compaq in the area of advanced development of storage products, I/O performance, and device driver development. Rama is a developer in the Performance Team and currently works on a Domino performance feature called platform stats and a forthcoming product yet to be named. In his spare time, he manages a not-for-profit music school and works on a dream to teach music in real-time, over the Internet.

Richard Kanosky is a Domino Performance Engineer who has worked at Iris since 1997. He previously worked at Lotus on the Network Protocols Quality Engineering team, which assisted in the Notes R4.5 quality engineering effort on the XPC protocol. He also worked with George Sprott on X.PC/ TCP/IP Passthru performance, which was presented at Lotusphere '96. Since joining the Performance Team, Richard has worked on Server.Planner, NotesBench, and Server.Load. In his free time, he enjoys open ocean swimming, and playing sports with his daughter, Christina.

Harry Murray joined the Domino Performance Team in 1998. He is currently involved in the testing of Domino IMAP on Windows 2000 systems. Prior to joining Iris, he worked for Digital Equipment Corp. in their performance group doing NotesBench testing of Domino on Digital servers. Before that, Harry was involved in the system management of many Digital production systems and was manager of System Technical Support in a number of Digital facilities.

Ray Sambrano is a Senior Product Manager with the Domino Server Performance Team. In this capacity, he defines server performance requirements, monitors market trends, and oversees the delivery of benchmark data. Previously, he worked at Lotus for seven years, most recently as a Senior Manager in Lotus Worldwide Business Partner business unit. Ray also worked as a Product Manager for Lotus in the system management arena, where he managed the release of Lotus NotesView. Prior to Lotus, Ray worked in the financial sector deploying corporate networks and messaging systems.

Harini Srivatsan has always worked with business applications using Java and Oracle, prior to joining Iris Associates this year. She is involved in designing user interfaces for some of the performance tools developed by the team.

Razeyah Stephen is the co-lead on the Domino Performance Team. She has worked at Iris since 1998. She came to Iris from Digital Equipment Corporation, now Compaq, where she worked for five years in their StorageWorks division. Razeyah's specialties include UNIX and I/O performance.

Nirmala Venkatraman works for Iris as a contractor. She started in April, 1998, and primarily works on UNIX performance. She previously worked at Sun Microsystems.

Carol Zimmet started working at Iris in 1994. She is the co-lead on the Domino Performance Team, and responsible for evaluating performance and performance tool development. Carol continues to search for the one-step solution to everyone's performance problems. She is also interested in a 'white box' approach towards improving the quality of the product. Carol enjoys bicycling with her kids and playing racquetball. She has a longing to return to stained glass!


ABOUT LYNDA URGOTIS
Lynda Urgotis began her career during the Paleolithic era writing about chipped-stone tools. She has documented her way from Data Resources to PSDI to Software House. She is rapidly approaching her thirteenth anniversary at Lotus where she contributed to Lotus Improv, SmartSuite, eSuite, and now, Notes. She wrote the Que book, "Quick Reference for Improv" as well as chapters for the Improv Que book. Gardening delights her spirit. Her daughter, Megan, and husband, Michael, are just plain delightful.