Uncommon ground: Lotus Engineering Test Common Services

Uncommon ground: Lotus Engineering Test Common Services

Interview by
Tara Hall

Level: All
Works with: Lotus Workplace
Updated: 22-Sep-2003

Common Services—what are they and why do you need to know about them? We spoke with Carol Zimmet, manager of the Lotus Solutions Engineering team; Steve Davis, one of the Lotus Solutions Engineering team managers; and Craig Garfinkel, project leader for Common Services to answer these and other questions.

Can you define for customers what Common Services are?
Carol Zimmet
Common Services represent an innovative approach to addressing two main initiatives. One is offering a centrally available set of components and products for Lotus development efforts and beyond (within IBM) to verify successful product integration and interoperability as defined by our customer configurations and deployments. These enterprise-level, integral product components are offered by our team in a hosting service capacity, in which we provide the platforms and technology and the teams provide the evaluation requirements and connecting components. For example, product development efforts meet release criteria when development and testing has been done with a variety of LDAP servers and single sign-on (SSO) servers as part of the Common Services offerings. Our team makes these key product components available to the wider community, before product release, and more importantly, before deployed in the customer environment.

The second interesting dimension of Common Services is having much more product technology available to the Lotus community than any one team could develop and support by itself. Common Services includes a lab that is administered and supported by people who are becoming experts in these technologies—LDAP, SSO, and so on. It's a product development and release challenge to expect all this technology to be developed and known by individual teams. Instead, having that knowledge centralized gives these development teams greater exposure for developing and testing their products with a better end deliverable result.

Externally, the Common Services team offers knowledge expertise focused on specific product areas. This concept is really taking off not only for the development teams who are developing new usage strategies, but also for other teams who find helpful integration points between what they want and what we have. Within a short time period, we have put our Common Services team on the path to becoming experts in these different technologies.

Steve Davis
Common Services are a robust set of functions that represent customer environments. We provide these services to the test organizations, so they have more real customer-like scenarios in which to test their software.

To give you an example, all of the Lotus products—the traditional Domino products and the new Lotus Workplace products—all have an initiative to use standard LDAP directory servers. If each sub-team—development and test teams—has to maintain a matrix of the seven, eight, or nine LDAP directory servers, then that requires each team to obtain hardware and to hire administrators to maintain those systems. Typically, the administrators on those sub-teams don't have the LDAP expertise required, and their environments may lack the kind of complexity that a customer has. So they have less than they need because they can't afford it, and the environments are simpler than real-world environments.

But we can provide that expertise for the teams. We make sure that the LDAP servers have a great deal of breadth and depth as any large enterprise customer would have. We save money because we provide one set of servers to multiple teams. We also save administrative costs. Those administrators are experts, so they can drive the understanding of how that service is utilized in test environments to be more customer-like.

Back row (left to right): Steve P Davis, Frank Sperling, Dave Curley; middle row (left to right): Jack Williams,
Kevin Bittner, Yuriy Veytsman; front row (left to right): Michael Gazda, Sam Alexander,
Steve Mark, Joe Grace, Ray Smith, Lori Davidson, Bruce Webster

Which teams have been the primary consumers of your services?
Craig Garfinkel
The Lotus Workplace development teams have been our biggest customer to date. We are supplying the System Verification Test (SVT) team with many non-core technologies to do their testing, including LDAP, SSO, HTTP, and proxy and performance monitoring. Additionally, we provide the Function Verification Team (FVT) with these same services and also provide them with shared Lotus Workplace servers. We have also provided services to Lotus Support, ISSL (IBM Software Services for Lotus), and Sales.

Carol Zimmet
We support teams around the world particularly in China and Ireland, so we make our services available around the clock. As a result, we've enabled others on remote administration, which luckily hasn't been exercised!

We receive different kinds of requests internally. For example, the Lotus Usability team wants access to our pre-GA product offerings to perform usability studies on the next release of Lotus Workplace. They are looking for an exclusive kind of access. The UA (User Assistance) team that provides product documentation has approached us about access to our Common Services to explore and to experiment with products. This access will help them meet their deliverables. That team takes a different tactic when it comes to using the systems, so we're planning for some challenges.

Another IBM team, ISSL—our product consultants—used Common Services components as part of their bootcamp training. You bet we had to make sure the services were up and available to meet that commitment!

You have quite an extensive lab to support these Common Services. Could you tell me what hardware is in the lab?
Steve Davis
We're a software group division, so we try to be hardware agnostic and to represent where our customers want to be now through the next 18 months. We're aggressively trying to get the latest hardware in place, while not forgetting that we have many customers who are still on older hardware, too. We have a mix of UNIX-based systems—like IBM's pSeries systems including 4-way p630's and a 16-way p670 Regatta divided into Logical Partitions (LPARs) that give us flexibility to provide the right size systems to meet the on demand needs of the test and development community.

We have Sun servers including 2-way 280R's and a 12-way SUN 6800 that we've configured into multiple hardware domains—Sun's equivalent to the pSeries LPAR—to demonstrate how this high-end hardware can be configured to satisfy on demand needs. We also have a mix of IBM Intel-based processors, including older Netfinity servers, x330's, x345's, x360's, and an IBM eServer BladeCenter. The BladeCenter has eight separate two Processor Blades—2.0 GHz Xeon Processors, 2.5 GB of Ram, and 40 GB Internal Drive Space on each Blade.

We also have various types of storage products like the IBM storage server known as the Shark, FAStT fiber connected disks, SCSI connected EXP300 disk arrays, SSA disks, and a Network Appliance Filer to represent network area storage (NAS) devices. We're implementing storage area networks (SANs) through switches as well. We have 1.5 Terabytes of FAStT storage—one FAStT 600 with two Exp 700 expansion drawers—connected to the Blade Center via its Fiber Switch module. We can allocate storage exclusively to each individual Blade as needed.

Carol Zimmet
We not only are offering software, but we're also offering hardware and different kinds of configurations. It's also exciting that we have an iSeries system available in our lab. This system is a real demonstration of the cross-server deployment strategies that our customers are following. For example, we can use the iSeries server to test Lotus products that utilize DB2 servers. In terms of interoperability exposure, it's very important for us to have this system as part of our Common Services. It's critical to have our software products run on these different leading edge systems before releasing them to our customers! And zSeries is not left out! In the short term, we've negotiated shared access to another system to support any requests that come in.

To highlight our appreciation of the most recent addition, the BladeCenter, we're finding different, interesting applications to evaluate on the BladeCenter as a platform. The Blade with its unique architecture presents interesting design options for failover and reduced total cost of ownership.

It sounds like many of the configurations represent the environments of enterprise customers. What about small- to medium-sized businesses?
Craig Garfinkel
We cater to the small to medium business, but we try to make configurations that are as complex as possible so that we include the bigger customers. We include the types of environment that they operate. As a result of having a configuration that complex, we necessarily handle the environments that are common in a small- or medium-sized business.

Steve Davis
To answer this question, we need to talk about the other major function inside this group. Carol manages the team called Lotus Solutions Engineering, and Common Services are part of that team. Another area is solutions test, and it is one of the stops on the testing cycle. Solutions test comes after system test and is the last phase in the testing process. Solutions test is about building a set of environments, scenarios, and tests that represent real-world customer environments: very large, complex, interactive software pieces, hardware pieces, and vendor software. It's intended to drive out those levels of interoperability and complexity that aren't captured downstream. The point being that small and medium business are part of our solution scenarios, but it's easier for the downstream test organizations—the system test organizations and the function test organizations—to build those rather tightly coupled, smaller configurations for the small to medium business. Those configurations use our Common Services as part of their test environment as do our very large solution scenarios used in Common Services. For example, currently we have two separate instances of Lotus Workplace on individual Blades in a demo deployment—DB2, WebSphere Application Server, WebSphere Portal, IBM HTTP server, and Lotus Workplace all on a single Blade. Our goal is to deploy a clustered instance of Lotus Workplace across three Blades with IBM HTTP server on a fourth Blade and remote DB2 on a separate pSeries box. We also have Domino 6.5 on a single Blade. Sametime 3.1 on a single Blade, and the latest version of QuickPlace on a single Blade. All servers are connected to the network with a 1 Gigabit connection via the BladeCenter Gigabit switch module.

Carol Zimmet
To build upon what Steve has said—and this isn't specifically for small to medium businesses, but it could be used to reassure those businesses—to achieve more complex goals, you have to show other types of configurations that address their needs. In Common Services, we have many different types of interactions going on simultaneously. We are already developing integration evaluation points for different products. They interact with one another in an integration flow that's similar to a small to medium business all the way up to an enterprise customer. We may have application A being tested against an LDAP server and application B also being tested against that same LDAP server, much the way that you would find in a customer environment. The real win is the simultaneous exposure of activities typically found in a customer environment.

Left to right: Geno Spinosa, Cynthia Barber-Mingo, Ivan Dell'Era, Bill Hankard, Carol Zimmet

Let's talk a little bit about software in the lab. What are you currently testing and which third-party products are you testing with?
Craig Garfinkel
We currently have six major categories of products that we host: LDAP, SSO, HTTP, proxy, monitoring, and Portal. Within each category, we support a combination of IBM technology and third-party products. The following table shows these categories and the products in each one.

Category	IBM technology	Third-party technology
LDAP
	IBM Directory Server 4.1 IBM Directory Server 5.1 IBM Lotus Domino 5.0.12 IBM Domino 6.0.2	Microsoft Active Directory 2000 Novell eDirectory 8.6.2 Sun ONE Directory 5
SSO
	Tivoli Access Manager 4.1	Netegrity Siteminder 5.5
HTTP
	IBM HTTP Server 2.0.42.2 IBM Lotus Domino 5.0.12 IBM Lotus Domino 6.0.1	Apache 1.3.2.0 iPlanet 4.1 Microsoft IIS 5 Sun ONE 7.1
Proxy
	WebSphere Edge Server 2.0	N/A
Monitoring
	Tivoli Framework 4.1 Web Infrastructure 5.1.2 Configuration Manager 4.2	N/A
Portal
	WebSphere Portal 4.2 WebSphere Portal 5.0	N/A

In addition, we have multiple servers running the Beta release of the next Lotus Workplace release.

Carol Zimmet
Another important dimension we've explored is to expose our products to other IBM product family technologies not only for products coexistence exposure, but to model and understand how enterprises use our software. We are utilizing Tivoli Monitoring in our Common Services Lab and have enabled the Workplace System Test and the Lotus Labs teams with monitoring capabilities through Tivoli Monitoring. So we're verifying product coexistence as well as using the product as it was intended for our own monitoring requirements. We have a proactive new relationship with that Tivoli team in which we've received Beta releases of their software to work with, so we can make an impact in the correct phase of their product cycle too.

We're also enabling Domino Everyplace for extended technology integration exposure.

So these are different areas and technologies that you're exploring?
Steve Davis
Yes, for example, DB2 is, of course, underneath most of the Lotus products. We'll introduce Oracle soon.

After we configure the solution scenarios test environments, if all or part of the configuration can be used by other test and development teams to enhance their own testing configurations, then we may offer these environments as a Common Service for all to use.

Carol Zimmet
We also use Rational tools in our environment. With the Rational Suite TestStudio offering, we're developing different kinds of workloads. We use the Rational tools to develop Lotus load and scalability type analysis. And we're putting some load on the components that are found in Common Services, so we can simulate our customer environment and varying user profiles. In a customer environment, there is always a lot of activity going on, on many different paths. And down the line, every component should be exercised by having some load put against it. Rational Suite TestStudio can help with simulating that user and component load activity. Examples are making sure lookups and updates are being done against directories and making sure the Domino indexer is active in the background, servicing a variety of quantity requests.

If you take a step back and review what's covered and accomplished, you recognize the wide representation of IBM Software brands—Websphere, DB2, Lotus, Tivoli, and Rational—in one lab. You also see the presence of the eServer systems—iSeries, pSeries, xSeries—and a zSeries system is being made available to us for our future upcoming product requirements.

Did you have any customer contact to determine which services to offer? Or had you spoken only with the development teams and with Field Support Services to determine the services?
Craig Garfinkel
Common Services are relatively new. To get started, we really relied on the Lotus Workplace development teams. Most of the early products that we chose were third-party or non-Lotus products that they want to support in the next release of Lotus Workplace. It just so happens that those products are also useful to the Notes/Domino teams as well. Going forward—now that we have this base of products—we're reaching out to customers, to people from sales and consulting, and to people who talk to customers and who know their needs. We get input from them, and we have triage meetings to discuss what we need to offer next and how to make it realistic.

Steve Davis
We are part of a new organization within the Lotus division that is an engineering test organization under the Vice President of Worldwide Technical Support, Bob Biamonte. It aligns our team with the support organization. In addition to all of the input that we get from the places that Craig mentioned, we also get input directly from Lotus Support Services. That input helps us to formalize mechanisms to ensure that if there is a pervasive interoperability problem with our products, it's exposed to us. Then we can figure out how to build a new test environment to ensure that we capture problems before a customer does.

Carol Zimmet
With our LDAP servers, we contacted and polled our customers about their LDAP designs and content strategies. Now we're in the process of incorporating that knowledge into our Common Services LDAP servers to benefit all our products. Our goal is to test our products against components—LDAP directory services in this case—that represent the design caliber and content quantity and type deployed in customer environments. Customers can now feel a greater level of confidence about the integrity and the exposure Lotus products have before release because all of these capabilities are available and exercised by the developers and the test teams before a product goes out the door!

Let's talk a little bit about how this testing has changed the development cycle for many products, particularly Lotus Workplace. What impact has Common Services had on development and test teams?
Carol Zimmet
I see two ways in which teams are affected. When the teams develop their tests, they design automation tests for multiple component verification. The greatest example includes LDAP servers. Before, teams designed their tests to work with one LDAP server. Now, they design their tests to point to multiple LDAP servers in an automated fashion. So we know that they run their tests successfully with multiple components that customers may have installed.

Secondly, we're also developing more expertise on the Common Services team, so that we can advise test engineers and developers about how best to integrate this technology with their products, so they'll be more successful. We're learning how customers use LDAP servers and how customers use single sign-on devices. Now, the test engineers don't have to take all the responsibility for making sure that their products properly integrate. For instance, WebSphere Edge Server is proving to be a real challenge because of the variety of roles and configurations. We can advise them how to use it.

From a bigger picture viewpoint, we're moving resources and quality emphasis up in the product cycle. We've learned that fixing an issue as early as possible in the development cycle is a wise investment—though we would rather invest in more product features—and we are planning proactively.

Are you now part of the testing cycle for many development teams or are many development teams still relying primarily on their own internal QE teams at this point?
Carol Zimmet
We're lined up with the feature verification test teams in their cycle. We're also lined up with the system verification test teams for more complex configurations based on platform exposure. So, we're integrated in the cycle. Remember, we operate in a hosting capacity, so we offer the services to the testing teams who can now leverage them.

Steve Davis
We work with every cycle. The development teams use our Common Services; the function test teams use our Common Services; the system test teams use our Common Services; and our own solutions test team uses our Common Services. So we're deeply involved with the test cycle. The support teams also use these Common Services to recreate support problems that are post-release issues.

Do you continue testing the product even after it ships? Or are you only involved in the pre-release test?
Steve Davis
Think of Common Services as hosted services available for any of the Lotus teams to use. Although we are not support analysts—people on our staff are not troubleshooting a specific support incident—a support analyst could use our Common Services to build a test environment to troubleshoot a support incident.

Carol Zimmet
Because we look at ourselves as a hosting organization, we are providing a set of services, and the teams with whom we work provide the integration of interface requirements and skills. For example, we work closely with the Domino LDAP development team. They are greatly appreciative of the Common Services offerings for a variety of analysis points. One way they recently were able to utilize our LDAP services was to compare how the different LDAP servers responded.

How many services are currently being utilized right now? Are all the services you offer being utilized?
Steve Davis
We can't get the servers up fast enough. There's tremendous enthusiasm from the user organizations for wanting these services. Apparently, Carol and Craig have done a great job of evangelizing these services to the teams, and they've had no problem—once they understood what they were and where they were—getting people to use them.

We talked a lot about internal Lotus teams taking advantage of the lab. What about people who are external to the organization? Can business partners, for instance, take advantage of the lab in some way?
Steve Davis
A program we have within Lotus Engineering Test that could be used by business partners is our residency program—similiar to the residencies that the IBM ITSO Redbook organization produce. This is a program that we hope will allow outside organizations to participate with us as we either put services together or put solution scenarios together, so we can gain from their expertise. We haven't done this yet, but if a business partner had an application that made sense to have in one of our solution simulation scenarios, he or she could participate with us in putting that together, could take away some expertise on our products, and leave us with some expertise about their products. Although that's not in place yet—we haven't got business partners in residencies—we're doing residencies with outside IBM organizations to ensure that we know how to do this and how to implement it.

Carol Zimmet
Discussions have opened up about working with business partners in a variety of capacities. One way is to bring partners in to work side-by-side with us to learn about the product technology integration. Another way is for business partners to bring their applications and services into our environment. We get the exposure of their applications in our lab as much as their offerings gets exposure to our services. This is another dimension in the concept of a LET interoperability lab. I know business partners will also have creative suggestions about bringing these two environments together.

You're trying to stay 18 months ahead of the current technology. How do you know what's going to come out 18 months from now and how do you ensure that you always have the latest release of the software or the most up-to-date hardware?
Steve Davis
We forge relationships within the company and with business partners to ensure that we're staying briefed by the hardware development teams and the software teams and vice versa, so that we know what's coming.

Carol Zimmet
We also have a good track record in this area. The hardware and software teams know that we have successfully leveraged Lotus products working with their configurations. We have executed on the plan and delivered on the proof-points, so we have willing teams to work with. We have started new relationships with new teams through this centralized service. Both teams are seeing the win-win benefits in supporting this type of effort.

Obviously, it costs a lot of money to always have the most up-to-date hardware and software. What's your break-even point?
Craig Garfinkel
We are successful when the number of complex interoperability problems in our products are reduced. Providing the Lotus community with hardware, software, and a realistic environment enhances the testing they can perform and results in a better product.

Carol Zimmet
Issues exposed as early as possible in the development cycle cost the least to address—not to mention the customer win by having a streamlined process when products are deployed. We are glad that we are converting some of our dreams into reality! Our payback comes with increased customer satisfaction with our products and shorter time to product deployment in a customer production environment.

Steve Davis
We work overtime to see a downward trend in the number of interoperability problems escaping into the field.

With all the Common Services testing going on, you must have some deliverables. There's information that needs to get out to customers. What are those deliverables and how can customers find them?
Carol Zimmet
From the beginning, we always keep our eyes and ears open for someone who may be able to leverage what we learn. For instance, when it comes to installing a product or component, if we have any clarification points or problems or if we have to update the installation instructions, we put out technotes or knowledge share internally, but we try to circulate information about how we learned how to better install the product in the form of a write-up that can help individuals externally. What we see ourselves doing is developing knowledge and expertise about different configurations people use with these components. That's another form of knowledge share that we provide. We hope in the future to offer technical articles or internal and external presentations about what we learned and how we see these components being used. We want to act as consultants and advisors. So there are many different types of knowledge share opportunities—verbal and written—that we plan to do and will continue to do.

Craig Garfinkel
We aggressively seek other teams to participate in residencies. They utilize the technology in our lab to solve a relevant business problem and agree to document the solution, so it is available both internally and externally.

Steve Davis
Release notes, installation guides, and configuration documentation for our products will provide more details because of Common Services testing. Plus, you'll see white papers, configuration recommendations, technical articles, and potentially even Redbooks that utilize our environments to produce best practices and best configurations methods recommendations.

What is your vision for the future of Common Services?
Carol Zimmet
Since the concept of Common Services has been translated into practice, there have been additional services and components that have been requested. We see ourselves in the near future developing our Common Services to support firewalls, portlet coexistence, VMWare, and Web Content Management (Aptrix). Another interesting dimension that we plan to grow into is providing important customer quality test case resources. We earlier described our implementation of LDAP directory services that model enterprise customer environments by design and content. There are also other enterprise caliber components that we can provide for teams to test against. Examples are large databases (based on complexity, size, number of records, and so on), datastores used for complex searches, and data files with known problems that will force the code to go down an unsuccessful path. These are just a few of the ideas we're exploring. We're open to hearing more.

If customers have suggestions for the Common Services lab, how can they contact the Lotus Engineering Team?
Carol Zimmet
Customers can respond to this interview (click the "What do you think about this article?" image to provide feedback) to let us know their suggestions for hardware, software, configurations, and so on. We want to know how customers have architected and configured their environments, so we can incorporate that knowledge into our environment. Remember your recommendations affect the way we develop and test our products; our goal is to move the analysis points up in the whole process. Our customers can make a difference, so we want to hear from you!

ABOUT THE AUTHORS
Steve Davis has worked for IBM since the 1980's in many technical and management capacities. He came to the Lotus division in 1998 where he has been managing the lab that currently houses the Lotus Engineering Test Common Services and Solutions Test systems.

Craig Garfinkel works for the IBM Lotus Engineering Test team where he is the Common Services project leader. He is also the primary engineer supporting Netegrity. Previously, Craig worked as a lead performance engineer focusing on Lotus Domino and Lotus Sametime.

Carol Zimmet, Sr. Manager of Solutions Engineering team, guides her team's Common Services, Solutions Tests, and special customer analysis efforts. She is also the Lotus representative on the team dedicated to integrating Rational tools in IBM's internal product development process. Previously Carol worked in-depth with Domino products. She recently gained product exposure working with the Sametime, QuickPlace, and Lotus Workplaces teams.