LDD Today

Lessons learned from the Notes/Domino 6 Early Deployment program


by
Tara
Hall
Level: Intermediate
Works with: Notes/Domino 6
Updated: 01-Oct-2002

The Notes/Domino 6 Early Deployment program began in early 2001 when the first Notes/Domino enterprise customer joined the program. The goal of this program was to invite three to five enterprise customers who could provide "real world" testing of Notes/Domino 6 before we shipped the product. In return for their commitment to deploy Notes/Domino 6 enterprise-wide, these customers receive 24-hour, on-call support from both keygrips and development relations managers (DRMs).

Keygrips are volunteers from the Notes/Domino development teams who act as liaisons between the early deployment customers and the Notes/Domino development team. When an early deployment customer runs into a deployment-blocking issue, the first person called is the keygrip. The keygrip's responsibilities include:
The DRMs are assigned to the early deployment customers to manage the accounts. While the keygrips address technical issues, the DRMs address nontechnical issues, including third-party product interoperability. They also provide executive support. But like the keygrips, the goal of the DRMs is to help enable the customer to move from a pilot testing environment to a production environment.

For this article, we interviewed the DRMs to find out what lessons these early deployment customers learned that can help you with your Notes/Domino 6 deployment. Environments vary for each customer as do deployment plans, but there is one commonality among all the customers: pilot programs. No customer deployed Notes/Domino 6 without testing the client and server in a non-production environment. When they were ready, all of these customers moved from pilot to production, deploying early builds of Notes/Domino 6 in their organizations.

This article is intended for Domino system administrators responsible for enterprise-wide deployment of Lotus Notes and Domino.

Customer 1

Industry:Automotive
Number of Notes clients:45,000 (North America only)
Notes client versions:5.0.x
Number of Domino servers:100+

Our first customer was selected for the Early Deployment program, in part, because of the number of Solaris servers in the organization. We wanted to ensure that each server platform received real-world testing. To date, this customer has deployed seven Domino 6 servers in their production environment:
In addition, the customer has deployed approximately 150 Notes 6 clients to volunteers in their pilot program, with all of them using the Notes 6 mail template.

Calendar and Scheduling testing
Another reason for selecting this enterprise customer was their avid use of Calendar and Scheduling (C&S). Much of their pilot program involves intensive testing of the Notes 6 mail template, including early deployment of the template to the organization's administrative assistants—their primary C&S users. In fact, part of their deployment criteria includes receiving the "okay" from their administrative assistants before deploying the template to the larger organization.

When planning your deployment, you should determine which users or groups of users you want to involve in your testing. For instance, do you have Notes "power users" in your organization? Do you have intensive mail or C&S users who you might include in your testing? What about your application developers? Make sure to choose testers who are representative of your larger user population.

One area of C&S testing that this customer focused on was interoperability—making certain that invitations created in a Notes 6 client can be read, accepted or declined, modified, and so on by a Notes 5 client. Overall, the customer found relatively few interoperability issues. Those that were found have either been addressed or resolved. But be aware that when using a Notes 5 client with a Notes 5 mail template to reply to an invitation created with a Notes 6 client and template, you are limited to the Notes 5 client and mail template functionality. For instance, in Notes 6, a meeting chair can send an update to a meeting in which the Notes 6 invitees can select the Update Calendar option from the update notice. Notes 5 invitees, however, do not have that option because it's new to Notes 6.

Upgrading mail templates
Additionally, Lotus strongly recommends upgrading clients before upgrading mail templates. Not only are new templates on older clients an unsupported configuration, but interoperability is also limited. For instance, you cannot respond to a Notes 6 invitation using a Notes 5 client with a Notes 6 mail template because the action bar does not appear on the invitation. When you open the invitation, you receive an error message asking you to upgrade your client.

C&S interoperability warning

This situation can occur when you have people who manage other people's calendars. For instance, in the case of executives and their administrative assistants who manage their calendars, you have two options:
Whichever option you choose, make sure to upgrade the administrative assistants' Notes 6 mail file template at the same time that you upgrade the executives' template or upgrade the administrative assistants' template prior to the executives upgrading their mail file templates.

In organizations that use a pool of administrative assistants who can be assigned to any executive, make sure to upgrade everyone's template last. Executives can use a Notes 6 client with the Notes 5 mail template, while the administrative assistants can use a Notes 5 client and Notes 5 mail template or vice versa. This avoids any potential interoperability issues.

Unless you are willing to sacrifice some functionality, upgrade your clients before you upgrade the mail templates. While the Notes 6 client supports a Notes 5 or earlier mail template without loss of functionality, earlier clients have only limited functionality with the Notes 6 mail template.

Customer 2

Industry:Health care
Number of Notes clients:70,000
Notes client versions:4.5.x, 4.6.x, 5.0.x
Number of Domino servers:190

This customer was chosen, in part, because of their extensive use of Notes/Domino 4 and 5 as well as their use of PDAs. This customer's users include a number of doctors who use PalmPilots to enhance their mobility. Because many of their users do not require full Lotus Notes functionality, customer 2 has a strong interest in iNotes Web Access. To date, this customer has deployed eight Domino 6 servers in their production environment:
In addition, they've deployed 75 Notes 6 clients to their users, with all using the Notes 6 mail template. They've also upgraded the mail databases to the Domino 6 database format (ODS43).

Smart Upgrade and Seamless Mail Upgrade testing
Much of their testing has focused on the Notes client and features that help deploy the Notes client and mail file template: Smart Upgrade and Seamless Mail Upgrade. Both of these features require you to implement the new Domino 6 policies to enable them. This customer has been testing Smart Upgrade with daily builds of the Notes client that include the Microsoft Windows installer. Smart Upgrade testing has been performed successfully on both Windows and Mac clients.

If you've installed the Notes 6 Release Candidate, you can use Smart Upgrade to upgrade to the Notes 6 Gold Candidate, and you will be able to use Smart Upgrade to upgrade to new Notes 6 Maintenance Releases as they are released. You cannot use Smart Upgrade to upgrade a Notes 4.x or Notes 5 client to Notes 6.

Customer 2 tested Seamless Mail Upgrade with their 4.5.x, 4.6.x, and 5.0.x mail templates after upgrading the clients to Notes 6 and found that the feature can be used with pre-6.0 mail templates. In some cases, the mail databases contained upwards of 300 folders, and Seamless Mail Upgrade was able to upgrade the databases and folders without data loss. Upgrading folders is optional with Seamless Mail Upgrade.

Your other mail template upgrade options include Upgrade-by-mail and the mail conversion utility, or the convert task. All options can upgrade Notes 4.x and Notes 5 mail templates to Notes 6 mail templates or to another specified template, like the iNotes Web Access template or a custom mail template. If you want to use Seamless Mail Upgrade to upgrade mail file templates, users must be assigned to either a policy or Setup Profile. Neither Upgrade-by-mail nor the mail conversion utility require policies or Setup Profiles; however, Upgrade-by-mail does require users to have Designer access to their mail databases.

For information about setting up Smart Upgrade, Seamless Mail Upgrade, the mail conversion utility, and Upgrade-by-mail, see the Domino 6 Administrator Help.

Hub servers and releases
One factor that makes this customer unique among the early deployment customers is the number of Notes 4.5.x and 4.6.x clients and servers still in use. One way in which the customer has dealt with the mixed-release environment has been to create hub servers that cater to each release. That is, all 4.x servers connect to a 4.x administrative hub server, all 5.0.x servers connect with a 5.0.x hub server, and with the deployment of Domino 6, all Domino 6 servers connect with a Domino 6 hub server.

While it is safe and recommended that you upgrade to the Domino 6 Directory design regardless of server release, this customer chose to use 4.x, 5.0.x, and 6.0 directory templates. Replication is set up between the hub servers to replicate documents but not the design elements of each template. If you operate a mixed-release environment like this one, you may want to consider doing the same. A configuration like this allows you to replicate your Domino Directory to your Domino 4.x servers without concern for the UniqueNameKey (UNK) table issue. The UNK table issue is discussed in more detail later in the "Domino 6 Directory and the UNK table" section.

Third-party software interoperability
This customer is a user of Network Associates Inc. McAfee GroupShield, virus protection software designed specifically for the Domino server. Due to the customer's size, they were able to influence McAfee to begin testing their GroupShield product on Notes/Domino 6. Your organization can benefit from this early deployment test of GroupShield with Domino 6. It's important to make sure that any third-party software that you are using is Notes/Domino 6 compatible. This includes other Lotus products like QuickPlace, Sametime, and so on. Make sure to include third-party software in your test plans to ensure compatibility with Notes/Domino 6. If you find software incompatible with Notes/Domino 6, you may need to upgrade to a compatible version if one is available or continue using the software in a pre-Notes/Domino 6 environment.

For instance, QuickPlace and Sametime 3.0 both require Domino 5.0.10 servers, so you need to maintain some Domino 5 servers in your environment to support QuickPlace 3.0 or Sametime 3.0. This should not impede your Domino 6 deployment plans because your Domino servers should be dedicated, so you won't be installing QuickPlace or Sametime on your Domino mail or application servers. Keep in mind, too, that Domino 6 can coexist with Sametime and QuickPlace even though they require Domino 5 servers. Your Domino 6 Directory can connect with Sametime, QuickPlace, and earlier versions of Domino.

A mixed-release environment should not pose any problems for you, but be aware of which features require an entirely Notes/Domino 6 environment and which can function in a mixed-release environment. This article and other resources can tell you which features work in which types of environments.

Customer 3

Industry:Electronics
Number of Notes clients:4,000
Notes client versions:4.5.x, 5.0.x
Number of Domino servers:40

To date, customer 3 has already deployed Domino 6 to about half of its 40 servers. This customer has a number of custom Notes applications and an interest in deploying other Lotus products like Sametime, QuickPlace, and Domino Everyplace. While many of the customers in the early deployment program had users numbering in the tens of thousands, this customer represents a mid-sized organization.

A model for upgrading
This customer's test plans called for a staging model. Taking daily builds of Domino 6, the system administrators installed the builds on their staging servers, and then both administrators and application developers began testing of the organization's mission critical applications. If the daily builds pass the tests, they are deployed to the production environment. To follow this model, then, you should include copies or replicas of mission critical applications for testing in your test environment. You can then make sure that the applications continue to work as they have in your production environment for your users, application developers, and administrators.

Lotus recommends that when you upgrade your servers, you should upgrade your Domino administration servers first. This customer began with its administration servers for two reasons:
Also, Lotus recommends that you upgrade the Domino Directory before you upgrade your servers. You have two options for upgrading the directory:
If you upgrade the Domino Administration server first, make sure to upgrade the Domino Directory immediately afterwards. When you restart the server after upgrading, Domino 6 will prompt you to upgrade the directory design.

Initially, customer 3 wanted to isolate the Domino 6 Directory design, but eventually, they replicated the design to other servers in the domain. While the design is backward compatible, this customer found that the appearance of the Person documents is altered when viewed in a Notes 4.6 client. Certain new design elements, such as tabbed tables, are not supported by Notes 4.x client releases, so when you work with the Domino 6 Directory, you should use the Domino 6 Administrator client. If you have more than one directory design in production, you can use the Domino 6 Administrator client for all directory releases.

Native network compression
Native network compression is new to Notes/Domino 6 and can eliminate the need for third-party network compression software. Native network compression compresses data sent between two Domino servers or between a Notes client and Domino server—as long as that data is uncompressed. You can enable native network compression on X.PC ports. For server-to-server network compression to work, both server ports must be enabled.

See the LDD Today Performance Perspectives column "Network compression in Domino 6" for more information about network compression.

Customer 3 tested native network compression with its mobile sales force and found that native network compression significantly reduced replication time for its employees. The greatest benefits of native network compression are experienced with dial-up connections. On a LAN with high-speed connections (T1, T3, and so on), you won't experience a noticeable difference.

Native network compression between clients and servers requires the Notes 6 client, so make sure to upgrade all clients to Notes 6 before you implement this feature. Notes 5 clients do not support native network compression and so will not benefit from having it enabled on a Domino 6 server. However, if you have both Notes 6 and Notes 5 clients connecting to a server, enable native network compression for your Notes 6 users. Unlike LZ1 compression for file attachments, enabling this feature for clients that do not support it has no impact on server performance. (The next section discusses how LZ1 compression can affect server performance in a mixed-release environment.)

You can enable the ports on the Notes 6 client using settings or desktop policies. Users can also enable their ports through the User Preferences dialog box:

User Preferences - Ports dialog box


File attachment compression
Native network compression does not compress already compressed file attachments. Notes/Domino 6 uses LZ1 compression for file attachments, which replaces Huffman compression in Notes/Domino 5. To take advantage of LZ1 compression for file attachments, you need a Notes/Domino 6 environment. You also need to compact your databases to upgrade them to ODS43, which is required to implement LZ1 compression. In a mixed-release environment with LZ1 compression enabled, you may experience server performance degradation. For instance, to serve file attachments for a Notes 5 client, the Domino 6 server must revert to Huffman compression, which uses CPU resources.

Turn on LZ1 compression if only Notes 6 clients connect with the Domino 6 server. If you have both Notes 5 and Notes 6 clients connecting with a Domino 6 server, wait until all clients are upgraded to Notes 6 before implementing this feature.

Application upgrades
Customer 3 has a large number of custom applications, so their largest concern was upgrading their applications. Fortunately, upgrading applications from a previous release to Notes/Domino 6 is relatively easy. For custom applications, make sure to test your applications in a Notes/Domino 6 environment. Then upgrade them to the new ODS.

If you find that a custom application does not work properly in Notes/Domino 6, it may be due to changes in programmatic features. See the LDD Today articles "Enhancements to the formula language in Notes/Domino 6" and "Decoding the new Notes/Domino 6 agent features" as well as the "What's new" section of the Domino 6 Designer Help for information on changes to the Notes formula language, LotusScript, and so on.

If you've coded special views or used undocumented features in the past, you may experience problems when upgrading. For instance, avoid referencing a $ view in a template because these views may change—and in past releases, they have changed due to a new feature being added.

For standard Notes applications, like the Domino Directory, you can apply the Note/Domino 6 template, and then compact the databases. Some Notes 6 system templates are backward compatible, some have only limited backward compatibility with earlier clients, and some are not backward compatible at all. To find out which system templates are backward compatible, see the Domino 6 Administrator Help.

Customer 4

Industry:Financial
Number of Notes clients:23,000
Notes client versions:4.6.x and 5.0.x
Number of Domino servers:100+

Customer 4 participated not only in the Notes/Domino 6 Early Deployment program but also in the iNotes Web Access beta program. Along with 23,000 Notes users, this customer has 1,650 iNotes Web Access users. To date, customer 4 has 19 Domino 6 servers in production:
While there are only about 90 Notes 6 client and mail templates deployed currently, more than 6,300 users in the organization use the Domino 6 servers. This customer routes all inbound Internet mail—approximately 80,000 to 100,000 messages a day—through its Domino 6 SMTP servers.

Clustering
This customer clustered its Domino 6 mails servers with Domino 5 mail servers, which raises the question, "Should users continue using a Notes 5 mail template until a cluster is upgraded to Domino 6?" While there are no issues reported with a Notes 6 mail template on a Domino 5 server, this customer was precautious and decided to retain the Notes 5 mail template as a fall back, in case problems arose with the Notes 6 mail template. Customer 4 used the Notes 5 mail template with the Domino 5 mail server and the Notes 6 mail template with the Domino 6 mail server.

You cannot use selective replication to prevent these mail templates from exchanging design elements because this type of replication is not supported in a clustered environment. To control replication, the customer used the database ACLs. By specifying each server—the Domino 6 and Domino 5 servers—in the ACL of the mail database, the customer could prevent replication of the design elements by designating roles for the servers.

Eventually, the customer allowed the Notes 6 mail template to overwrite the Notes 5 template, and no problems occurred on either the Domino 6 or Domino 5 mail server. This indicates that if you cluster mixed-release servers, you can upgrade your templates to the Notes/Domino 6 design. The Domino 5 server ignores any design elements that it does not support.

Domino 6 Directory and the UNK table
While the Domino 6 Directory template is backward compatible, it does contain more fields than previous directory templates. This increases the size of the UniqueNameKey (UNK) table. Notes/Domino 5 and 6 can support this size increase if you select the Domino Directory Database properties option "Allow more fields in database."

Database Properties box


However, Domino 4.5.x and 4.6.x servers—prior to release 4.6.7a—do not support this option. In release 4.x, the fields in a database cannot exceed 64 KB. If you have Domino 4.5.x or 4.6.x servers in production, upgrade them to the maintenance release 4.6.7a before you replicate the Domino 6 Directory template to those servers. Release 4.6.7a resolves the UNK table issue by providing the Notes.ini setting CONVERT_NAB_UNKS_TO_RARELYUSED=1. Make sure to enable this setting after upgrading to 4.6.7a, but before you replicate the Domino Directory to those servers. In the Notes.ini file, set the value of this setting to 1 to enable the feature.

If you do not upgrade your servers to release 4.6.7a, there are three options that may help avoid the UNK table issue; however, none of the following options is guaranteed and upgrading to 4.6.7a is still your best option.
Policies contain a number of new fields that can be avoided if necessary. If you implemented Setup Profiles in previous releases of Notes/Domino, you can continue using those documents until you have an environment that supports databases with UNK tables that exceed 64 KB.

ODS conversion
Heavy server utilization prevented customer 4 from compacting its databases right away. Remember that while the server is compacting databases, users experience a slowdown. Currently, there are no metrics available to help you determine how long it will take to compact your databases, so your best option is to compact various databases in your test environment and to time how long it takes to compact them. Then you can schedule your database ODS upgrades accordingly.

You do not have to upgrade your database formats to the new ODS immediately after upgrading your servers. However, the following Notes/Domino 6 features require that databases be upgraded to ODS43 for the feature to work:
(For more about the new ODS and other database changes, see the LDD Today interview, "What's in store for the Domino 6 database.")

Domino server administration tools
With previous versions of Notes/Domino, customer 4 installed the Domino Administrator client on the Domino server. This configuration, although supported, is discouraged primarily because the Domino Administrator client uses intensive server resources for monitoring and administration and so reduces server performance. The new administrative tools and features in Domino 6 make it unnecessary to install the client on the server. To eliminate the need to install the client and server on the same machine, customer 4 takes advantage of the administrator access rights, the Server Controller and Domino Console, and the Web Administrator client to administer their servers remotely.

Administrator access rights are specified in the Server document and include the following levels of access:
Any person or group listed in the Full access administrator or Administrator fields of the Server document is granted Manager access to the Web Administrator client (webadmin.nsf) by default. No other administrator access level is granted automatic access to the Web Administrator client.

The Server Controller is a Java application, and the Domino Console is a Java console that can connect with the Server Controller to send commands to the Domino server. The Domino Console has the functionality of the Domino server console. Full remote console administrators and View-only administrators can use a remote console to access a server, but System administrators and Restricted system administrators can only access servers running the Server Controller. In addition to the Domino Console, both the Domino Administrator client and Web Administrator client can connect with the Server Controller.

For more information about the full access administrator feature, the Server Controller and Domino Console, and the Web Administrator client, see the LDD Today article "Domino 6 Technical Overview" and the LDD Today interview "Mallareddy Karra on the Domino Console."

Customer 5

Industry:Web site
Number of Notes clients:20+
Notes client versions:Notes 6 Release candidate
Number of Domino servers:25

Can you guess who this customer is? The Lotus Developer Domain began participation in the Notes/Domino 6 early deployment program in January of this year when we deployed Beta 1 to two of our servers in our Web domain. Since then, we've upgraded 24 of our 25 production servers to Domino 6, and all of our clients are Notes 6 clients. For information about our server platforms and hardware, see the LDD site news article "Lotus Developer Domain Powered by."

We allotted 16 hours to upgrade one Web server. Our process included upgrading the server software as well as compacting the databases. We have approximately 60 databases on each Web server, varying in size from a little over one megabyte to more than ten gigabytes. So what did we learn from our early deployment experience?

Internet site documents
We implemented Internet site documents to manage the Web site. Internet site documents configure the server's Internet protocols. We use Internet site documents to configure our Web servers, but we still use Server documents to manage our mail servers. Initially, we had created a Web (HTTP) Site document. What we didn't know at that time was that when you implement Internet site documents, you can't implement one document or only the documents that you want to use. You need to implement Internet site documents for all protocols. So we continued to use the Server document to specify our SMTP settings for our Web servers. Soon we weren't routing Internet mail properly to our mail servers. After creating an SMTP Inbound Internet site document, however, we were back on track and answering Webmaster mail again.

The Internet site documents support the following protocols:
Internet site documents don't replace all Server documents. You still need Server documents to configure your TCP/IP and SSL ports.

Review the documentation for Internet site documents carefully before you implement them. If you're a Service Provider, want to use WebDAV, or use a certificate revocation list (CRL) for SSL, then the Internet site documents are required. If you have a Domino powered Web site, then these documents are recommended; but if you are an enterprise customer, implementing these documents is optional. For more information, see the LDD Today article "Building Web applications in Domino 6: A tutorial on Web site addressing," and the Domino 6 Administrator Help.

Fault recovery
Fault recovery is a new Domino 6 server feature that restarts the server automatically after a server failure to ensure server availability. We implemented fault recovery on some of our servers, but not on our Web servers, most of which run production builds of Domino 6. Our reasons for not implementing fault recovery are:
If your environment is similar to ours (high traffic, clustered servers that use an IP sprayer for load-balancing), then you may want to consider whether or not to implement fault recovery. While the feature makes the servers more reliable, it will take some time after restart—how long depends on the amount of content—to synchronize the content lost during the server failure. Whether or not servers are clustered, you may want to implement fault recovery.

View logging
View logging is another new feature for Notes/Domino 6. View logging is transaction logging for database views and folders. With view logging enabled, recovery of data for a view or folder is much faster. However, despite the obvious benefits of this feature, we chose not to implement it. View logging requires you to expose hidden design elements, and our databases use many hidden design elements. Unhiding the design elements, selecting the views to log, hiding the design elements again, and then applying the template can be a tedious task. View logging certain Domino Directory views can be worthwhile though. When you implement view logging, remember that you must expose hidden design elements.

Resource database
We reuse many design elements, such as graphics and HTML files, on the Lotus Developer Domain. In the past, we stored each design element in the individual databases. With Notes/Domino 6, we can use a resource database to store our shared resources, saving us both space and time. When we need to change a design element, we make the change once to the resource stored in the database, rather than to the individual databases. When you work with a database linking to shared resources, you need to bookmark the resource database on your local workstation. Otherwise, when you open the database linking to shared resources without the resource database, the shared resources do not appear.

Links to shared resources are relative rather than absolute, so if you have database replicas that link to the resource database, make sure that you have replicas of the resource databases residing on the same servers. On LDD, each Web server has a replica of the resource database.

Shared resources are used in a Notes/Domino 6 environment only. To access a resource database, you need a Domino Designer 6 client.

Key conclusions
The Notes/Domino 6 early deployment program has uncovered thousands of software bugs, the majority of which have been resolved for the 6.0 release. This program has benefitted Lotus, the Early Deployment program participants, and you—our customers. To sum up the lessons learned from early deployment:
The Notes/Domino 6 Early Deployment program has been so successful that Lotus has extended it to its other products, including QuickPlace and Sametime. If you are an enterprise customer and are interested in participating in an Early Deployment program, contact your Lotus Support Manager (LSM) for more information.