How to Administer a Domino Cluster

Creating a Domino Cluster

Adding Servers to a Cluster

Configuring Failover

Configuring Load Balancing

Replicating Databases within a Cluster

Managing Databases within a Cluster

Analyzing Cluster Configuration

Monitoring Cluster Performance and Activity

Managing Servers in a Cluster

What is a Domino Cluster?

Domino clusters
A Domino cluster is a group of two to six servers that work together as a unit to meet capacity and availability goals that exceed the capacity of a single machine. Clusters provide the level of service needed by large organizations, using replicated databases to achieve high availability. If a server is down, databases are still available on other servers in the cluster.

Clustered servers redirect a client request to another available server in the cluster under the following conditions:

Note: Only the Lotus Domino Enterprise Server License includes the ability to create Domino clusters.

For additional information on requirements, see Clustering requirements in the Domino 5 Administration Help database.

Planning a Domino cluster
For information on the issues related to planning a Domino cluster, see Planning a cluster and related documents in the Domino 5 Administration Help database.

Back to Table of Contents

Other uses for Domino clusters
In addition to the features covered in this offering, the following Domino features exploit all of the advantages of Domino clusters.

Domino feature	Documentation Reference
Directory Assistance	Using Directory Assistance in a cluster
Calendaring and Scheduling	Calendars and scheduling and How calendars work in a cluster
Server Web Navigator	Using the Server Web Navigator in a cluster
Domino Web Server	Using clusters over the Internet
Microsoft Cluster Server	Running Domino with Microsoft Cluster Server
Shared Mail	Setting up mail in a cluster
Mobile Access	Setting up cluster access for mobile users

Back to Table of Contents

How to Administer a Domino Cluster

Procedure: Set up a Domino cluster
Complete these tasks to set up a Domino cluster.

	Task	Procedure
o	1	Create a cluster.
o	2	Add other servers to the cluster.
o	3	Configure failover. • Create database replicas on clustered servers to enable automatic database failover. • Restrict the server in order to force failover to occur. • Enable failover for mail routing and delivery.
o	4	Configure load balancing. • View the current server availability index. • Set NOTES.INI variables that affect load balancing.
o	5	Replicate databases within a cluster. • Allow replication to occur automatically within a cluster. • Force replication of databases within a cluster. • Schedule standard replication between clustered servers using Connection documents. • Start multiple cluster replicators on clustered servers at server startup or manually.

Back to Table of Contents

Procedure: Manage and Monitor a Domino cluster
Complete these tasks to maintain the servers and databases within a Domino cluster.

	Task	Procedure
o	1	Manage databases within a cluster. • Disable cluster replication for a database. • Move a database from a cluster server. • Manage cluster database availability.
o	2	Manage servers in a cluster. • Move a server to a different cluster. • Remove a server from a cluster.
o	3	Analyze cluster configuration.
o	4	Monitor cluster performance and activity.

Back to Table of Contents

Creating a Domino Cluster

Procedure: Create a Domino Cluster
Follow these steps to create a Domino cluster.

Note: This procedure requires the following access:

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Configuration tab > Cluster section > All Server Documents view.
3	Select the first server to be placed in the cluster, and click Add to Cluster.
4	Click Yes to begin the process of adding the first server to the new cluster.
5	Select Create new cluster, and click OK.
6	Enter the new cluster name, and click OK.
7	Click one of the following options: • Yes to immediately update the Server document in the Domino Directory on the currently selected server. • No to submit a request to the Administration Requests database to have the Administration Process update the Server document on the designated administration server for the Domino Directory.
8	Click OK when presented with the confirmation message.

How Domino sets up the new cluster
The Domino server completes these tasks to set up the new Domino cluster.

Step	Action
1	Once the Server document contains the cluster server name, and the first server in the new cluster receives the replicated Server document changes, the server starts the following tasks: Component Description Cluster Administrator (Cladmin) Responsible for the correct operation of all cluster components. The Cluster Administrator runs: • Automatically at server startup • Whenever the cluster membership changes Cluster Database Directory Manager (Cldbdir) Responsible for keeping the Cluster Database Directory (CLDBDIR.NSF) up-to-date with the most current database information and manages databases with cluster-specific attributes such as databases marked out of service or pending delete. Cluster Manager Tracks the state of all members in a cluster by: • Keeping a list of which servers in the cluster are currently available. • Maintaining information about the workload on each server. Cluster Replicator (Clrepl) Responsible for the tight synchronization of data among databases and their replicas in a cluster.
2	The Cluster Administrator adds the Cluster Database Directory Manager (Cldbdir) and the Cluster Replicator (Clrepl) tasks to the ServerTasks line in the NOTES.INI file.
3	The Cluster Database Directory Manager creates the Cluster Database Directory (CLDBDIR.NSF) and populates it with information for all of the databases on this server.
4	When additional servers are added to the cluster, the Cluster Replicator replicates the Cluster Database Directory with the most available server in the cluster.

Back to Table of Contents| Back to Set Up a Domino Cluster

Adding Servers to a Cluster

Procedure: Add servers to an existing Domino Cluster
After creating the Domino cluster, follow these steps to add servers to the existing cluster.

Note: This procedure requires the following access:

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Configuration tab > Cluster section > All Server Documents view.
3	Select one or more servers to be placed in the existing cluster, and click Add to Cluster.
4	Click Yes to begin the process of adding the selected servers to the existing cluster.
5	Select the cluster name from the list of existing Domino clusters, and click OK.
6	Click one of the following options: • Yes to immediately update the Server document in the Domino Directory on the currently selected server. • No to submit a request to the Administration Requests database to have the Administration Process update the Server document on the designated administration server for the Domino Directory.
7	Click OK when presented with the confirmation message.

Configuring Failover

When does failover occur?
Failover occurs automatically when a database is unavailable on one server and the Cluster Manager is able to redirect the Open Database request to another server in the cluster that stores a replica.

Failover usually occurs when a user tries to open a database that is marked:
• Out of service
• Pending deletion

Failover can also occur when the server is:
• Unreachable on the network (for example, shut down)
• Restricted
• Unavailable because it has reached the specified maximum number of users
• Busy

For a complete list of circumstances under which failover occurs, see When failover occurs and related documents in the Domino 5 Administration Help database.

Back to Set Up a Domino Cluster

Methods to manage failover
Use any of the following methods to set up database failover:

Back to Set Up a Domino Cluster

Procedure: Restrict the server in order to force failover to occur
Consider forcing failover to occur under the following circumstances:

Follow these steps to restrict the server.

Step	Action
1	From Domino Administrator, select the server to restrict.
2	Select the Server tab > Status tab.
3	Click the Console button > Live button to begin a live console session.
4	Enter the following console command, and press Enter: Set Config Server_Restricted=n where n is one of the following values: Value Description 0 Indicates that the server is not restricted. 1 Indicates that the server is restricted for the current server session only — new user requests will be redirected. 2 Indicates that the server is restricted for subsequent server sessions — new user requests will be redirected.

Back to Set Up a Domino Cluster

Procedure: Enable failover for mail routing and delivery
When a mail server is unavailable, Domino clusters support routing mail to a replica of the user mail files that are stored on another server within the cluster. Follow these steps to enable failover for mail files.

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Configuration tab > Messaging section > Configurations view.
3	Edit the Configuration document for the mail servers.
4	Select the Router/SMTP tab > Advanced tab > Controls tab.
5	In the Cluster failover field, select one of the following options: • Enabled for last hop only (Default) • Enabled for all transfers in this domain — useful when using hub servers in a cluster to route mail. Note: Selecting Disabled would prevent mail failover from occurring.
6	Click Save and Close.

Back to Table of Contents | Back to Set Up a Domino Cluster

Configuring Load Balancing

Server availability index
The server availability index is a number between 0 and 100 that indicates the current workload on the server. Each cluster server determines the current workload by measuring the average response time of the requests it has recently processed. An availability index of 0 indicates that the server is heavily loaded, whereas a server availability index of 100 indicates that the server is lightly loaded. As response time increases, the server availability index decreases.

For more information on how the server calculates the server availability index, see The server availability index in the Domino 5 Administration Help database.

Procedure: View the current server availability index
Before configuring load balancing, it is important to carefully monitor the server availability index. Follow these steps to view the current server availability index.

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Server tab > Statistics tab.
3	Select the Server section and note the Server Availability index value.

Back to Set Up a Domino Cluster

NOTES.INI variables that affect load balancing
To get the best performance from the cluster servers, carefully monitor the server availability index, then set the load balancing variables accordingly. The following table describes the NOTES.INI variables related to load balancing on cluster servers.

NOTES.INI variable	Description	Usage
Server_Availability_Threshold	Specifies the acceptable level of system resources available to the server – acceptable values are between 0 and 100. The higher the value, the less available the server is.	• 0 indicates that the server is fully available and workload balancing is disabled (default). • 100 indicates that the server is busy and will not accept any new sessions provided that there is another available server in the cluster to which the client can be routed.
Server_MaxUsers	Specifies the maximum number of users allowed to access a server. There is no default.	• 0 indicates unlimited number of users can access the server. • Any number indicates that when this number of users is reached, the server state becomes MaxUsers which can be viewed as restricted and new user requests will be redirected.

For more guidance on how to determine the appropriate server availability threshold, see Choosing a server availability threshold in the Domino 5 Administration Help database.

Back to Set Up a Domino Cluster

Procedure: Set NOTES.INI variables that affect load balancing
After creating replicas of databases on cluster servers, follow these steps to configure load balancing.

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Configuration tab > Messaging section > Configurations view.
3	Edit the Configuration document for the servers.
4	Select the NOTES.INI Settings tab.
5	Click Set/Modify Parameters.
6	Enter or select the appropriate NOTES.INI variable and value based on the Help description and the table above.
7	Click Save and Close.

Replicating Databases within a Cluster

Cluster Database Directory
The Cluster Database Directory (CLDBDIR.NSF) is a database that resides on each server in a cluster and contains information about all the databases within a cluster. Each server in the cluster stores a replica of the Cluster Database Directory. The Cluster Replicator keeps replicas on clustered servers synchronized.

Methods to replicate databases within a cluster
Use any of the following methods to replicate databases within a cluster:

Back to Set Up a Domino Cluster

Allow replication to occur automatically within a cluster
The Cluster Replicator server task (Clrepl) synchronizes the databases stored on cluster servers. When a change occurs in a database, the Cluster Replicator pushes the change to the replicas on other servers in the cluster. If the Cluster Replicator is busy, it stores the changes in memory, then replicates the changes when it is no longer busy. The Cluster Replicator uses the Cluster Database Directory (CLDBDIR.NSF) to determine which other servers in the cluster store replicas of the database that changed.

Note: The Cluster Replicator also replicates private folders in a database to the other replicas on cluster servers.

Back to Set Up a Domino Cluster

Procedure: Force replication of databases within a cluster
If the server shuts down before the Cluster Replicator can synchronize all cluster replicas, the changes stored in the server's memory will be lost. Follow these steps to force replication between clustered servers when the server restarts and distribute the changes saved to the database before the server shut down.

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Server tab > Status tab.
3	Choose Server > Replicate from the tools menu.
4	Enter the cluster name in the Which server do you want to replicate with? field.
5	Select Push for the replication style.
6	To select only a specific database to replicate, follow these steps: a. Select Selected database. b. Click Database. c. Select one or more databases from the list. d. Click OK.
7	Click Replicate.

Back to Set Up a Domino Cluster

Procedure: Schedule standard replication between clustered servers using Connection documents
Consider scheduling standard replication using Connection documents under the following circumstances:

• To replicate a cluster server with a server that is not a member of the cluster.
• To distribute changes saved to databases that were lost from server memory in the event that the server shut down efore the Cluster Replicator could replicate the changes.
• To replicate the following database elements that the Cluster Replicator does not support:
  • Processing selective replication formulas.
  • Recognizing the Replication Settings > Advanced panel.

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Configuration tab > Replication section > Connections view.
3	Click Add Connection.
4	On the Basics tab, select a Connection type.
5	Enter/verify that the Source server and Source Domain fields are correct.
6	Enter the Destination server or server group, and Destination domain.
7	Click Choose ports, select the ports to use to attempt this connection, and click OK.
8	On the Routing/Replication tab, enter information in the appropriate fields according to the descriptions below: Field Description Replication Task Set to Enabled. Replicate databases of ___ priority The priority of the databases to be replicated for this schedule. Replication Type The type of replication to be used for this schedule. The default is Pull-Push. Files/Directories to Replicate The specific databases or directories containing databases to replicate. A blank field results in all databases in common in the Domino\data directory structure replicate on this schedule. Replication Time Limit If this field has a value in it and the replication is not complete at the end of the specified time, or if the server crashes, then replication will begin where it left off once scheduled replication restarts.
9	On the Schedule tab, enter the information in the appropriate fields according to the descriptions below: Field Description Schedule Set to Enabled. Call at times Specifies either one discrete time, a list of times (each separated by a comma), or a time range. Repeat interval Specifies the frequency of calls over the time range. Days of week Specifies the days of the week that the schedule should run.
10	Click Save and Close.

Procedure: Start multiple cluster replicators at server startup
As with the Replicator (Replica), administrators can load multiple occurrences of the Cluster Replicator (Clrepl), monitor cluster replication events, and then consider enabling multiple replicators if replication events are occurring back-to-back throughout the day.

Follow these steps to start multiple cluster replicators automatically at server startup.

Step	Action
1	Open the NOTES.INI file in a text editor. The NOTES.INI file is located, by default, in the Domino program directory.
2	Search for the line beginning with ServerTasks.
3	Add an additional Clrepl to the ServerTasks line. For example, the ServerTasks line may contain the following: ServerTasks=HTTP,Replica,Router,Stats,Amgr,Adminp,Event,Cldbdir,Clrepl,Clrepl,...
4	Save the NOTES.INI file, and close the text editor.
5	Restart the server for the changes to take effect.

Back to Set Up a Domino Cluster

Procedure: Start multiple cluster replicators manually
If the cluster replicator is constantly busy during the current server session, follow these steps to start multiple cluster replicators manually.

Step	Action
1	From Domino Administrator, select the server on which to start additional cluster replicators.
2	Select the Server tab > Status tab.
3	Choose Task > Start from the tools menu.
4	Select Cluster Replicator from the list of tasks.
5	Click Start Task.

Back to Table of Contents | Back to Set Up a Domino Cluster

Managing Databases within a Cluster

Procedure: Disable cluster replication for a database
In a cluster, all databases are automatically set to replicate in real time so that replicas are always synchronized. However, real-time replication is needed only for databases that require constant synchronization of data.

Consider disabling cluster replication for a database under the following circumstances:

Follow these steps to disable cluster replication for a particular database.

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Configuration tab > Cluster section > Cluster Directory (R5) section > Databases by Server view.
3	Select the appropriate Servername section.
4	Select one or more databases for which to disable cluster replication from the list.
5	Click the Tools action button, then select Disable Cluster Replication on Selected Databases.

Note: Disabling cluster replication for a database does not affect scheduled replication, which still occurs.

Back to Manage and Monitor a Domino Cluster

Procedure: Move a database from a cluster server
Use the Database tools menu on the Files tab in Domino Administrator to move a database from one cluster server to another server. The Administration Process creates a replica on the destination server, then marks the database on the source server for deletion. Follow these steps to move a database from one cluster server to another cluster server.

Note: This procedure requires the following access:

• Manage Access with the Delete documents privilege in the source database.
• Create Replica access on the destination server.

Back to Manage and Monitor a Domino Cluster

Procedure: Manage cluster database availability
Follow these steps to manage cluster database availability using the Database Tools menu on the Files tab from Domino Administrator.

Step	Action
1	From Domino Administrator, select the server that stores the databases to manage.
2	Select the Files tab.
3	Select one or more databases to manage.
4	Choose Database > Cluster from the tools menu. The figure below shows the Manage Clusters dialog box:
5	Select the appropriate cluster availability status option from the table below: Option Description Out of service Indicates that users should not be allowed to open the database -- however, the server keeps all existing connections to the database and allows replication updates from other replicas to continue to occur. In service Indicates that the selected out-of-service database should be placed back into service. This restores access to the database, and makes it fully available to users. Pending delete Indicates that no new database sessions will be opened. Once every active user has closed the database, the changes will be pushed to another replica in the cluster, and the database will be deleted. Note: Each of these options results in the Cluster Database Directory Manager updating the availability of the database in the Cluster Database Directory.
6	Click OK.
7	Click OK to confirm the number of databases processed.

Analyzing Cluster Configuration

Procedure: Generate a cluster analysis report
Use the Cluster Analysis reporting tool to review cluster configuration and help determine the cause of server problems within the cluster. Follow these steps to generate a cluster analysis report.

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Server tab > Analysis tab.
3	Choose Analyze > Cluster from the tools menu.
4	Select the items to include in the report as shown in the figure below:
5	Click Results Database to change the default Cluster Analysis database name and location.
6	Select whether to Overwrite or Append the reports to the database.
7	Click OK. Result: Once the Cluster Analysis database is populated, the database will open in a window to allow viewing the documents in the database.

Monitoring Cluster Performance and Activity

Tools to monitor cluster performance and activity
Monitor cluster performance and activity using real-time server statistics and statistics reports that show:

Managing Servers in a Cluster

Procedure: Move a server to a different cluster
After determining that a server should belong to a different cluster, follow these steps to move the server to a new cluster.

Note: This procedure requires the following access:

For detailed information on how the Administration Process moves a server to a different cluster, see How Domino removes the clustered server and How Domino sets up the new cluster.

Back to Table of Contents | Back to Manage and Monitor a Domino Cluster

Remove a server from a cluster
After reconsidering the servers that belong to a cluster, follow these steps to remove a server from a cluster.

Note: This procedure requires the following access:

Step	Action
1	From Domino Administrator, select the server to administer.
2	Select the Configuration tab > Cluster section > Clusters view.
3	Select the server to remove from the cluster.
4	Click Remove from Cluster.
5	Click Yes to confirm removing the selected server from a cluster.
6	Choose one of the following options: • Yes to immediately update the Server document in the Domino Directory on the currently selected server. • No to submit a request to the Administration Requests database to have the Administration Process update the Server document on the designated administration server for the Domino Directory.
7	Click Yes when presented with the confirmation message.

Back to Table of Contents | Back to Manage and Monitor a Domino Cluster