Prior to the explosive growth of the Internet, electronic mail (e-mail) was something that only large or geographically dispersed organizations needed. Companies with 10, 50, or even 100 people rarely needed e-mail. Overnight mail, faxes, or inter-office memos were perfectly adequate before everyone started racing to "get connected." Today, however, thanks to the multitude of Internet Service Providers (ISPs), nearly every company, regardless of size, can send and receive e-mail anywhere in the world.

Ever since Notes Release 3, Lotus has been offering customers the capability of sending and receiving e-mail across the Internet through the use of the Simple Mail Transfer Protocol (SMTP) Gateway (a Notes add-in product). When the SMTP Gateway first became available, connections to the Internet for message transfer were expensive and required additional technical resources to maintain. At that time, only large organizations could afford such a luxury. The "Web" still wasn't very "World Wide," and ".com" looked like a typo. Most companies just couldn't justify the cost of an Internet connection, especially those that didn't even have an internal e-mail system.

Today, attitudes toward e-mail have changed, and Internet e-mail capability is no longer perceived as a luxury, but rather as an absolute necessity. Lotus has kept pace with these changing attitudes and introduced the SMTP Message Transfer Agent (MTA), corresponding with Notes Release 4.x, as the next-generation Notes-to-Internet e-mail connectivity add-in.

Of course, if you're a Domino/Notes administrator, none of what I've said so far is anything new. You've probably already installed and configured the SMTP MTA for your environment, and more than likely, you've been using the MTA for quite awhile. You're not terribly interested in reading another article about the amazing growth of the Internet and the power of the SMTP MTA (But if you do want an introductory look at the SMTP MTA, see the Iris Today article "Internet messaging with the SMTP MTA"). What you'd probably like to read is something that takes a close look at the individual MTA transport tasks. Then you might want to read about how these tasks work together to send and receive e-mail. Finally, an article with a couple of hands-on exercises might help to clarify these explanations. Well, I'm happy to say that you've come to the right place. So, without further delay, let's get started with the first of a three-part series and put the SMTP MTA Under the Microscope.

The transport tasks: Getting mail from here to there
When it comes to revealing how the SMTP MTA works, the best place to start is with message transport services. This part of the MTA is responsible for sending and receiving Internet e-mail based on SMTP standards defined in the Request for Comment 821 document (RFC 821). The RFC 821 document defines a series of commands and responses for message exchange between two host computer systems. The SMTP MTA adheres to the standards defined in this document with the help of four primary server tasks: the Inbound Session Controller, the Outbound Session Controller, the Inbound Session Handler, and the Outbound Session Handler.

The Inbound Session Controller
This task is started by the SMTP add-in task when the MTA loads. The Inbound Session Controller appears on the server console as SMTPMTA isesctl Listening on SMTP port 25. This controller listens for connection requests on TCP port 25. When a connection request is made, the controller checks an internal list for available session handlers, transfers the connection to such a handler, disconnects, and returns to its listening state. If no handler is available, the controller spawns a new handler, provided the maximum number of inbound handlers has not been reached. If the maximum number of inbound handlers has been reached (as defined in the Internet Message Transfer Agent section of the Server document), the controller stops listening on TCP port 25. This state of "not listening" is very infrequent; however, when it does occur, the controller relies on the standard TCP retry time-out to ensure that the request is eventually handled.

Note: Some IP stack implementations queue a certain amount of session requests when all session handlers are busy. In such cases, all session handlers and the session queue would have to be busy or full before a connection fails.

The Outbound Session Controller
This task is also started by the SMTP add-in task during the SMTP initialization and appears on the server console as SMTPMTA osesctl. This controller waits for notification that there are new messages to send from the Outbound Message Converter (the task that converts outbound messages from Notes format to RFC-standard Internet format). When the controller receives such notification, it checks an internal list for available outbound session handlers and passes the available handler a reference ID to the pending message. If there are no available handlers, the controller spawns a new handler, provided the maximum number of handlers has not been reached. If the maximum number of outbound handlers has been reached (as defined in the Internet Message Transfer Agent section of the Server document), the controller keeps checking its internal list of handlers until a handler becomes available and then passes the handler a reference ID to the pending message.

Note: As a secondary method for locating new messages to be sent, the Outbound Session Controller routinely polls the Outbound Work Queue for new messages.

The Inbound Session Handler
This task is spawned by the Inbound Session Controller on an as-needed basis and appears on the server console as SMTPMTA iseshlr0. The handler "speaks" SMTP (as defined by RFC 821) with connected hosts that have been passed to the handler by the session controller. Information received during an SMTP session is placed into a document in the Inbound Work Queue (the Notes database that temporarily stores messages that need conversion to Notes format) for processing by the conversion task. When the message is complete, the handler checks the state of the Inbound Message Conversion task (which converts messages from RFC-standard Internet format to Notes format). If the task is idle, the handler "kick starts" the task through Inter-Process Communication (IPC). The handler then updates its state information in the internal list of inbound handlers and returns to an available state.

Outbound Session Handler
This task is spawned by the Outbound Session Controller on an as-needed basis and appears on the server console as SMTPMTA oseshlr0. The handler transfers messages to destination host systems. As new messages appear in the Outbound Work Queue (the Notes database that temporarily stores messages ready for transmission), the Outbound Session Controller passes a message reference ID to an available, or newly spawned, session handler. The handler is responsible for domain name resolution and makes the connection with the destination host system of the message recipient. Once a SMTP connection has been established, the message is transferred according to SMTP commands defined in RFC 821. When the handler has completed a session, it updates the status of the message in the work queue, updates its own state in the internal list of handlers, and returns to an available state.

Note: oseshlr0 and iseshlr0 represent the first handlers spawned by their respective controllers. As more handlers are spawned, the number appended to the end of the task name increases. For example, the second Outbound Session Handler spawned appears on the server console as SMTPMTA oseshlr1.

The Message Transport Process: A group effort
The previously defined server tasks help provide a functional understanding of the individual components involved in message transport. However, message transport isn't an individual activity, but rather a group effort. So, to fully understand how the MTA handles message transport, we need to look at the overall process of message transport and not just the individual pieces.

Outbound messages
Each outbound message begins the journey to the world outside of Notes when the message arrives in the Outbound Work Queue. This queue is a Notes database that accepts converted messages and holds them for processing by the Outbound Session Controller and individual session handlers.

Similar to any other Notes database, you can add the work queue to your workspace, open it, and examine its contents. Although the work queue appears to you as a database containing individual messages, the Outbound Session Controller doesn't view the contents of this database in that manner.

The Outbound Work Queue sorts and groups messages according to their "Next Hop Entity." Depending on how the MTA is configured, this next hop might be the recipient's domain or a relay host. In either case, this process is referred to as Domain List Processing, and it helps make efficient use of the connections established by session handlers by allowing messages headed to the same destination to be delivered by a single handler.

For example, 10 converted messages are placed in the Outbound Work Queue, with five of these messages destined for users at ibm.com, and the other five destined for users at lotus.com. Because of Domain List Processing, when the controller polls the work queue, the controller sees two lists of messages. The controller then takes the first list of messages and checks an internal table of Outbound Session Handlers. Assuming a handler is available, the controller passes the handler an ID to the list of messages in the queue. The controller then takes the next list of messages and repeats the process.



Each of the session handlers looks up (resolves) the domain name for the list of messages and opens a TCP connection on port 25 with the next hop host. Once the connection is established, the handler begins to transfer each message one at a time to the destination, using RFC 821 SMTP commands. After a message is transferred successfully, the handler updates the status of the message in the work queue and moves on to the next message in the list. When each message in the list is transferred, the handler updates its state in the internal table of session handlers and returns to a state of available.

Domain List Processing doesn't concern itself with how many messages are destined for a particular domain. Messages are simply grouped together and transferred by a single handler. There could be one message going to lotus.com and 100 messages going to ibm.com; the controller, however, will still see two lists of messages, each needing one handler.

Domain List Processing in the real world
Domain List Processing, at first glance, appears simple and sensible. However, Internet e-mail isn't a one-time event. Messages are constantly arriving in the work queue and the controller is periodically polling the queue for new messages. Some of those new messages are obviously destined for domains that may already have active handlers open. How are these messages handled, and how does Domain List Processing work in these situations?

To further understand outbound message handling, let's consider the 10-outbound-message example again. Remember that five messages are destined for ibm.com, and five messages are destined for lotus.com. The controller passes a message list reference ID to an available session handler. These handlers then look up the domain name for their respective list of messages, open a TCP session on port 25 with the destination host, and begin transferring the list of messages. In the meantime, six new messages arrive in the work queue. What happens then?

Because the six new messages are in the queue, Domain List Processing has already sorted them into two message lists: three messages destined for lotus.com, and the other three destined for iris.com. When the controller polls the work queue, 10 messages are in a state of Locked for Transmission (WQ_MSG_LOCKED_FOR_XMIT), and the six new messages are in a state of Pending (WQ_PENDING_XMIT). These six new messages are sorted into two groups. Next, the controller looks for an available session handler in its internal table. If a handler is available, the reference ID is sent to the handler. If not, the controller frequently checks the internal list until a handler becomes available and then passes a reference ID to the handler. This process is repeated for the other list of new messages.

Since three of the new messages were destined for lotus.com, and a handler was already sending messages to that domain, why doesn't the controller add these three new messages to that active list? Once messages are sorted into domain destinations, a reference ID is assigned to that list of messages, and Domain List Processing is not performed on those messages again. The controller processes message lists based on their reference ID and isn't concerned with whether there is an active handler already connected to the destination domain for the message associated with that ID. Furthermore, session handlers accept their message list assignments from the controller and don't get involved with domain similarities between active sessions and pending messages.

Closing a TCP connection when there are more messages destined for that domain might appear like a foolish action. However, in order for all outbound messages to get delivered, closing sessions only to reopen them again is necessary.

For example, consider a company that sends a tremendous amount of e-mail to one particular domain. If session handlers were allowed to remain connected to a particular host for as long as there were pending messages for that domain, and the maximum number of outbound sessions was set to 1, then there is a high possibility that the session would never close. If the session never closes, then the handler never becomes available to transfer messages to other domains.

This implementation of outbound message handling is a very efficient design. By using Domain List Processing, individual session handlers can deliver several messages to the same destination domain during a single session. At the same time, sessions are not infinite, and therefore pending messages to other domains will eventually have the opportunity to reach their destination as well.

Note: Even though a session handler deals with a list of messages, each message is still managed on an individual basis. If one message in a list fails, that does not mean all remaining messages in the list also fail.

Inbound messages
If you have a firm understanding of the outbound message process, you should have no problem understanding the inbound message process. For one thing, there is no Domain List Processing that takes place. Messages are handled on an individual basis by individual handlers.

The inbound message process starts with the Inbound Session Controller sitting on TCP port 25 waiting for connection requests. Once the controller receives a request, the internal table of session handlers is checked to find an available handler. If a handler is found, the connection is passed to the Inbound Session Handler, which receives the message and places it in the Inbound Work Queue.



This queue is similar to the Outbound Work Queue in that it is a Notes database that can be added to your workspace, opened, and contents viewed. The Inbound Conversion Task processes messages in the queue until there are no more messages. If the conversion task isn't converting messages when a new one is placed in the queue, the Inbound Session Handler will notify the conversion it has a message waiting, thereby kicking off the conversion process.

After the controller has passed a connection to a handler, the controller returns to TCP port 25 to listen for more connection requests. If, however, a handler is not available and the maximum number of Inbound Session Handlers has not been reached, the controller spawns a new Inbound Session Handler, passes the connection to this handler, and returns to TCP port 25 to listen for more connection requests.

The controller in this SMTP implementation is designed to be highly responsive to incoming connection requests. Unfortunately, there are situations in which all the Inbound Session Handlers are busy and the controller doesn't have a handler to pass a connection to. The controller handles this situation by monitoring its internal table session handlers. When all of the handlers become busy, the controller stops listening on TCP port 25 until an inbound session handler updates its status in the internal table as available. During this period when the controller isn't listening, new connection requests are not serviced; however, the requesting system has a retry period and therefore will eventually be serviced when the controller starts listening again.

Hands-on work with the SMTP MTA
Technical explanations and pretty-looking diagrams can help illustrate a complex technical topic. However, the best way to completely understand a topic such as the SMTP MTA is to do a little hands-on work with the feature itself. The exercises below are designed to help reinforce the concepts presented in this article.

Hands-on environment setup
To complete the exercises, you'll need the following:

• Two computers (one for the Domino server and one for the Notes client)
• Domino server 4.x (installed and configured with TCP/IP)
• SMTP MTA 1.x (installed and configured for manual loading, that is, do not let the Domino server automatically start the MTA)
• A Notes client (installed and configured for a registered Notes user, and communicating with your Domino server via TCP/IP)
• At least one valid Internet e-mail address to which you can send test messages

1. Start your Domino server.
2. Once all of the initial server tasks are started, start the SMTP MTA by entering "LOAD SMTPMTA" at the server console prompt. The following tasks will start:
   
   -- SMTPMTA: The MTA add-in task (communicates "TELL" commands to the various MTA tasks and manages SMTP.BOX and message queue compacting)
   -- omsgcnv: The Outbound Message Conversion task (converts outbound messages from Notes format to RFC-standard Internet format)
   -- osesctl: The Outbound Session Controller task (defined earlier in this article)
   -- isesctl: The Inbound Session Controller task (defined earlier in this article)
   -- imsgcnv: The Inbound Message Conversion task (converts inbound messages from RFC-standard Internet format to Notes format)
   -- drt: The Delivery Report Task (gathers and reports message statistics)
3. To verify that the six primary MTA tasks are started, enter "SHOW TASKS" at the server console. Notice that the Inbound and Outbound Session Handlers do not appear. Remember, these handlers are managed by the session controllers on an as-needed basis.
4. To get a handler such as the Outbound Session Handler to start, create a simple test message addressed to someone on the Internet, click the Send button, and watch the server console.
   
   Once your test message is sent, the workstation's MAILER (a core component of the Notes workstation) places this message in the MAIL.BOX on your Domino server. The router then picks up the message, and since it's destined for the Internet, the router places the message in the SMTP.BOX. At this point, the Outbound Message Conversion task converts the message from Notes format to RFC-standard Internet format and places the converted message in the Outbound Work Queue. The Outbound Session Controller now takes over and checks for an available Outbound Session Handler. Since we just started the MTA, no handlers are available. Therefore, the controller spawns a new handler and passes it a reference to the pending message. Using RFC 821 SMTP commands, the handler establishes an SMTP session with the destination host and begins transferring the message.
5. To verify that the Outbound Session Handler has started, enter "SHOW TASKS" at the server console. Notice that an extra SMTP task is listed: oseshlr0. Unlike the other tasks, however, this newly listed task (oseshlr0) can't be stopped using a TELL command (try it).
6. To stop the session handler task, stop its controller by entering "TELL SMTPMTA STOP OUTBOUND TRANSPORT" at the server console. This command stops not only the Outbound Session Controller, but also the idle Outbound Session Handler that the controller previously spawned.
7. Once these tasks are completely shut down, enter "TELL SMTPMTA START OUTBOUND TRANSPORT" at the server console. The Outbound Session Controller starts up again, but the previously stopped handler does not.

1. To get started, make sure your Domino server is running and that the MTA is loaded and all its tasks are started. You can verify the status of MTA tasks by entering "SHOW TASKS" at the server console. All of the tasks listed in Exercise #1 should be listed on the screen. If not, restart the MTA by entering at the server console "TELL SMTPMTA QUIT" and then "LOAD SMTPMTA."
2. Once the SMTP MTA is running again, open a telnet session with the MTA server. For example, if you're using Windows 95, click the Start button, choose Run, enter Telnet, and click ENTER.
3. After the telnet application starts, choose Terminal - Preferences. In the dialog box, select the Local Echo option so you can see what you type on the screen as you type it. Click OK. The setting is saved, and the dialog box disappears.
4. To start a telnet session with the MTA, choose Connect - Remote System. In the Connect dialog box, enter the host name or IP address of your MTA machine in the Host Name field, and enter 25 in the Port field. Leave the Term Type field set to VT100, and click Connect. If you've entered all of the correct information, the MTA machine sends back the following response:
   
   220 your host name here Lotus SMTP MTA Service Ready
   
5. At this point, look at the server console of your MTA machine. A new Inbound Session Handler has been spawned: iseshlr0.
6. Complete this session with the MTA by sending a simple message to a valid Notes user. The following commands shown in bold are what you need to type, and the other text shows the responses from the MTA. Keep in mind, if you mis-type something, the Backspace key won't work. You're better off closing the session and starting over.
   
   Note: Substitute the machine and user name used in this example with valid names for your environment.
   
   220 enaic.lotus.com Lotus SMTP MTA Service Ready
   HELO univac.com
   250 enaic.lotus.com
   MAIL FROM:<bsmith@univac.com>
   250 OK
   RCPT TO:<jdoe@eniac.lotus.com>
   250 OK
   DATA
   354 Enter Mail, end by a line with only '.'
   (Press ENTER)
   Hello World
   .
   250 Message received OK.
   QUIT
   221 Goodbye

1. To get started, make sure your Domino server is running and that the MTA is loaded and all its tasks are started. You can verify the status of MTA tasks by entering "SHOW TASKS" at the server console. All of the tasks listed in Exercise #1 should be listed on the screen. If not, restart the MTA by entering at the server console "TELL SMTPMTA QUIT" and then "LOAD SMTPMTA."
2. From your Notes client, prepare two messages with the same Internet destination address. In the first message, include a large attachment (greater than or equal to 1 MB). For the second message, simply enter "Hello" on the subject line.
3. Send the first message and watch the server console on the MTA machine. Once the Outbound Session Controller spawns a new handler and a connection is made with the destination host, send the second message.