"The best thing about standards," says the old joke, "is that there are so many of them."
This is certainly true of the Internet. One of the best things about the Internet is standards -- from the low-level protocols like IP and TCP, up to POP3, IMAP, LDAP, MIME, NNTP, HTTP, and HTML -- but there certainly are a lot of them.
The good news about these standards is that they have opened up client-server computing. It's now commonplace to have a choice of clients to use to access a server. For example, if you have a POP3-based mail account with an Internet Service Provider (ISP), you can use the mail client built into Notes, or Netscape Communicator, or Microsoft's Internet Explorer, or a stand-alone client like Eudora Pro, or one of literally dozens of other mail clients.
The bad news about Internet standards is that with so many of them, you can wind up with a small army of software applications that live on your hard disk. Each one is specialized in a particular protocol, and each one forces you to learn and use a different user interface. So, you must wade through mail clients, Web browsers, directory tools, news readers, FTP clients, and on and on.
The ideal would be a single, integrated client that would speak all the Internet protocols and let you do all of your work within one user interface. This client would be easy to learn and easy to use, with good tools for customization. It would do for the Internet protocols what the Notes client does. Then, why shouldn't it just be Notes?
The creators of Notes thought the same thing.
This article takes you down to the front lines to show you how the developers at Iris implemented Internet standards in Notes R5. First, we'll talk about the goals behind the release. Then, we'll look at each of the major standards, including POP3 and SMTP, IMAP, NNTP, HTML, MIME, and LDAP, and what implementing them in Notes means.
Note: For in-depth information on using the new Notes R5 features, see the Notes 5 Help.
The goals behind Notes R5
The announced goal for Notes R5 was to make it the best desktop client. But Mike Kudla wants to correct any misconceptions. "Notes doesn't have to become the best desktop client," says Kudla, the project leader for Notes R5. "It's always been the best desktop client." The reason, he says, is that its architecture and user interface have always been flexible enough, and extensible enough to accommodate new things.
"You can look at it the other way around," Kudla says. "What are the Internet and corporate intranets, in a way, but a standards-based implementation of Notes? Why shouldn't Notes be the best desktop client? It has years of head-start on the competition. And it already does things that the Internet doesn't do yet in any standard way, like calendar functions and replication." From this perspective, the job of Kudla's group of developers at Iris was not to make Notes behave like the Internet, but to make the Internet behave like Notes.
On first impression, says Kudla, R5 will look very different. "If you've seen the prototypes, the new user pane and features that make the user interface more Web-like, you've probably thought, 'Wow, this is really different.' And at the top level, it is. The Web, after all, has created some new paradigms of the way people interact with computers. But at the deeper levels of R5, it's still all Notes."
Adding or enhancing Notes support for Internet protocols yielded many new features in R5: native MIME support, world-class Web browsing, a Directory Catalog, and LDAP capabilities. The Notes client is a great integrator, says Kudla. "There's a great advantage for end-users. All these new features that use various data sources and protocols are presented in a consistent format. The IMAP Inbox looks just like the Notes Inbox, and NNTP discussions look just like Notes discussions."
The Internet: An alphabet soup of standards
The Internet standards incorporated into R5 are an alphabet soup of functions and formats. There seems to be a lot of them mostly because each one represents a solution to a particular problem -- how to send an e-mail message from a client to a server, for example. The next sections of this article dive down into the following standards, and what they mean in the new Notes R5 client:
- POP3 and SMTP (Post Office Protocol Version 3 and Simple Mail Transfer Protocol) -- the most widely used protocols for receiving and sending electronic mail on the Internet. POP3 allows you to access incoming mail, and SMTP allows you to send mail. The protocols are about as simple as they can be -- in general, you maintain your mail files locally, and the servers act only as temporary storage for unretrieved messages.
- IMAP (Internet Message Access Protocol) -- the "newer" mail protocol that supports more features than POP3; you can choose to maintain your mail file either locally or on the server, for example.
- NNTP (Network News Transfer Protocol) -- the standard behind Usenet newsgroups, the Internet's favorite discussion databases. Many companies use NNTP for internal discussion and notification databases, as well.
- HTML (Hypertext Markup Language) -- the standard for encoding information about how to display a Web page. An HTML parser is the central piece of a Web browser -- the engine that reads the HTML code; applies it to the text, graphics and program code that make up the page; and displays the result on the screen.
- MIME (Multipurpose Internet Mail Extensions) -- a "container" protocol that is used with e-mail, news, or Web pages to define types of message content and how they are encoded for transport. MIME types are registered with the client, and tell it what applications to open the content file in.
- LDAP (Lightweight Directory Access Protocol) -- the protocol that supports interaction with X.500 directories. The protocol is "lightweight" because it doesn't include the overhead for supporting some of the features of the X.500 standard.
In addition, Notes R5 supports Secure Sockets Layer (SSL) for secure communication using POP3, SMTP, IMAP, NNTP, and LDAP. You can enable SSL in the respective Internet accounts that you set up.
POP3 and SMTP
POP3 and SMTP are actually "old-timers" when it comes to Internet standards supported by Notes. In fact, Notes R4.6 was a client for both POP3 and SMTP. (For more information on the POP3 and SMTP support in R4.6, see the article "Notes 4.6: POP3 and SMTP Support.")
Even so, POP3 and SMTP lack some mail functions familiar to Notes users. POP3 is for client-based mail, which means that your mail file is stored on your local PC -- you don't have the option of a server-based mail file as you do in Notes. The POP3 client must connect to the server, determine whether any new messages are available, then download them and disconnect. (In R5, an entry for this one-way "pull" function for each source account appears on the Replicator page.)
POP3 and SMTP haven't changed much in R5, according to Jackie Ferguson, the developer who has worked on them since they were introduced into Notes. "The biggest changes are things they have in common with other protocols -- things like the inclusion of native MIME storage and configuration changes."
For Ferguson, the most important result of the configuration changes is the Universal Inbox. "Different people think a Universal Inbox means different things, but what it means in our context, is that you can have multiple sources being pulled into the same place." In R4.6, she says, you needed to configure your different e-mail accounts in separate Location documents, so you had to switch locations to check multiple POP3 mailboxes. In R5, the switching of locations goes away. Now, there's a new kind of document in your address book called an Account document. You set up a document for each mail account. You just specify the basics, such as your account server name (for example, pop.myisp.com), login name and password, and then the protocol to use (in this case, POP3). On the Protocol Configuration tab, you can then specify whether you want to leave mail on the server.
The last line of the Account document still asks you to select the locations in which the document will be active. Your location is important, because if you are a mobile user, your POP3 account may be accessible from some of the places you run Notes, but not others. For example, if your office intranet is behind a firewall, you may not be able to access your ISP's POP3 server. Likewise, if you're connected to the Internet, you may not be able to connect through the firewall to your Domino server on the intranet.
So, after you create the Account document, you may also need to create or edit a Location document. The Location document contains an "Internet mail address" field on the Basics tab, which is where you specify the e-mail address that displays on your mail when you send mail (for example, jjones@myisp.com). On the Mail tab, set the "Mail file location" to the location that will collect your POP3 mail (usually Local) and enter the name of your mail file (for example, mail\jjones.nsf -- This is a Notes mail file that can become a universal Inbox for mail from your POP3, IMAP, and Domino servers). Then, set the "Send outgoing mail" to "directly to the Internet" to send mail through your ISP's SMTP server. (For more information on setting up your POP3 mail, see the Notes 5 Help.)
Sending mail via SMTP
While you may want to collect messages from several mail accounts, sending mail is simpler. You don't need more than one outbound mail server, and you will choose one based on your location. So, as mentioned above, the Location document includes the "Send outgoing mail" field for you to specify whether your outgoing mail routes through your Domino server or directly to the Internet. You'd specify to send mail directly to the Internet if your Domino server isn't accessible from this location, such as when you're working from home, or if you're using your ISP's mail servers.
If you select to send mail directly to the Internet, you must also create an Account document in your address book for the outgoing SMTP mail server. Again, you simply enter your account server name (for example, smtp.myisp.com), and then the protocol to use. SMTP servers usually do not require a login name and password. (In R4.6, you needed to specify this information in each separate Location document.)
IMAP
IMAP is like POP3 in that it's strictly for accessing incoming mail, not sending mail. Domino included an IMAP server with R4.6. Notes R5 includes IMAP client functions that allow you to connect to multiple IMAP servers and mailboxes.
IMAP can work in three ways in Notes R5, according to Mike Palone, the principal software engineer who did much of the IMAP protocol implementation:
- Like POP3, it can work in "pull" mode (IMAP offline mode): You connect to the server, pull any new messages down to your client's local mail file, and then disconnect. This way, you can pull the messages into a universal Inbox containing mail from POP3, IMAP, and Domino servers.
- Like server-based Notes mail, it can work in connected mode (IMAP online mode): Your mail file is on the server, and you work with its contents interactively using a proxy database. You can read, modify, and delete messages; create and delete storage folders; and move messages among folders.
- Like client-based Notes mail, it can work in disconnected mode: You maintain a local copy of your mail file. When the IMAP server is accessible, you can synchronize your local copy with the server's copy, and then disconnect. You can work in your local file while disconnected, reading, modifying and deleting messages, and modifying the file's folder structure. The next time you synchronize with the server from the Notes Replicator page, your Notes client transmits the changes to the server and receives any changes made to the server-based mailboxes that occurred while you were disconnected.
This synchronization is done entirely by the client, Palone stresses. It's not functionality that is found in IMAP servers, but it's one of the places where Notes can lend some of its sophistication to an Internet protocol. And because it's done entirely by the client, it works with any IMAP server. "IMAP synchronization is not as rich as Notes replication, because there's only so much that the protocol supports," says Palone. There's no replication of agents or your local database design, but it does synchronize folders and messages.
To start working with your IMAP mail account, you again create an Account document in your address book. You again specify your account server name (for example, imap.myisp.com), login name and password, and then the protocol to use (either IMAP Offline or IMAP Online). If you select to use IMAP offline mode, your messages are copied into the Inbox of the mail file listed in your Location document -- the universal Inbox method. If you select to use IMAP online mode, you work directly with your mail on the server through a Notes proxy database that is automatically created for you. The Protocol Configuration tab allows you to specify your Sent and Drafts folder names. (For more information on setting up your IMAP mail, see the Notes 5 Help.)
The type of IMAP mode you choose determines how the mail file appears on your Replicator page. If you choose the IMAP online mode, the mail file appears with a two-way arrow on the Replicator page. This shows that you use IMAP synchronization with the file. If you choose IMAP offline mode, the mail file instead appears with one-way arrows to represent IMAP "pull" or "push"-only tasks. The following screen shows that the "jjones Yahoo mail" file uses IMAP offline mode, and the "Judy Jones" mail file uses IMAP online mode:
"Architecturally, we wanted to make it as seamless as possible," says Palone. The developers have succeeded to such an extent that you can use your IMAP mail just as you would use Notes mail.
NNTP
NNTP is the protocol behind the popular Usenet newsgroups discussion hierarchy. Usenet is a collection of newsgroups (similar to bulletin boards) on every conceivable topic. For example, the alt.fan.bill-gates newsgroup allows users to discuss the life and works of Microsoft's chairman, and the comp.groupware.lotus-notes newsgroup is a self-help group for users and administrators of Lotus Notes.
While Usenet newsgroups are publicly available and administered by a disinterested third-party (Usenet), individual companies are increasingly using NNTP to manage their own intranet and extranet discussions. A typical intranet application is a collection of office discussions. A typical extranet application contains one or more technical support discussions for the company's products.
The Notes R5 client can access Usenet, intranet, or extranet discussions using a proxy database. Like with POP3, SMTP, and IMAP, you must create an Account document that specifies your account server name (for example, news.myisp.com) and the protocol to use. NNTP servers usually do not require a login name and password.
When you create the NNTP account, Notes automatically adds a news bookmark to your Databases bookmark pane. Click the bookmark and you connect to the NNTP server. To subscribe to one or more newsgroups, you click the Newsgroups button, select from the list of newsgroups available on the NNTP server, and then click Subscribe. When you click Done, a folder is created for your subscribed newsgroups. The rest is just like Notes. Each subscribed newsgroup looks and feels just like a Notes folder; each newsgroup message looks and feels just like a Notes document. Here's a look at how the newsgroups appear in Notes R5:
You can make a replica of your NNTP proxy database so that you can store messages and your responses locally, then replicate with news servers on a regular basis. This convenience allows you to work offline and only connect to upload or download new messages to the newsgroups to which you are subscribed.
NNTP is an Internet standard that originated in the early 1980's. Traditionally, newsgroup messages were plain text only. To transfer binary data, like images, the Usenet community uses a form of data representation called UUEncoding. When displaying newsgroup messages, the Notes client automatically decodes UUEncoded images, but more importantly, it also can generate HTML and MIME messages. This is important because in the future, HTML and MIME may be the way vendors' products exchange rich content. (To learn more about the development of the newsgroup features in Notes R5, see our interview with Dave Delay, Internet client project leader.)
HTML
HTML is the standard that controls the display of Web pages, and support for this cornerstone Internet standard is greatly enhanced in R5. "We've really worked to improve the HTML rendering engine in Notes R5," says developer Jeff Eisen. This means that Notes won't have to take a back seat to Netscape Communicator or Microsoft Internet Explorer as a Web browser. "We've got much better support for things like tables and frames," says Eisen. Here's a screen shot that demonstrates the ability of Notes R5 to now handle a frames-based page:
The HTML improvements "give us two wins for the price of one," Eisen says. "Where previously only our Web browser used our HTML parser, now other areas of Notes use it, including the areas for reading and sending mail." Many products use HTML encoding to generate rich-text messages -- documents with multiple text sizes and colors, multiple fonts, embedded tables and graphics, and executable objects like animated GIFs and JavaScript -- and with R5, Notes mail interacts with these products.
Sending and receiving HTML-encoded messages, in turn, depends on R5's greatly improved support for MIME. So, let's now take a closer look at that MIME support.
MIME
MIME is a standard used to represent objects more complex than plain text, along with information that tells the recipient what application to open the object in. The native MIME support in R5 brings rich text capabilities to Internet mail.
"With R5, we can now store MIME natively," says Eisen. "Notes used to always convert everything to our proprietary rich text format. It used to be if mail came in from the Internet as MIME, we stripped the MIME formatting and presented a plain-text message. Now we can preserve it."
There's another win in the MIME improvements that Eisen points out: "In international messages, there may be character-set issues. Something may come in saying it's in a Western character set, but it's actually in Japanese characters. Previously, Notes had to guess the character set for the message, and if it guessed wrong, the original was lost. Now, Notes saves the original MIME object and, after some user interaction, displays it."
Notes R5 can send MIME-encoded mail, as well. "This means we can send mail out to the Internet with richness that we could only get in Notes mail before," says Eisen. "Now the client can generate messages as HTML and send them to other clients, such as Outlook and Netscape, and they'll look the same in those clients as they do in Notes."
One of the concerns of sending and receiving messages over the Internet, of course, is security. Since Notes' inception, it has provided secure mechanisms for signing and encrypting mail (as well as other documents). Notes does this by providing users with certificates and by creating certificate authorities to essentially bless the certificates. That is, if you trust a certificate authority (such as the IS group in your company), then you can trust that someone using a certificate issued by them is really who they say they are. Historically, Notes has used a proprietary certificate format.
With R5, Notes now supports X.509 certificates. X.509 is an Internet standard for certificates. The X.509 support is in addition to Notes' "old-style" of certificates. Using these X.509 certificates, Notes can use the Secure MIME or S/MIME protocol to send and receive signed and/or encrypted mail messages over the Internet. This allows secure interoperability with other mail (and Web) products from other vendors.
How does Notes know what format to use? Well, it depends. Each Person document in the Domino Directory (previously known as the Public Address Book) includes the "Format preference for incoming mail" field. This field specifies the storage format for your incoming mail -- No preference, Prefers MIME, or Prefer Notes Rich Text.
For outgoing messages sent through the Domino server, Notes looks up the recipient's format in the Domino Directory. If the "Format preference for incoming mail" field in the recipient's Person document has a value of "Prefers MIME," Notes sends the message in MIME format. If the recipient is not in the Domino Directory, Notes uses the format specified in the "Format for messages addressed to internet addresses" field in the Location document's Mail tab. Otherwise, the default format is Notes Rich Text.
For outgoing messages sent directly to the Internet, Notes uses the Internet mail format specified in the Mail and News pane of the User Preferences (File - Preferences - User Preferences). You can also specify an Internet message format for an individual message by clicking the Delivery Options on the new mail message, and using the "MIME Character Set" field on the Advanced tab.
LDAP
LDAP ("Lightweight Directory Access Protocol") has a name that hints at its history. It is indeed an alternative to a heavyweight protocol for searching X.500 directories, and it is emerging as an Internet standard. As with IMAP, Domino R4.6 delivered an LDAP server, and the first client implementation is in Notes R5. This means that from within Notes, you can use LDAP to search either the Domino Directory, or a foreign LDAP directory such as Bigfoot, InfoSpace, and so on. By using the LDAP Boolean search queries in these directories, you can retrieve e-mail addresses for easy mail addressing and copy information about people to your address book.
Adding support for the LDAP protocol to R5 wasn't difficult, according to developer Tim Shaw. "Most of the support was already there from our work in designing the LDAP server for R4.6. We just needed to expose it through the Notes UI." That required more changes to the Notes UI than for any of the other Internet protocols. However, the LDAP support is a perfect complement to other new directory features going into Notes R5, such as the new cc:Mail-style type-down addressing and the new Directory Catalog.
To set up access to an LDAP directory, you again specify an account server name (for example, ldap.bigfoot.com), login name and password, and then the protocol to use. In most cases, you can keep the default settings on the Protocol Configuration tab. (For more information on setting up LDAP accounts, see the Notes 5 Help.)
The biggest challenge in implementing an LDAP client in Notes was that LDAP's approach to helping a user find an address is fundamentally different from that of Notes. The mail addressing dialog in Notes R4.6 presents you with a dialog box that lists the names in your address book even before you start to search for your intended addressee. LDAP, on the other hand, requires you to conduct a search first. You must submit a name or some other search term before you can see any possible addressees. This difference required changes in the mail addressing dialog box -- it now instructs you to "select addresses" -- and the addition of a new dialog box to the address book that allows you to do LDAP searches. Here's the new Select Addresses dialog box, which includes the extended LDAP searching options in the lower part of the screen:
Especially important, the developers wrote the search function so that it treats cascaded local Notes address books as a single LDAP address book. You can cascade multiple address books saved on your Notes client, so that Notes searches all of them for an address as if they were a single file.
To cascade your address books:
- Choose File - Preferences - User Preferences.
- Click the Mail and News icon.
- Enter your directories in the Local address books field. By default, the field contains your address book, called names.nsf. You can add any other address books saved on your local hard drive by entering their file names separated by commas, or browsing for them.
Type-down for choosing an address
The advantage of cascading address books is that you can then use another feature new to R5, type-down addressing, against all your local address books. Type-down has been a feature of the cc:Mail user interface. When you enter a partial name in the address field, cc:Mail searches your address book and suggests the nearest match by using it to complete the address. If it isn't right, you can see other matches by using the down-arrow key to step through them. Each time you press the down-arrow key, the next match appears in the address field. To accept one, you press Enter or tab out of the field.
In contrast, Notes has used type-ahead. When you enter a partial address, Notes searches for a match and displays it by "typing ahead" to complete the address. If it isn't the right address, you type more of the name you want until either the directory search completes the right address, or you complete the name.
You can configure type-ahead to work just with your cascaded local address books, or with both local address books and the Domino Directory on the Domino server. For this reason, type-ahead appears in the Mail tab of the Location document -- because if the Domino Directory is not available from the location, trying to search it would return an error. In R5, if you configure type-ahead to "Local address books only," the new type-down function works with all your cascaded address books.
The new Directory Catalog: A good thing in a small package
One of those cascaded local address books could be your company's entire Domino Directory, if it's the new Mobile Directory Catalog.
The Domino Directory has grown in size over the years as the feature set for Notes and Domino has grown. However, when you're just interested in addressing your mail, you're interested in only a small subset of the data stored in the Domino Directory. The Mobile Directory Catalog takes its cue from LDAP in that it compresses the Domino Directory down to its minimal addressing information. By using this database locally, you can quickly look up the mail address for anyone in your enterprise. (Previously, you needed to access your home server for this information.) In addition, mobile users can use the Mobile Directory Catalog for error-free mail addressing while disconnected from the network.
"The typical Domino Directory on a server is so huge you wouldn't be able to do this," says Shaw. "But Mike O'Brien, who wrote the code, reduced the Lotus/IBM address book, for example, down to 10MB, so I've got the entire company directory on my laptop." The compression means that the Mobile Directory Catalog doesn't include some important data, like public keys for encrypting messages. For POP3 and IMAP messages, this doesn't matter, and for Notes messages, the Domino server compensates by doing just-in-time encryption as the message replicates from the client to the server.
The Internet of the future
The Mobile Directory Catalog is a prime example, says Mike Kudla, of making the Internet behave like Notes -- in this case, by opening up LDAP searching so that it works in the disconnected mode that has long been one of the most valuable features of Notes. "There's a lot more work to do," says Kudla. "We're still trying to drive and define some of the Internet standards so that they emerge matching the things Notes already does. This will allow us to work with Internet content in more Notes-like ways. We could, for example, use replication so that you can replicate a Web site or an intranet application in the same way that you can replicate a Notes database. However, this is really an extra thing. The important point is that with R5, you can now use Notes to access all of the information that is important to you, regardless of where that data is stored, how that data is stored or how that data is transmitted."