Discovery Server's linguistic engine

Profiles: Alternate name support

Spider basics

Best practice: Generate taxonomy on the single common business language

Related links:
Think globally: Creating a multilingual Notes/Domino environment

Worldwide messaging: Using International MIME in R5

Creating meaningful K-map taxonomies

Importing a file system taxonomy into a K-map

Get the PDF:
(183 KB)

Brazilian Portuguese	Japanese
Danish	Korean
Dutch	Norwegian (Bokmal and Nynorsk)
English	Simplified Chinese
Finnish	Spanish
French	Swedish
German	Traditional Chinese
Italian

• The language preference of the user’s browser settings (in Internet Explorer, this is specified in the Internet Options dialog on the Language tab)
  
  
  
• The language resource set installed on the server

1. Open the XML file \Lotus\DS\inxight\lx-2\lang33\std.langid-config in a text editor. Remember to back up the file before making any changes.
2. Under the tag <encodings-languages-covered> delete the language(s) you don't need.
   
   <encodings-languages-covered>
   <!-- Some list tags have been omitted from this code sample -->
   <list key = "utf_16">
   <item key = "english" />
   <item key = "french" />
   <item key = "german" />
   <item key = "italian" />
   <item key = "spanish" />
   <item key = "portuguese" />
   <item key = "dutch" />
   <item key = "swedish" />
   <item key = "finnish" />
   <item key = "danish" />
   <item key = "bokmal" />
   <item key = "nynorsk" />
   <item key = "simplified-chinese" />
   <item key = "traditional-chinese" />
   <item key = "korean" />
   <item key = "japanese" />
   </list>
   </encodings-languages-covered>
   
3. Save the file.
4. Restart the server.

1. Read a document from the repository. If the document is from the file system, a Web site, or an attached file, the spider uses the file filter module or the HTML filter to process the document.
2. Convert the document to Unicode from its native encoding and encode it in a structured XML.
3. Detect the human language of the document using the linguistic engine.
4. Generate a summary for the document using the linguistic engine if one does not already exist.
5. Tokenize the document using the linguistic engine according to the rules associated with the language determined in step 3.
6. Count the tokens and include a vector of token counts in the XML for the K-map builder.

• Numbers
• Hexadecimal numbers
• Punctuation
• Apostrophe/conjunctions and the equivalents in other languages
• Possessives and the equivalent in other languages
• Single alphabet letters in isolation, such as a, b, c, and so on
• Email addresses, file locations like C:\DS, and URLs
• Tokens containing any of the following symbols: `~!@$%^&*()+-=[]{}\|;:'",.<>/?
• Conjunctions, such as "and" and "or" (only applies to Chinese)
• Time phrases, such as Monday (only applies to Chinese)

• Three factors shape the initial taxonomy: one, the distribution of spoken languages and locales for the audience and expected access of the audience to the secondary languages represented; two, the distribution of languages in the data set; and three, its subject matter and how technical that content is. It is very important that you consider these factors when assembling the training set from which the initial taxonomy is built, because the initial taxonomy influences the way documents are later classified. For example:
  • If your data set is heavily weighted towards one language—90% English and 10% German, say—your initial taxonomy might contain one or two categories in which all German documents are placed, regardless of content. It is also likely that the categories containing German documents will exist in their own subsection of the taxonomy.
  • If the same distribution of languages exists for a data set consisting of IT or otherwise highly technical data, it is more likely that German documents and categories will be scattered among the predominantly English categories. In fact, in the IT business area multilingual data sets work out well because IT uses so many of the same (English) terms.
  • Distribution of languages in the initial taxonomy influence the way in which documents in other languages are later classified. If your initial taxonomy is built using English and German documents, then classifying Spanish documents later on will probably cause the Spanish documents to be placed in the Uncategorized category, unless the Spanish documents share some technical tokens, for example, Lotus, IBM, OS/2, and so on.
  • If your data set consists of team rooms for three different locales: UK, Spain, and Japan, and there is an equal distribution of documents in English, Spanish, and Japanese, your initial K-map is likely to subdivide into three distinct subsections. In that case, if your business goal specified multilingual categories, you might distribute languages into categories as suggested later in this article; if it better serves your audiences, you may decide to divide the three into separate K-maps for separate distribution.
  • How pertinent is the language difference—are the documents in Spanish contributed by a Spanish-speaking sales force and fairly coherent, targeted to the locale of the specific language? Or is the language difference not relevant to the goal of the taxonomy, authored only incidentally in Spanish, its topic common to the international audience?
• When editing labels for a multilingual taxonomy, you may be tempted to offer translations of the labels in all represented languages to allow access across the languages. Remember, though, that affinity generation pulls these labels into the profiles where long strings of text may become unwieldy.
  

1. Create an English training set—a selected subset of your data set that contains the 1% of the documents, written in the common business language, that best represents your business goal and the mental model you want to present.
2. Spider the training set to create an initial taxonomy. Edit this initial English taxonomy until you are satisfied with the category labels and the distribution of documents in the categories.
3. Enable classification and spider a larger subset of your data—the next best 10%. In our example, these repositories contain German documents.
4. Edit the resulting taxonomy, pulling the German documents into the English categories if they relate to them, and keeping them in separate subcategories if they do not.
5. Add more data repositories from your entire data set. Documents from the entire data set will enter existing categories, with the German documents already in the categories attracting the new documents in German, and English documents attracting English documents with similar content. When a substantial number of new documents appear, check to make sure you are satisfied with the categories and the distribution of documents in them.