iVia Notes on Working with Natural Language and WordNet to Provide Word Sense Disambiguation and to Help Map to Ontologies

There are several compendiums/nets/corpora of languages out there. WordNet is one for the English language. Homonyms, synonyms, antonyms. These can be used to provide background concepts/word sense disambiguation to yield more accurate classification. That.s the promise. E.g., jaguar the animal and Jaguar the car (very different clumpings of words associated with them in the background). These language nets can also be used to increase accuracy in mappings from natural language phrases to controlled languages. See Bhattacharyya, Ramakrishnan and Mihalcea among others below. Ankle deep linguistics./p>

WordNet Worth Pursuing???:
According to Soumen, general comments:
> Our interests are in its usage to increase accuracy in classification
> and in helping to better map natural language terms to controlled
> language terms. I'm looking at it now with an eye towards working on
> it in the next grant cycle, perhaps.

As far as "standard" benchmarks (Reuters, 20 newsgroups) go, the effect of
adding WN info has been mild or negative. I had a student (in CMU now as
it turns out) who got negligible gains using WN synsets as features as
against the word itself, and got accuracy losses upon expanding features a
bit in WN. Agrees with broad concerns that WN "connects up too much" and
you need to pick and choose carefully. We did have feature selection in
place, even so, could not make it really work. Perhaps if the
classification problem is more challenging, WN assistance could be more
valuable. Also, mapping to controlled language could be a more promising
application. The best use of WN I have found is in open-domain question
answering ("general knowledge") and WSD, esp. the is-a hierarchies. But
WN is not very complete for technical jargon wrt most specialized fields.

WordNet
http://www.cogsci.princeton.edu/~wn/
WordNet� is an online lexical reference system whose design is inspired by current psycholinguistic theories of human lexical memory. English nouns, verbs, adjectives and adverbs are organized into synonym sets, each representing one underlying lexical concept. Different relations link the synonym sets.

WordNet Bibliography
http://engr.smu.edu/~rada/wnb/

WordNets of the World

http://www.globalwordnet.org/gwa/wordnet_table.htm

Projects:

Senseval
http://www.senseval.org/
There are now many computer programs for automatically determining the sense of a word in context (Word Sense Disambiguation or WSD). The purpose of Senseval is to evaluate the strengths and weaknesses of such programs with respect to different words, different varieties of language, and different languages.

Suggested Upper Merged Ontology: IEEE
http://virtual.cvut.cz/kifb/en/
http://suo.ieee.org/
The SUO WG is developing a Standard that will specify an upper ontology to support computer applications such as data interoperability, information search and retrieval, automated inferencing, and natural language processing.

MEANING
Developing Multilingual Web-scale Language Technologies
http://www.lsi.upc.es/~nlp/projectes/meaning.html
MEANING will be concerned with automatically collecting and analysing language data from the WWW on a large scale, and building more comprehensive multilingual lexical knowledge bases to support improved word sense disambiguation (WSD).

Mimida
http://www.gittens.nl/SemanticNetworks.html
a WordNet-based mechanically-generated multilingual semantic network for more than 20 languages based on dictionaries found on the Web.

WordNet Software and/or interesting projects:

WordNet::Similarity
http://sourceforge.net/projects/wn-similarity

SenseRelate
http://senserelate.sourceforge.net/

WordNet
http://www-inf.enst.fr/~milc/wordnet.us.html

HyperDic: WordNet
http://www.hyperdic.net/

WordNet Component Object Model
http://wncom.sourceforge.net/
This package provides a access to the WordNet database though a self contained component that offers both a general-purpose interface that can be used by scripting languages such as Visual Basic or JScript and an advanced interface to be used in C or C++.

WordNet to SQL
http://www.highlyillogical.org/wn2sql/
http://wordnet2sql.infocity.cjb.net/

Rada Mihalcea WordNet related software
Open Mind Word Expert OMWE1.0 sense tagged data ; TWA sense tagged data set Word Alignment Resources ; SemCor 1.6, SemCor 1.7, SemCor 1.7.1, SemCor 2.0 ; Mappings among various WordNet versions ; Evaluation code for text filtering
http://www.cs.unt.edu/~rada/downloads.html

Senseval
http://www.senseval.org/
There are now many computer programs for automatically determining the sense of a word in context (Word Sense Disambiguation or WSD). The purpose of Senseval is to evaluate the strengths and weaknesses of such programs with respect to different words, different varieties of language, and different languages.

OpenCyc
http://www.opencyc.org/
http://sourceforge.net/project/showfiles.php?group_id=27274 (LGPL)
OpenCyc is the open source version of the Cyc technology, the world's largest and most complete general knowledge base and commonsense reasoning engine.
(see Tom O. Hara articles)

Papers:

Text Representation with WordNet Synsets using Soft Sense Disambiguation
Ganesh Ramakrishnan and Pushpak Bhattacharyya
Ingenierie des Systemes d'Information Journal (ISI-NIS Journal),
http://www.cse.iitb.ac.in/~pb/papers/isi-nis.pdf
.Text information processing depends critically on the proper representation of texts. A common and naive way of representing a text is as a bag of its component words. This representation suffers primarily from two drawbacks, viz., polysemy and synonymy which arise because of the ambiguity of the words and the lack of information about the relations between the words. This paper presents a model for representing a text in terms of the synsets in the WordNet- the lexical knowledge base of English words along with the semantic relations. These synsets stand for concepts which correspond to the words of the text. In particular, a soft sense disambiguation approach has been proposed. The text representation so obtained is found to convey the key ideas that the texts deal with. WordNet relations with other words in the sentence are exploited to disambiguate the senses. This scheme has been evaluated using a goodness measure based the information content of the representation of the text. As an actual application, the problem of text classification has been taken up, and the results are encouraging.. .. The conclusion is that WordNet does helps relate the words in a document and in the emergence of meaning through mutual reinforcement of related words. This method of ranking synsets for a text can find use in many other applications like clustering, question answering and summarization, some of which are ongoing. Future work consists in assigning weights top the edges of the semantic graph and incorporating verbs.

Soft Word Sense Disambiguation,
Ganesh Ramakrishnan, Prithviraj B.P., A. Deepa, Pushpak Bhattacharyya, and Soumen Chakrabarti,
International Conference on Global Wordnet (GWC 04), Brno, Czeck Republic, January, 2004
http://www.cse.iitb.ac.in/~pb/papers/soft-wsd.pdf
Word sense disambiguation is a core problem in many tasks related to language processing. In this paper, we introduce the notion of soft word sense disambiguation which states that given a word, the sense disambiguation system should not commit to a particular sense, but rather, to a set of senses which are not necessarily orthogonal or mutually exclusive. The senses of a word are expressed by its WordNet synsets, arranged according to their relevance. The relevance of these senses are probabilistically determined through a Bayesian Belief Network. The main contribution of
the work is a completely probabilistic framework for word-sense disambiguation with a semi-supervised learning technique utilising WordNet. WordNet can be customized to a domain using corpora from that domain. This idea applied to question answering has been evaluated on TREC data and the results are promising.

Automatic Lexicon Generation through Wordnet
Nitin Verma and Pushpak Bhattacharyya,
International Conference on Global Wordnet (GWC 04), Brno, Czeck Republic, January, 2004
http://www.fi.muni.cz/gwc2004/proc/81.pdf
A lexicon is the heart of any language processing system. Accurate words with grammatical and semantic attributes are essential or highly desirable for any application- be it machine translation, information extraction, various forms of tagging or text mining. However, good quality lexicons are difficult to construct requiring enormous amount of time and manpower. In this paper, we present a method for automatically generating multilingual Universal Word (UW) dictionaries (for English, Hindi and Marathi) from an input document- making use of English, Hindi and Marathi WordNets. The dictionary entries are in the form of Universal Words (UWs) which are language words (primarily English) concatenated with disambiguation information. The entries are associated with syntactic and semantic properties- most of which too are generated automatically. In addition to the WordNet, the system uses a word sense disambiguator, an inferencer and the knowledge base (KB) of the Universal Networking Language which is a recently proposed interlingua. The lexicon so constructed is sufficiently accurate and reduces the manual labor substantially

Pushpak Bhattacharyya
Personal Page: Work in Information Retrieval and Extraction . Word Sense Disambiguation/WordNet
http://www.cse.iitb.ac.in/~pb/pubs-tmir.html

Word Sense Disambiguation and Text Similarity Measurement Using WordNet ,
P. Bhattacharyya and Narayan Unny,
chapter in Real World Semanic Web Applications, IOS Press, Amsterdam, 2002. Vipual Kashyap and Leon Shklar (ed), ISBN: 1 58603 306 9
http://www.cse.iitb.ac.in/~pb/papers/ios_pushpak.ps

Text Categorization and Information Retrieval Using WordNet Senses
http://www.fi.muni.cz/gwc2004/proc/110.pdf
.In this paper we study the influence of semantics in the Text Categorization (TC) and Information Retrieval (IR) tasks. The K Nearest Neighbours (K-NN) method was used to perform the text categorization. The experimental results were obtained taking into account for a relevant term of a document its corresponding WordNet synset. For the IR task, three techniques were investigated: the direct use of a weighted matrix, the Singular Value Decomposition (SVD) technique in the Latent Semantic
Indexing (LSI) model, and the bisecting spherical k-means clustering technique. The experimental results we obtained taking into account the semantics of the documents, allowed for an improvement of the performance for the text categorization whereas they were not so promising for the IR task..... .With regard to the study of how the semantic 30-KNN performed, it can be remarked that when documents are indexed with WordNet synsets, the performance slightly improved. Therefore, the use of words which refer to the same concept is a research direction we plan to investigate further. As future work, it would be
interesting to carry out some experiments using other data sets (e.g. the TREC document collection). In these experiments, the two vector representations should be also combined, in order to take into account with different weights, terms and WordNet synsets at the same time. With regard to the poor performance we obtained for the IR task, it could be due to mainly three reasons. First, the queries of 15 words were pretty long (normal queries are 1.5 words on average) and such long queries implicitly have a disambiguation effect. We should expect better effect of using WordNet for the normal 1 or 2 queries. Second, the semantic lemmatisation related synonyms when they are in the same morphologic group: it should be combined with standard morphological lemmatisation because they could complement
each other. Moreover, also other relations could be exploited in the semantic lemmatisation, possibly including the contextual information of the glosses of all the hyponyms. Last, but not least, indexing by WordNet synsets can be very helpful for text retrieval tasks only if the error rate is below 30% and, unfortunately, the state-of-the-art of WSD techniques perform with error rates ranging from 30% to 60% which cannot guarantee better results than standard word indexing

Using Measures of Semantic Relatedness for Word Sense Disambiguation
Patwardhan, Banerjee and Pedersen, Appears in the Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, February 17-21, 2003, Mexico City
http://www.d.umn.edu/~tpederse/Pubs/cicling2003-3.pdf
.This paper generalizes the Adapted Lesk Algorithm of Banerjee and Pedersen (2002) to a method of word sense disambiguation based on semantic relatedness. This is possible since Lesk's original Algorithm (1986) is based on gloss overlaps which can be viewed as a measure of semantic relatedness. We evaluate a variety of measures of semantic relatedness when applied to word sense disambiguation by carrying out
experiments using the English lexical sample data of Senseval-2. We find that the gloss overlaps of Adapted Lesk and the semantic distance measure of Jiang and Conrath (1997) result in the highest accuracy..

Text Classification Using WordNet Hypernyms
Sam Scott and Sam Matwin, 1998
http://acl.ldc.upenn.edu/W/W98/W98-0706.pdf
This paper describes a method of incorporating WordNet knowledge into text representation that can
lead to significant reductions in error rates on certain types of text classification tasks. The method uses the
lexical and semantic knowledge embodied in WordNet to move from a bag-of.words representation to a representation based on hypernym density. The appropriate value for the height of generalization parameter h depends on the characteristics of each classification task. A side benefit of the hypernym density representation is that the classification rules induced are often simpler and more comprehensible than rules induced using the bag-of-words. Our experience indicates that the hypernym density
representation can work well for texts that use an extended or unusual vocabulary, or are written by
multiple authors employing different terminologies. It is not likely to work well for text that is guaranteed
to be written concisely and efficiently, such as the text in Reuters-21578. In particular, hypernym
density is more likely to perform well on classification tasks involving narrowly defined and/or
semantically distant classes (such as SONGI and USENETI). In the case of classes that are broadly
defined and/or semantically related (such as SONG2 and USENET2) hypernym density does not always
outperform bag-of-words.

Word Sense Disambiguation Using Semantic Graphs
Narayan Unnikrishnan and P. Bhattacharyya
International Conference on Global WordNet (GWC 02), Mysore, India, January, 2002

Exploring automatic word sense disambiguation with decision lists and the Web
Agirre, Eneko and David Martinez.
In: Proceedings of the Semantic Annotation And Intelligent Annotation workshop organized by COLING, Luxembourg, 2000.
http://arxiv.org/abs/cs.CL/0010024
The most effective paradigm for word sense disambiguation, supervised learning, seems to be stuck because of the knowledge acquisition bottleneck. In this paper we take an in-depth study of the performance of decision lists on two publicly available corpora and an additional corpus automatically acquired from the Web, using the fine-grained highly polysemous senses in WordNet. Decision lists are shown a versatile state-of-the-art technique. The experiments reveal, among other facts, that SemCor can be an acceptable (0.7 precision for polysemous words) starting point for an all-words system. The results on the DSO corpus show that for some highly polysemous words 0.7 precision seems to be the current state-of-the-art limit. On the other hand, independently constructed hand-tagged corpora are not mutually useful, and a corpus automatically acquired from the Web is shown to fail.

WordNet enrichment with classification systems
Montoyo, Andr�s, Manuel Palomar and German Rigau.
In: Proceedings of the NAACL 2001 Workshop on WordNet and Other Lexical Resources, Pittsburgh, June 2001. http://www.seas.smu.edu/~rada/mwnw/papers/WNW-NAACL-202.pdf
This paper presents a new method to enrich semantically WordNet with categories from general domain classification systems. The method is performed in two consecutive steps. First, a lexical knowledge word sense disambiguation process. Second, a set of rules to select the main concepts as representatives for each category. The method has been applied to label automatically WordNet synsets with Subject Codes from a standard news agencies classification system. Experimental results show than the proposed method achieves more than 95% accuracy selecting the main concepts for each category.

Enriching WordNet concepts with topic signatures.
Agirre, Eneko, Olatz Ansa, David Martinez, and Eduard Hovy.
In: Proceedings of the NAACL 2001 Workshop on WordNet and Other Lexical Resources, Pittsburgh, June 2001.
http://www.seas.smu.edu/~rada/mwnw/papers/WNW-NAACL-228.pdf
http://arxiv.org/ftp/cs/papers/0109/0109031.pdf
.This paper explores the possibility of enriching the content of existing ontologies. The overall goal is to overcome the lack of topical links among concepts in WordNet. Each concept is to be associated to a topic signature, i.e., a set of related words with associated weights. The signatures can be automatically constructed from the WWW or from sense-tagged corpora. Both approaches are compared and evaluated on a word sense disambiguation task. The results show that it is possible to construct clean signatures from the WWW using some filtering techniques..

Manual and automatic semantic annotation with WordNet.
Fellbaum, Christiane, Martha Palmer, Hoa Trang Dang, Lauren Delfs, and Susanne Wolf.
In: Proceedings of the NAACL 2001 Workshop on WordNet and Other Lexical Resources, Pittsburgh, June 2001.

Automatic WordNet mapping using Word Sense Disambiguation
Lee, Changki and Geunbae Lee and Jungyun Seo. In: Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC 2000), Hong Kong, 2000.
http://acl.ldc.upenn.edu/W/W00/W00-1318.pdf

Instance Based Learning with Automatic Feature Selection Applied to Word Sense Disambiguation,
Rada Mihalcea
in Proceedings of the 19th International Conference on Computational Linguistics COLING 2002, Taiwan, August 2002.
http://www.cs.unt.edu/~rada/papers/coling.2002.ps
Instance based learning with automatic feature selection is a new approach in the WSD field. The algorithm was implemented in a system that achieves excellent performance on the data released during the senseval-2 English lexical task. The feature selection process is completely automated and it practically creates a classifier tailored to the behaviour of each specific word.

Highly Accurate Bootstrapping Algorithm for Word Sense Disambiguation
Mihalcea, Rada and Dan I. Moldovan. In: International Journal on Artificial Intelligence Tools, vol.10, no.1-2, [pg.5-21], 2001.
http://www.worldscinet.com/ijait/10/1001n02/S0218213001000398.html
http://www.cs.unt.edu/~rada/papers/ijait01.ps.gz
In this paper, we present a bootstrapping algorithm for Word Sense Disambiguation which succeeds in disambiguating a subset of the words in the input text with very high precision. It uses WordNet and a semantic tagged corpus, for the purpose of identifying the correct sense of the words in a given text. The bootstrapping process initializes a set of ambiguous words with all the nouns and verbs in the text. It then applies various disambiguation procedures and builds a set of disambiguated words: new words are sense tagged based on their relation to the already disambiguated words, and then added to the set. This process allows us to identify, in the original text, a set of words which can be disambiguated with high precision; 55% of the verbs and nouns are disambiguated with an accuracy of 92%.

Bootstrapping Large Sense Tagged Corpora,
Rada Mihalcea,
in Proceedings of the 3rd International Conference on Languages Resources and Evaluations LREC 2002, Las Palmas, Spain, May 2002.
http://www.cs.unt.edu/~rada/papers/lrec.2002.ps
The performance of Word Sense Disambiguation systems largely depends on the availability of sense tagged corpora. Since the semantic annotations are usually done by humans, the size of such corpora is
limited to a handful of tagged texts. This paper proposes a generation algorithm that may be used to automatically create large sense tagged corpora. The approach is evaluated through comparative sense
disambiguation experiments performed on data provided during the SENSEVAL-2 English all words and English lexical sample tasks.

``An Automatic Method for Generating Sense Tagged Corpora''
Mihalcea, Rada and Dan I. Moldovan.
In: Proceedings of AAAI '99, [pg.461-466], Orlando, FL, July 1999.
http://www.cs.unt.edu/~rada/papers/aaai99.ps.gz
The unavailability of very large corpora with semantically disambiguated words is a major limitation
in text processing research. For example, statistical methods for word sense disambiguation of free text are known to achieve high accuracy results when large corpora are available to develop context rules, to train and test them. This paper presents a novel approach to automatically generate arbitrarily large corpora for word senses. The method is based on (1) the information provided in WordNet, used to formulate
queries consisting of synonyms or definitions of word senses, and (2) the information gathered
from Internet using existing search engines. The method was tested on 120 word senses and a precision
of 91% was observed.

``Word Semantics for Information Retrieval: moving one step closer to the Semantic Web''
Mihalcea, Rada and Dan I. Moldovan.
In: International Conference on Tools in Artificial Intelligence, November, 2001.

``AutoASC - A System for Automatic Acquisition of Sense tagged Corpora ''
Mihalcea, Rada and Dan I. Moldovan.
In: International Journal of Pattern Recognition and Artificial Intelligence, [pg.3-17], vol.14, no.1, 2000.

``A Semi-Complete Disambiguation Algorithm for Open Text''
Mihalcea, Rada and Dan I. Moldovan.
In: Proceedings of AAAI 2000, [pg.1085], Austin, TX, July 2000.

``An Iterative Approach to Word Sense Disambiguation''
Mihalcea, Rada and Dan I. Moldovan.
In: Proceedings of Flairs 2000, [pg.219-223] Orlando, FL, May 2000. http://www.seas.smu.edu/~rada/papers/flairs00.ps.gz

``Constructing Bayesian networks from WordNet for word-sense disambiguation: representational and processing issues.''
Wiebe, Janyce, Tom O'Hara and Rebecca Bruce.
In: Proceedings of the COLING/ACL Workshop on Usage of WordNet in Natural Language Processing Systems, Montreal, 1998. http://www.CS.NMSU.Edu/~wiebe/pubs/papers/siglex98.ps (also see OpenCyc)
This paper describes a probabilistic model that is formed from the integration of an analytical and empirical component. The analytical component is a Bayesian network derived from WordNet, and the empirical component is composed of compatible probabilistic models formulated from tagged training data. The components are integrated in a formal, uniform framework based on the semantics of causal dependence. The paper explores various representational issues that must be addressed.

"Classifying functional relations in Factotum via WordNet hypernym associations",
O'Hara, Tom, and Janyce Wiebe (2003),
in Proc. Fourth International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2003). (also see OpenCyc)
This paper describes how to automatically classify the functional relations from the Factotum knowledge base via a statistical machine learning algorithm. This incorporates a method for inferring prepositional relation indicators from corpus data. It also uses lexical collocations (i.e., word associations) and class-based collocations based on the WordNet hypernym relations (i.e., is-subset-of). The result shows substantial improvement over a baseline approach.

"A Comparison of Word- and Sense-based Text Categorization Using Several Classification Algorithms".
Ath. Kehagias, V. Petridis, V.G. Kaburlasos, and P. Fragkou.
J. of Int. Inf. Sys., vol.21, pp.227-247, 2003.
http://users.auth.gr/~kehagiat/kehPub/journal/2001WordsSenses.ps

``Induction of Classification from Lexicon Expansion: Assigning Domain Tags to WordNet Entries''
Chang, Echa and Chu-Ren Huang and Sue-Jin Ker and Chang-Hua Yang
In: Proceedings of the Coling 2002 Workshop ''SemaNet'02: Building and Using Semantic Networks'', Taipei, August 2002. http://www.cs.ust.hk/~hltc/semanet02/pdf/chang.pdf

Word-Sense Disambiguation using Statistical Models of {Roget}'s Categories Trained on Large Corpora,
David Yarowsky booktitle = "Proceedings of {COLING}-92", month = "July", address = "Nantes, France", pages = "454--460", year = "1992", url = "citeseer.ist.psu.edu/yarowsky92wordsense.html" }
Perhaps one of the simplest and oldest paper. Very highly cited. Yarowsky proposes a solution to the problem of WSD using a thesaurus in a supervised learning environment. Word associations are recorded and for an unseen text, the senses of words are detected from the learnt associations.
Keywords (my own): Naive Bayes, thesaurus based disambiguation

Word sense disambiguation using conceptual density,
E. Agirre and G. Rigau",. 1996. In Proceedings of COLING'96, pages 16--22, Copenhagen, Danmark.",
year = "1996", url = "citeseer.ist.psu.edu/agirre96word.html" }
Agirre uses a measure based on the proximity of the text words in WordNet (conceptual density) to disambiguate the words. Perhaps one of the first papers that proposed a model for exploiting the proximity of word senses in WordNet. However, this paper talks only about proximity based on hypernym relations.
Keywords : conceptual density, proximity in WordNet.

Word sense disambiguation using decomposable models
Rebecca Bruce and Janyce Wiebe, booktitle = "Proceedings of the {ACL}-94, 32nd Annual Meeting of the Association for Computational Linguistics", address = "Las Cruces, US", pages = "139--145",
year = "1994", url = "citeseer.ist.psu.edu/article/bruce94wordsense.html" }
Bruce and Weibe introduce the use of decomposable models (maximum-entropy, and logistic regression are some decomposable models. Log-linear models are decomposable) for incorporating a host of discrete and continuous valued features for sense disambiguation which is treated as a classification task.
The features used were : part of speech tags one and two places to the left and right of the ambiguous word, part of speech tag of the word and a collocation variable for each sense of the word (a binary feature)
The paper has a lots of maths but not much elaboration on WSD and experiments are far from satisfactory.
In my opinion - a bit hyped up !

Two Languages Are More Informative Than One,
Dagan, I., Itai, A., and Schwall, U., text = "Proceedings of the 29th Annual Meeting of the Association for Computational Linguistics (ACL-91), pp. 130-137", year = "1991", url = "http://acl.ldc.upenn.edu/P/P91/P91-1017.pdf" }
A good representative paper illustrating the use of bi-lingual corpus for word-sense-disambiguation. Central idea is that word in a language can be disambiguated by looking up its translation in another language. Scale of experiment was again very very small. It hinges on disambiguating words in one language usin statistical data about lexical (or surface) relations in another language.
But this paper exposes a chicken-egg problem ! One major requirement of WSD is for machine translation. And this paper requires machine translated documents for effective WSD !!

Word sense distinguishability and inter-coder agreement,
R. Bruce and J. Wiebe, (1998). Proceedings of the 3rd Conference on Empirical Methods in Natural
Language Processing (EMNLP-98). Association for Computational Linguistics SIGDAT, Granada, Spain, June 1998.", year = "1998", url = "citeseer.ist.psu.edu/bruce98wordsense.html" }
This paper tries and analyses the distinguishability between word senses within and across dictionaries (category distinguishability). It also analyses and presents tests for observer differences in sense tagging (of a highly ambiguous word - ``interest'').
Again scale of experiment is small

WASP-Bench: an MT Lexicographer's Workstation Supporting State-of-the-art
Lexical Disambiguation,
Adam Kilgariff and David Tugwell, url = "citeseer.ist.psu.edu/kilgariff01waspbench.html" }
This paper describes more of a work-bench for sense disambiguation than an algorithm. Mutual information between a word-sense and the words in context, is used as a measure of disambiguation. So this paper does not provide a novel algorithm as such. But it describes a tool for lexical disambiguation that was built on this idea. Moreover, the tool supports human-assisted sense disambiguation.