Day: March 13, 2016

Yes, the Porter Stemmer is the most popular approach by far. See A survey of stemming algorithms in information retrieval  for a survey,  nltk.stem package  for NLTK implementations, and Porter Stemming Algorithm  for Porter’s own description of it. There are tweaks of it around, but noone has gone for anything different; and English being the way it is, there’s no real interest in the more powerful lemmatisers, which would do actual dictionary work.

As a linguist, I (and I’m sure many another linguist) am aghast at what the Porter Stemmer doesn’t do. stupider for example goes to stupid, but bigger does not go to big: Porter does not touch bisyllabic words—there’s too much risk of error. Similarly, Porter has no idea or interest in irregular forms.

It is a decent compromise on doing too much versus doing too little (and doing too much is a real problem). What people always forget is that it has to be customised, to deal with the vocabulary you’re likely to encounter, with an exceptions list. That applies in particular to its use in Lucene/SOLR.

What Spyros Theodoritsis said. In addition, for medical terminology in particular, many of the words for parts of the body have changed in Modern Greek, and the ancient words only survive in… Greek medical terminology.

Modern Greek has replaced the word for liver with συκώτι, “figged”; it’s the same dish that gave us French foie. Nothing in Modern Greek will tell you that the word used in medicine is hepar, or that the compounds are formed with hepato-.

What Niko Vasileas said: παρακαλώ “please”, as in “please don’t mention it”. Add δεν κάνει τίποτα, “it does nothing” (presumably, “it costs nothing”), truncated to τίποτα “nothing!”