org.apache.lucene.queries.mlt

Class MoreLikeThis



  • public final class MoreLikeThis
    extends Object
    Generate "more like this" similarity queries. Based on this mail:
    
     Lucene does let you access the document frequency of terms, with IndexReader.docFreq().
     Term frequencies can be computed by re-tokenizing the text, which, for a single document,
     is usually fast enough.  But looking up the docFreq() of every term in the document is
     probably too slow.
     
     You can use some heuristics to prune the set of terms, to avoid calling docFreq() too much,
     or at all.  Since you're trying to maximize a tf*idf score, you're probably most interested
     in terms with a high tf. Choosing a tf threshold even as low as two or three will radically
     reduce the number of terms under consideration.  Another heuristic is that terms with a
     high idf (i.e., a low df) tend to be longer.  So you could threshold the terms by the
     number of characters, not selecting anything less than, e.g., six or seven characters.
     With these sorts of heuristics you can usually find small set of, e.g., ten or fewer terms
     that do a pretty good job of characterizing a document.
     
     It all depends on what you're trying to do.  If you're trying to eek out that last percent
     of precision and recall regardless of computational difficulty so that you can win a TREC
     competition, then the techniques I mention above are useless.  But if you're trying to
     provide a "more like this" button on a search results page that does a decent job and has
     good performance, such techniques might be useful.
     
     An efficient, effective "more-like-this" query generator would be a great contribution, if
     anyone's interested.  I'd imagine that it would take a Reader or a String (the document's
     text), analyzer Analyzer, and return a set of representative terms using heuristics like those
     above.  The frequency and length thresholds could be parameters, etc.
     
     Doug
     

    Initial Usage

    This class has lots of options to try to make it efficient and flexible. The simplest possible usage is as follows. The bold fragment is specific to this class.

     IndexReader ir = ...
     IndexSearcher is = ...
    
     MoreLikeThis mlt = new MoreLikeThis(ir);
     Reader target = ... // orig source of doc you want to find similarities to
     Query query = mlt.like( target);
     
     Hits hits = is.search(query);
     // now the usual iteration thru 'hits' - the only thing to watch for is to make sure
     //you ignore the doc if it matches your 'target' document, as it should be similar to itself
    
     

    Thus you:

    1. do your normal, Lucene setup for searching,
    2. create a MoreLikeThis,
    3. get the text of the doc you want to find similarities to
    4. then call one of the like() calls to generate a similarity query
    5. call the searcher to find the similar docs

    More Advanced Usage

    You may want to use setFieldNames(...) so you can examine multiple fields (e.g. body and title) for similarity.

    Depending on the size of your index and the size and makeup of your documents you may want to call the other set methods to control how the similarity queries are generated:



     Changes: Mark Harwood 29/02/04
     Some bugfixing, some refactoring, some optimisation.
     - bugfix: retrieveTerms(int docNum) was not working for indexes without a termvector -added missing code
     - bugfix: No significant terms being created for fields with a termvector - because
     was only counting one occurrence per term/field pair in calculations(ie not including frequency info from TermVector)
     - refactor: moved common code into isNoiseWord()
     - optimise: when no termvector support available - used maxNumTermsParsed to limit amount of tokenization
     
    • Method Detail

      • getBoostFactor

        public float getBoostFactor()
        Returns the boost factor used when boosting terms
        Returns:
        the boost factor used when boosting terms
        See Also:
        setBoostFactor(float)
      • getAnalyzer

        public Analyzer getAnalyzer()
        Returns an analyzer that will be used to parse source doc with. The default analyzer is not set.
        Returns:
        the analyzer that will be used to parse source doc with.
      • setAnalyzer

        public void setAnalyzer(Analyzer analyzer)
        Sets the analyzer to use. An analyzer is not required for generating a query with the like(int) method, all other 'like' methods require an analyzer.
        Parameters:
        analyzer - the analyzer to use to tokenize text.
      • getMinTermFreq

        public int getMinTermFreq()
        Returns the frequency below which terms will be ignored in the source doc. The default frequency is the DEFAULT_MIN_TERM_FREQ.
        Returns:
        the frequency below which terms will be ignored in the source doc.
      • setMinTermFreq

        public void setMinTermFreq(int minTermFreq)
        Sets the frequency below which terms will be ignored in the source doc.
        Parameters:
        minTermFreq - the frequency below which terms will be ignored in the source doc.
      • getMinDocFreq

        public int getMinDocFreq()
        Returns the frequency at which words will be ignored which do not occur in at least this many docs. The default frequency is DEFAULT_MIN_DOC_FREQ.
        Returns:
        the frequency at which words will be ignored which do not occur in at least this many docs.
      • setMinDocFreq

        public void setMinDocFreq(int minDocFreq)
        Sets the frequency at which words will be ignored which do not occur in at least this many docs.
        Parameters:
        minDocFreq - the frequency at which words will be ignored which do not occur in at least this many docs.
      • getMaxDocFreq

        public int getMaxDocFreq()
        Returns the maximum frequency in which words may still appear. Words that appear in more than this many docs will be ignored. The default frequency is DEFAULT_MAX_DOC_FREQ.
        Returns:
        get the maximum frequency at which words are still allowed, words which occur in more docs than this are ignored.
      • setMaxDocFreq

        public void setMaxDocFreq(int maxFreq)
        Set the maximum frequency in which words may still appear. Words that appear in more than this many docs will be ignored.
        Parameters:
        maxFreq - the maximum count of documents that a term may appear in to be still considered relevant
      • setMaxDocFreqPct

        public void setMaxDocFreqPct(int maxPercentage)
        Set the maximum percentage in which words may still appear. Words that appear in more than this many percent of all docs will be ignored.
        Parameters:
        maxPercentage - the maximum percentage of documents (0-100) that a term may appear in to be still considered relevant
      • isBoost

        public boolean isBoost()
        Returns whether to boost terms in query based on "score" or not. The default is DEFAULT_BOOST.
        Returns:
        whether to boost terms in query based on "score" or not.
        See Also:
        setBoost(boolean)
      • setBoost

        public void setBoost(boolean boost)
        Sets whether to boost terms in query based on "score" or not.
        Parameters:
        boost - true to boost terms in query based on "score", false otherwise.
        See Also:
        isBoost()
      • getFieldNames

        public String[] getFieldNames()
        Returns the field names that will be used when generating the 'More Like This' query. The default field names that will be used is DEFAULT_FIELD_NAMES.
        Returns:
        the field names that will be used when generating the 'More Like This' query.
      • setFieldNames

        public void setFieldNames(String[] fieldNames)
        Sets the field names that will be used when generating the 'More Like This' query. Set this to null for the field names to be determined at runtime from the IndexReader provided in the constructor.
        Parameters:
        fieldNames - the field names that will be used when generating the 'More Like This' query.
      • getMinWordLen

        public int getMinWordLen()
        Returns the minimum word length below which words will be ignored. Set this to 0 for no minimum word length. The default is DEFAULT_MIN_WORD_LENGTH.
        Returns:
        the minimum word length below which words will be ignored.
      • setMinWordLen

        public void setMinWordLen(int minWordLen)
        Sets the minimum word length below which words will be ignored.
        Parameters:
        minWordLen - the minimum word length below which words will be ignored.
      • getMaxWordLen

        public int getMaxWordLen()
        Returns the maximum word length above which words will be ignored. Set this to 0 for no maximum word length. The default is DEFAULT_MAX_WORD_LENGTH.
        Returns:
        the maximum word length above which words will be ignored.
      • setMaxWordLen

        public void setMaxWordLen(int maxWordLen)
        Sets the maximum word length above which words will be ignored.
        Parameters:
        maxWordLen - the maximum word length above which words will be ignored.
      • setStopWords

        public void setStopWords(Set<?> stopWords)
        Set the set of stopwords. Any word in this set is considered "uninteresting" and ignored. Even if your Analyzer allows stopwords, you might want to tell the MoreLikeThis code to ignore them, as for the purposes of document similarity it seems reasonable to assume that "a stop word is never interesting".
        Parameters:
        stopWords - set of stopwords, if null it means to allow stop words
        See Also:
        getStopWords()
      • getMaxQueryTerms

        public int getMaxQueryTerms()
        Returns the maximum number of query terms that will be included in any generated query. The default is DEFAULT_MAX_QUERY_TERMS.
        Returns:
        the maximum number of query terms that will be included in any generated query.
      • setMaxQueryTerms

        public void setMaxQueryTerms(int maxQueryTerms)
        Sets the maximum number of query terms that will be included in any generated query.
        Parameters:
        maxQueryTerms - the maximum number of query terms that will be included in any generated query.
      • setMaxNumTokensParsed

        public void setMaxNumTokensParsed(int i)
        Parameters:
        i - The maximum number of tokens to parse in each example doc field that is not stored with TermVector support
      • like

        public Query like(int docNum)
                   throws IOException
        Return a query that will return docs like the passed lucene document ID.
        Parameters:
        docNum - the documentID of the lucene doc to generate the 'More Like This" query for.
        Returns:
        a query that will return docs like the passed lucene document ID.
        Throws:
        IOException
      • like

        public Query like(String fieldName,
                          Reader... readers)
                   throws IOException
        Return a query that will return docs like the passed Readers. This was added in order to treat multi-value fields.
        Returns:
        a query that will return docs like the passed Readers.
        Throws:
        IOException
      • describeParams

        public String describeParams()
        Describe the parameters that control how the "more like this" query is formed.