org.apache.lucene.analysis.cn.smart

Class SmartChineseAnalyzer

  • All Implemented Interfaces:
    Closeable, AutoCloseable


    public final class SmartChineseAnalyzer
    extends Analyzer

    SmartChineseAnalyzer is an analyzer for Chinese or mixed Chinese-English text. The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.

    Segmentation is based upon the Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.

    This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.

    The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!