In most cases LowerCaseFilterFactory is sufficient. In general most languages don't require special tokenization (and will work just fine with Whitespace + WordDelimiterFilter), so you can safely tailor the English "text" example schema definition to fit. If you have problems (your language is highly-inflectional, etc), you might want to try using an n-gram approach as an alternative. Remember, things like stemming and stopwords aren't mandatory for the search to work, only optional language-specific improvements.
A first step is to start with the "textgen" type in the example schema. Note: Be sure to use the Turkish-specific TurkishLowerCaseFilterFactory Solr3.1 My language is not listed!!! These two are simple rule based stemmers, not handing exceptions or irregular forms.Įxample set of Turkish stopwords (Be sure to switch your browser encoding to UTF-8)
Since Solr3.6 you can also use solr.NorwegianLightStemFilterFactory for a lighter variant or solr.NorwegianMinimalStemFilterFactory attempting to normalize plural endings only. Solr includes support for stemming Norwegian via solr.SnowballPorterFilterFactory, and Lucene includes an example stopword list. Note: Be sure to use PositionFilter at query-time (only) as these languages do not use spaces between words. To use this tokenizer, see solr/contrib/analysis-extras/README.txt for instructions on which jars you need to add to your SOLR_HOME/lib Lucene provides support for segmenting these languages into syllables with solr.ICUTokenizerFactory in the analysis-extras contrib module.
#Apache lucene relevance models how to#
Key takeaways from the talk will be a thorough understanding of how to make Lucene powered search a first class citizen to build interactive machine learning pipelines using Spark Mllib.Example set of Italian stopwords (Be sure to switch your browser encoding to UTF-8) Lao, Myanmar, Khmer We will show algorithmic details of our model building pipelines that are employed for performance optimization and demonstrate latency of the APIs on a suite of queries generated over 1M+ terms. In this talk we will demonstrate LuceneDAO write and read performance on millions of documents with 1M+ terms. Our synchronous API uses Spark-as-a-Service while our asynchronous API uses kafka, spark streaming and HBase for maintaining job status and results.
We developed Spark Mllib based estimators for classification and recommendation pipelines and use the vector space representation to train, cross-validate and score models to power synchronous and asynchronous APIs. LuceneDAO is used to load the shards to Spark executors and power sub-second distributed document retrieval for the queries. We used Spark as our distributed query processing engine where each query is represented as boolean combination over terms. Lucene shards maintain the document-term view for search and vector space representation for machine learning pipelines. We introduced LuceneDAO (data access object) that supports building distributed lucene shards from dataframe and save the shards to HDFS for query processors like SolrCloud. However when dealing with document datasets where the features are represented as variable number of columns in each document and use-cases demand searching over columns to retrieve documents and generate machine learning models on the documents retrieved in near realtime, a close integration within Spark and Lucene was needed.
#Apache lucene relevance models full#
Spark SQL and Mllib are optimized for running feature extraction and machine learning algorithms on large columnar datasets through full scan.
0 kommentar(er)
