sphinx全文索引教程

Sphinx是一个基于SQL的全文检索引擎,可以结合MySQL,PostgreSQL做全文搜索,它可以提供比数据库本身更专业的搜索功能,使得应用程序更容易实现专业化的全文检索。Sphinx特别为一些脚本语言设计搜索API接口,如PHP,Python,Perl,Ruby等,同时为MySQL也设计了一个存储引擎插件。

Sphinx单一索引最大可包含1亿条记录,在1千万条记录情况下的查询速度为0.x秒(毫秒级)。Sphinx创建索引的速度为:创建100万条记录的索引只需 3~4分钟,创建1000万条记录的索引可以在50分钟内完成,而只包含最新10万条记录的增量索引,重建一次只需几十秒。

英文介绍:http://www.sphinxsearch.com/docs/manual-0.9.9.html

一、首先需要在服务器上安装sphinx
在Windows上安装sphinx
1.下载支持mysql的包  http://www.sphinxsearch.com/downloads/sphinx-0.9.9-win32.zip
2.解压缩 sphinx-0.9.9-win32.zip 到 D:sphinx
3.安装sphinx服务,在命令行执行命令

D:sphinxsearchd –install –config d:sphinxsphinx.conf –servicename SphinxSearch

英文参照:http://www.sphinxsearch.com/docs … #installing-windows

在Linux服务器上安装sphinx
1.下载源码包 http://www.sphinxsearch.com/downloads/sphinx-0.9.9.tar.gz

$ tar xzvf sphinx-0.9.8.tar.gz
$ cd sphinx
$ ./configure –prefix=/usr/local/sphinx –with-mysql=/usr/local/mysql
$ make
$ make install

常见问题1

/usr/local/sphinx-0.9.9/src/sphinx.cpp:20060: undefined reference to libiconv_open’
/usr/local/sphinx-0.9.9/src/sphinx.cpp:20078: undefined reference to
libiconv’
/usr/local/sphinx-0.9.9/src/sphinx.cpp:20084: undefined reference to libiconv_close’
collect2: ld returned 1 exit status
make[2]: *** [indexer] Error 1
make[2]: Leaving directory
/home/jling/sphinx-0.9.9/src’
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/jling/sphinx-0.9.9/src’
make: *** [all-recursive] Error 1

解决办法:打开configure文件,找到“#define USE_LIBICONV 1”,将注释去掉,并将1改成0。
常见问题2

error while loading shared libraries: libmysqlclient.so.16: cannot open shared object file: No such file or directory

解决办法:

64位系统ln -s /usr/local/webserver/mysql/lib/mysql/libmysqlclient.so.16.0.0 /usr/lib64/libmysqlclient.so.16
32位系统ln -s /usr/local/webserver/mysql/lib/mysql/libmysqlclient.so.16.0.0 /usr/lib/libmysqlclient.so.16

 

sphinx.conf样例

source main
{
type     = mysql    #数据库类型
sql_host    = 10.228.134.211 #数据库ip
sql_user    = admin    #数据库用户名
sql_pass    = admin    #数据库密码
sql_db     = phpcms_v9   #数据库名
sql_port    = 3306    # 数据库端口

sql_query_pre = SET NAMES utf8
sql_query_pre = REPLACE INTO v9_sphinx_counter SELECT 1, MAX(searchid) FROM v9_search
sql_query = SELECT searchid, adddate, siteid, typeid, id, data FROM v9_search
WHERE searchid>=$start AND searchid<=$end
sql_query_range  = SELECT 1,max_doc_id FROM v9_sphinx_counter WHERE counter_id=1
sql_range_step = 5000

#字符串属性设置、需要过滤、排序的时候用到
sql_attr_uint  = typeid
sql_attr_uint  = siteid
sql_attr_uint  = id
sql_attr_timestamp  = adddate
sql_query_info  = SELECT * FROM v9_search WHERE searchid=$id
}

source delta
{
type     = mysql    #数据库类型
sql_host    = 10.228.134.211 #数据库ip
sql_user    = admin    #数据库用户名
sql_pass    = admin    #数据库密码
sql_db     = phpcms_v9   #数据库名
sql_port    = 3306    # 数据库端口

sql_query_pre = SET NAMES utf8
sql_query = SELECT searchid, adddate, siteid, typeid, id, data FROM v9_search
WHERE searchid >( SELECT max_doc_id FROM v9_sphinx_counter WHERE counter_id=1 )
sql_query_post = REPLACE INTO v9_sphinx_counter SELECT 1, MAX(searchid) FROM v9_search
#字符串属性设置、需要过滤、排序的时候用到
sql_attr_uint  = typeid
sql_attr_uint  = siteid
sql_attr_uint  = id
sql_attr_timestamp  = adddate
sql_query_info  = SELECT * FROM v9_search WHERE searchid=$id
}

#主索引
index main
{
source = main
# 放索引的目录
path = D:sphinxdatamain
# 编码
charset_type = utf-8
# 指定utf-8的编码表
charset_table = 0..9, A..Z->a..z, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F
# 简单分词,只支持0和1,如果要搜索中文,请指定为1
ngram_len = 1
# 需要分词的字符,如果要搜索中文,去掉前面的注释
ngram_chars   = U+3000..U+2FA1F
}

#增量索引
index delta
{
source = delta
path = D:sphinxdatadelta
# 编码
charset_type = utf-8
# 指定utf-8的编码表
charset_table = 0..9, A..Z->a..z, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F
# 简单分词,只支持0和1,如果要搜索中文,请指定为1
ngram_len = 1
# 需要分词的字符,如果要搜索中文,去掉前面的注释
ngram_chars   = U+3000..U+2FA1F
}
indexer
{
mem_limit    = 128M
}

searchd
{
port     = 9312
log      = D:sphinxdataphpcmssearchd.log
query_log    = D:sphinxdataphpcmsquery.log
read_timeout   = 5
max_children   = 30
pid_file    = D:sphinxdataphpcmssearchd.pid
max_matches    = 2000
seamless_rotate   = 0
preopen_indexes   = 0
unlink_old    = 1
}

 

附件:设置计划任务更新索引
1.windows下
需要设置计划任务
#凌晨4点合并索引,执行merge.bat
#其余时间每分钟更新索引,执行delta.bat

merge.bat

@ECHO off
D:sphinxbinindexer.exe –config D:sphinxsphinx.conf –merge main delta –rotate
echo indexing, window will close when complete

 

delta.bat

@ECHO off
D:sphinxbinindexer.exe –config D:sphinxsphinx.conf delta –rotate
echo indexing, window will close when complete

2.linux下编辑定时任务 crontab -e

#凌晨4点合并索引,其余时间每分钟更新索引
* 0-3 * * * /usr/local/sphinx/bin/indexer –config /usr/local/sphinx/etc/sphinx.conf delta –rotate
* 6-23 * * * /usr/local/sphinx/bin/indexer –config /usr/local/sphinx/etc/sphinx.conf delta –rotate
0 4 * * * /usr/local/sphinx/bin/indexer –config /usr/local/sphinx/etc/sphinx.conf –merge main delta –rotate

各种路径、权限需要应用所在服务器一致,如:
sphinx.conf 中需要配置
sql_host 数据库主机地址
sql_user 数据库用户名
sql_pass 数据库密码
sql_db 数据库名
sql_port 数据库端口
phpcms表前缀样例中为phpcms_
索引路径 D:sphinxdatadelta

phpcmsv9_withsphinx.zip