ES索引瘦身 压缩——_source _all 均disable filed store为no,引入第三方DB存储原始数据,去掉pos倒排和doc_values,强制定期merge segments,将所有fileds合并为一个field big string-阿里云开发者社区

开发者社区> 开发与运维> 正文
登录阅读全文

ES索引瘦身 压缩——_source _all 均disable filed store为no,引入第三方DB存储原始数据,去掉pos倒排和doc_values,强制定期merge segments,将所有fileds合并为一个field big string

简介:

原始数据:835MB

 

ES 设置了_source _all disabled 且设置了仅仅存docs倒排
Wed Feb 22 11:58:27 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
435 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 11:59:43 CST 2017
done

不做任何压缩:
Wed Feb 22 14:56:04 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1656 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 14:57:54 CST 2017
done
curl -XPOST 'localhost:9200/hec_test3/_forcemerge'
bone@PEK1000074003:~/test_data/splunk_es_cmp$ du -sm ~/elasticsearch-2.4.1/data/
969 /home/bone/elasticsearch-2.4.1/data/


_all disabled:
Wed Feb 22 15:22:09 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1576 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:23:25 CST 2017
done
merge后:
787

_source disabled: field store为false 搜索不到东西了 如果不存DB的话
Wed Feb 22 15:26:06 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1288 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:27:45 CST 2017
done
merge后:
512


_source disabled: field store为yes
Wed Feb 22 15:33:28 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1583 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:34:45 CST 2017
done
merge后:
780


最终设置:
curl -XPUT 'http://localhost:9200/hec_test3' -d '
{
"mappings": {
"hec_type3": {
"_source": {
"enabled": false
},
"_all": {
"enabled": false
},
"properties": {
"event": {
"type": "string",
"include_in_all": false, 
"doc_values": false,
"index_options": "docs"
}
}

}
}
'
压缩配置:
index.codec: best_compression

bash test_es24.sh
Wed Feb 22 15:39:39 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
892 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 15:40:36 CST 2017
done
merge后:
202 MB

50个字段拆分后:
用上述最终设置的效果:
Wed Feb 22 16:02:59 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
1423 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 16:05:04 CST 2017
done
merge后:686

不做任何配置:
Wed Feb 22 16:08:30 CST 2017
Before size:
1 /home/bone/elasticsearch-2.4.1/data/
OK, waiting...
0
After size:
2641 /home/bone/elasticsearch-2.4.1/data/
Wed Feb 22 16:11:54 CST 2017
done
merge后:
1650














本文转自张昺华-sky博客园博客,原文链接:http://www.cnblogs.com/bonelee/p/6430378.html,如需转载请自行联系原作者


版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

分享:
开发与运维
使用钉钉扫一扫加入圈子
+ 订阅

集结各类场景实战经验,助你开发运维畅行无忧

其他文章
最新文章
相关文章