Elastic Stack学习--elasticsearch部署常见问题

本文涉及的产品
检索分析服务 Elasticsearch 版,2核4GB开发者规格 1个月
文本翻译,文本翻译 100万字符
图片翻译,图片翻译 100张
简介: linux内核版本低于3.5,不支持seccomp; [2018-03-12T11:56:52,328][WARN ][o.e.b.JNANatives ] unable to install syscall filter: java.

linux内核版本低于3.5,不支持seccomp;

[2018-03-12T11:56:52,328][WARN ][o.e.b.JNANatives         ] unable to install syscall filter:
java.lang.UnsupportedOperationException: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
        at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:328) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:616) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:258) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:110) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.2.jar:6.2.2]

解决办法:
修改elasticsearch.yaml文件,设置禁用seccomp;

bootstrap.system_call_filter: false

官方文档参考

用户没有权限使用mlockall;

[2018-03-12T11:56:52,328][WARN ][o.e.b.JNANatives         ] unable to install syscall filter:
java.lang.UnsupportedOperationException: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
        at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:328) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:616) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:258) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:110) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.2.jar:6.2.2]
[2018-03-12T11:56:52,359][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-03-12T11:56:52,359][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2018-03-12T11:56:52,359][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2018-03-12T11:56:52,359][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'work' mlockall
        work soft memlock unlimited
        work hard memlock unlimited
[2018-03-12T11:56:52,359][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.

解决办法:
使用root用户修改/etc/security/limits.conf文件:

work soft memlock unlimited
work hard memlock unlimited

官方文档参考

最大文件句柄数设置过低;

解决办法:
使用root用户修改/etc/security/limits.conf文件:

work soft nofile 65536
work hard nofile 65536

官方文档参考

最大线程数设置过低;

解决办法:
使用root用户修改/etc/security/limits.conf文件:

work soft nproc 4096
work hard nproc 4096

官方文档参考

vm.max_map_count设置过低

[2018-03-12T14:25:44,413][WARN ][o.e.b.BootstrapChecks    ] [yf-beidou-dmp00.yf01.baidu.com] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

解决办法:
1)执行如下命令,实时生效:

sysctl -w vm.max_map_count=262144

2)修改/etc/sysctl.conf,重启后生效:

vm.max_map_count = 262144

官方文档参考

注:要使 limits.conf 文件配置生效,必须要确保 pam_limits.so 文件被加入到启动文件中。查看 /etc/pam.d/login 文件中有:session required /lib/security/pam_limits.so这一行配置;

安装x-pack重启后报错:Failed to create native process factories for Machine Learning

org.elasticsearch.ElasticsearchException: Failed to create native process factories for Machine Learning
        at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:422) ~[?:?]
        at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:373) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$7(Node.java:397) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.node.Node$$Lambda$1374/1560391896.apply(Unknown Source) ~[?:?]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_45]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_45]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512) ~[?:1.8.0_45]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502) ~[?:1.8.0_45]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_45]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_45]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_45]
        at org.elasticsearch.node.Node.<init>(Node.java:400) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.2.jar:6.2.2]

这是由于x-pack中的machine learning功能依赖于2.9以上版本GLIBC,环境中缺乏glibc库所致。可通过如下命令定位问题:

${ES_HOME}/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller --version

输出缺少依赖的日志如下:

./controller: /lib64/tls/libc.so.6: version `GLIBC_2.4' not found (required by ./controller)
./controller: /lib64/tls/libc.so.6: version `GLIBC_2.4' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlCore.so)
./controller: /lib64/tls/libc.so.6: version `GLIBC_2.7' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlCore.so)
./controller: /lib64/tls/libpthread.so.0: version `GLIBC_2.4' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libapr-1.so.0)
./controller: /lib64/tls/libc.so.6: version `GLIBC_2.9' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libapr-1.so.0)
./controller: /lib64/tls/libc.so.6: version `GLIBC_2.7' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libxml2.so.2)

或者通过ldd命令查看controller的依赖:

ldd ${ES_HOME}/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/controller

输出日志如下:

./controller: /lib64/tls/libc.so.6: version `GLIBC_2.4' not found (required by ./controller)
./controller: /lib64/tls/libc.so.6: version `GLIBC_2.4' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlCore.so)
./controller: /lib64/tls/libc.so.6: version `GLIBC_2.7' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlCore.so)
./controller: /lib64/tls/libpthread.so.0: version `GLIBC_2.4' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libapr-1.so.0)
./controller: /lib64/tls/libc.so.6: version `GLIBC_2.9' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libapr-1.so.0)
./controller: /lib64/tls/libc.so.6: version `GLIBC_2.7' not found (required by /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libxml2.so.2)
    libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x00007fb285c82000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fb285b7f000)
    librt.so.1 => /lib64/tls/librt.so.1 (0x00007fb285a65000)
    liblog4cxx.so.10 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/liblog4cxx.so.10 (0x00007fb28567e000)
    libboost_program_options-gcc62-mt-1_65_1.so.1.65.1 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/libboost_program_options-gcc62-mt-1_65_1.so.1.65.1 (0x00007fb2853fd000)
    libMlCore.so => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlCore.so (0x00007fb2850d1000)
    libstdc++.so.6 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/libstdc++.so.6 (0x00007fb284d27000)
    libm.so.6 => /lib64/tls/libm.so.6 (0x00007fb284ba1000)
    libgcc_s.so.1 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/libgcc_s.so.1 (0x00007fb28498a000)
    libc.so.6 => /lib64/tls/libc.so.6 (0x00007fb284756000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb285d97000)
    libaprutil-1.so.0 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libaprutil-1.so.0 (0x00007fb28452f000)
    libexpat.so.0 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libexpat.so.0 (0x00007fb284304000)
    libapr-1.so.0 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libapr-1.so.0 (0x00007fb2840ce000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fb283f9a000)
    libxml2.so.2 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libxml2.so.2 (0x00007fb283c20000)
    libz.so.1 => /usr/lib64/libz.so.1 (0x00007fb283b0d000)
    libboost_regex-gcc62-mt-1_65_1.so.1.65.1 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libboost_regex-gcc62-mt-1_65_1.so.1.65.1 (0x00007fb283815000)
    libboost_iostreams-gcc62-mt-1_65_1.so.1.65.1 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libboost_iostreams-gcc62-mt-1_65_1.so.1.65.1 (0x00007fb283600000)
    libboost_filesystem-gcc62-mt-1_65_1.so.1.65.1 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libboost_filesystem-gcc62-mt-1_65_1.so.1.65.1 (0x00007fb2833e5000)
    libboost_system-gcc62-mt-1_65_1.so.1.65.1 => /home/work/elasticsearch-6.2.2/plugins/x-pack/x-pack-ml/platform/linux-x86_64/bin/../lib/./libboost_system-gcc62-mt-1_65_1.so.1.65.1 (0x00007fb2831e1000)

解决办法1:如果不需要使用machine learning功能,则可以在elasticsearch.yml中设置禁用:

xpack.ml.enabled: false

解决办法2:安装或者升级glibc;如果使用centos4.3版本,则果断升级操作系统;

参考官方论坛

如何修改临时文件目录

elasticsearch以及x-pack插件的运行,依赖于ES_TMPDIR环境变量的值作为临时文件目录;如果未设置,则默认为/tmp/elasticsearch;可通过在.bashrc或者.bash_profile文件中添加环境变量显式设置临时文件目录:

export ES_HOME=/home/work/elasticsearch-6.2.2
export ES_TMPDIR="${ES_HOME}/tmp"
相关实践学习
使用阿里云Elasticsearch体验信息检索加速
通过创建登录阿里云Elasticsearch集群,使用DataWorks将MySQL数据同步至Elasticsearch,体验多条件检索效果,简单展示数据同步和信息检索加速的过程和操作。
ElasticSearch 入门精讲
ElasticSearch是一个开源的、基于Lucene的、分布式、高扩展、高实时的搜索与数据分析引擎。根据DB-Engines的排名显示,Elasticsearch是最受欢迎的企业搜索引擎,其次是Apache Solr(也是基于Lucene)。 ElasticSearch的实现原理主要分为以下几个步骤: 用户将数据提交到Elastic Search 数据库中 通过分词控制器去将对应的语句分词,将其权重和分词结果一并存入数据 当用户搜索数据时候,再根据权重将结果排名、打分 将返回结果呈现给用户 Elasticsearch可以用于搜索各种文档。它提供可扩展的搜索,具有接近实时的搜索,并支持多租户。
目录
相关文章
|
26天前
|
自然语言处理 Java 网络架构
elasticsearch学习三:elasticsearch-ik分词器的自定义配置 分词内容
这篇文章是关于如何自定义Elasticsearch的ik分词器配置以满足特定的中文分词需求。
108 0
elasticsearch学习三:elasticsearch-ik分词器的自定义配置 分词内容
|
26天前
|
JSON Java 网络架构
elasticsearch学习四:使用springboot整合 rest 进行搭建elasticsearch服务
这篇文章介绍了如何使用Spring Boot整合REST方式来搭建和操作Elasticsearch服务。
103 4
elasticsearch学习四:使用springboot整合 rest 进行搭建elasticsearch服务
|
26天前
|
自然语言处理 搜索推荐 关系型数据库
elasticsearch学习六:学习 全文搜索引擎 elasticsearch的语法,使用kibana进行模拟测试(持续更新学习)
这篇文章是关于Elasticsearch全文搜索引擎的学习指南,涵盖了基本概念、命令风格、索引操作、分词器使用,以及数据的增加、修改、删除和查询等操作。
17 0
elasticsearch学习六:学习 全文搜索引擎 elasticsearch的语法,使用kibana进行模拟测试(持续更新学习)
|
26天前
|
Web App开发 JavaScript Java
elasticsearch学习五:springboot整合 rest 操作elasticsearch的 实际案例操作,编写搜索的前后端,爬取京东数据到elasticsearch中。
这篇文章是关于如何使用Spring Boot整合Elasticsearch,并通过REST客户端操作Elasticsearch,实现一个简单的搜索前后端,以及如何爬取京东数据到Elasticsearch的案例教程。
156 0
elasticsearch学习五:springboot整合 rest 操作elasticsearch的 实际案例操作,编写搜索的前后端,爬取京东数据到elasticsearch中。
|
26天前
|
自然语言处理 Java Maven
elasticsearch学习二:使用springboot整合TransportClient 进行搭建elasticsearch服务
这篇博客介绍了如何使用Spring Boot整合TransportClient搭建Elasticsearch服务,包括项目创建、Maven依赖、业务代码和测试示例。
76 0
elasticsearch学习二:使用springboot整合TransportClient 进行搭建elasticsearch服务
|
26天前
|
存储 JSON Java
elasticsearch学习一:了解 ES,版本之间的对应。安装elasticsearch,kibana,head插件、elasticsearch-ik分词器。
这篇文章是关于Elasticsearch的学习指南,包括了解Elasticsearch、版本对应、安装运行Elasticsearch和Kibana、安装head插件和elasticsearch-ik分词器的步骤。
92 0
elasticsearch学习一:了解 ES,版本之间的对应。安装elasticsearch,kibana,head插件、elasticsearch-ik分词器。
|
28天前
|
SQL 分布式计算 大数据
大数据-168 Elasticsearch 单机云服务器部署运行 详细流程
大数据-168 Elasticsearch 单机云服务器部署运行 详细流程
51 2
|
2月前
|
NoSQL 关系型数据库 Redis
mall在linux环境下的部署(基于Docker容器),Docker安装mysql、redis、nginx、rabbitmq、elasticsearch、logstash、kibana、mongo
mall在linux环境下的部署(基于Docker容器),docker安装mysql、redis、nginx、rabbitmq、elasticsearch、logstash、kibana、mongodb、minio详细教程,拉取镜像、运行容器
mall在linux环境下的部署(基于Docker容器),Docker安装mysql、redis、nginx、rabbitmq、elasticsearch、logstash、kibana、mongo
|
3月前
|
Kubernetes 容器 Perl
k8s 部署 elasticsearch 单节点
k8s 部署 elasticsearch 单节点
|
3月前
|
JSON 测试技术 API
黑马商城 Elasticsearch从入门到部署 RestClient操作文档
这篇文章详细介绍了如何使用Java的RestHighLevelClient客户端与Elasticsearch进行文档操作,包括新增、查询、删除、修改文档以及批量导入文档的方法,并提供了相应的代码示例和操作步骤。