Reroute Unassigned Shards——遇到主shard 出现的解决方法就是重新路由

本文涉及的产品
检索分析服务 Elasticsearch 版,2核4GB开发者规格 1个月
简介:

Red Cluster!

摘自:http://blog.kiyanpro.com/2016/03/06/elasticsearch/reroute-unassigned-shards/

There are 3 cluster states:

  1. green: All primary and replica shards are active
  2. yellow: All primary shards are active, but not all replica shards are active
  3. red: Not all primary shards are active

When cluster health is red, it means cluster is dead. And that means you can do nothing until it’s recovered, which is very bad indeed. I will share with you how to deal with one common situation: when cluster is red due to unassigned shards.

Steps

The general idea is pretty simple: find those shards which are unassigned, manually assign them to a node with reroute API. Let’s see how we can do that step by step. Then we can combine them into a configurable simple script.

Step 1: Check Unassigned Shards

To get cluster information, we usually use cat APIs. There is a GET /_cat/shards endpoint to show a detailed view of what nodes contain which shards[1].

Cat shards

1
2
3
4
5
6
7
8
9
# cat shards verbose
curl  "http://your.elasticsearch.host.com:9200/_cat/shards?v"
 
# cat shards index
curl  "http://your.elasticsearch.host.com:9200/_cat/shards/wiki2"
# example return
# wiki2 0 p STARTED 197 3.2mb 192.168.56.10 Stiletto
# wiki2 1 p STARTED 205 5.9mb 192.168.56.30 Frankie Raye
# wiki2 2 p STARTED 275 7.8mb 192.168.56.20 Commander Kraken

By piping cat shards to fgrep, we can get all unassigned shards.

Get unassigned shards

1
2
3
4
5
6
# cat shards with fgrep
curl  "http://your.elasticsearch.host.com:9200/_cat/shards" | fgrep UNASSIGNED
# example return
# wiki1 0 r UNASSIGNED ALLOCATION_FAILED
# wiki1 1 r UNASSIGNED ALLOCATION_FAILED
# wiki1 2 r UNASSIGNED ALLOCATION_FAILED

 

If you don’t want to deal with shell script, you can also find these unassigned shards using another endpoint POST /_flush/synced[2]. This endpoint is actually not just some information. It allows an administrator to initiate a synced flush manually. This can be particularly useful for a planned (rolling) cluster restart where you can stop indexing and don’t want to wait the default 5 minutes for idle indices to be sync-flushed automatically. It returns with a json response.

_flush/synced

1
curl -XPOST  "http://your.elasticsearch.host.com:9200/twitter/_flush/synced"

If there are failed shards in the response, we can iterate through a failures array to get all unassigned ones.

Example response with failed shards

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
"_shards": {
"total": 4,
"successful": 1,
"failed": 1
},
"twitter": {
"total": 4,
"successful": 3,
"failed": 1,
"failures": [
{
"shard": 1,
"reason": "unexpected error",
"routing": {
"state": "STARTED",
"primary": false,
"node": "SZNr2J_ORxKTLUCydGX4zA",
"relocating_node": null,
"shard": 1,
"index": "twitter"
}
}
]
}
}

 

Step 2: Reroute

The reroute command allows to explicitly execute a cluster reroute allocation command including specific commands[3] . An unassigned shard can be explicitly allocated on a specific node.

Reroute example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
curl -XPOST  'localhost:9200/_cluster/reroute' -d '{
"commands" : [ {
"move" :
{
"index" : "test", "shard" : 0,
"from_node" : "node1", "to_node" : "node2"
}
},
{
"allocate" : {
"index" : "test", "shard" : 1, "node" : "node3"
}
}
]
}'

There are 3 kinds of commands you can use:

move: Move a started shard from one node to another node. Accepts index and shard for index name and shard number, from_node for the node to move the shard from, and to_node for the node to move the shard to.

cancel: Cancel allocation of a shard (or recovery). Accepts index and shard for index name and shard number, and node for the node to cancel the shard allocation on. It also accepts allow_primary flag to explicitly specify that it is allowed to cancel allocation for a primary shard. This can be used to force resynchronization of existing replicas from the primary shard by cancelling them and allowing them to be reinitialized through the standard reallocation process.

allocate: Allocate an unassigned shard to a node. Accepts the index and shard for index name and shard number, and node to allocate the shard to. It also accepts allow_primary flag to explicitly specify that it is allowed to explicitly allocate a primary shard (might result in data loss).

Combining step 2 with the unassigned shards from Step 1, we can reroute all unassigned shards 1 by 1, thus getting faster cluster recovery from red state.

Example Solutions

Python

Below is a python script I wrote using POST /_flush/synced and POST /reroute

Shell Script

Below is a shell script I found elsewhere in a blog post[4]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
for shard in $(curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print $2}'); do
curl -XPOST  'localhost:9200/_cluster/reroute' -d '{
"commands" : [ {
"allocate" : {
"index" : "t37", # index name
"shard" : $shard,
"node" : "datanode15", # node name
"allow_primary" : true
}
}
]
}'
sleep 5
done

EDIT: Based on Vincent’s comment I updated the shell script:

Possible Unassigned Shard Reasons

FYI, these are the possible reasons for a shard be in a unassigned state[1]:

 

Name Comment
INDEX_CREATED Unassigned as a result of an API creation of an index
CLUSTER_RECOVERED Unassigned as a result of a full cluster recovery
INDEX_REOPENED Unassigned as a result of opening a closed index
DANGLING_INDEX_IMPORTED Unassigned as a result of importing a dangling index
NEW_INDEX_RESTORED Unassigned as a result of restoring into a new index
EXISTING_INDEX_RESTORED Unassigned as a result of restoring into a closed index
REPLICA_ADDED Unassigned as a result of explicit addition of a replica
ALLOCATION_FAILED Unassigned as a result of a failed allocation of the shard
NODE_LEFT Unassigned as a result of the node hosting it leaving the cluster
REROUTE_CANCELLED Unassigned as a result of explicit cancel reroute command
REINITIALIZED When a shard moves from started back to initializing, for example, with shadow replicas
REALLOCATED_REPLICA A better replica location is identified and causes the existing replica allocation to be cancelled

References

  1. ElasticSearch Document Cat Shards
  2. ElasticSearch Document Synced Flush
  3. ElasticSearch Document Cluster Reroute
  4. How to fix your elasticsearch cluster stuck in initializing shards mode?


















本文转自张昺华-sky博客园博客,原文链接:http://www.cnblogs.com/bonelee/p/7459408.html,如需转载请自行联系原作者

相关实践学习
使用阿里云Elasticsearch体验信息检索加速
通过创建登录阿里云Elasticsearch集群,使用DataWorks将MySQL数据同步至Elasticsearch,体验多条件检索效果,简单展示数据同步和信息检索加速的过程和操作。
ElasticSearch 入门精讲
ElasticSearch是一个开源的、基于Lucene的、分布式、高扩展、高实时的搜索与数据分析引擎。根据DB-Engines的排名显示,Elasticsearch是最受欢迎的企业搜索引擎,其次是Apache Solr(也是基于Lucene)。 ElasticSearch的实现原理主要分为以下几个步骤: 用户将数据提交到Elastic Search 数据库中 通过分词控制器去将对应的语句分词,将其权重和分词结果一并存入数据 当用户搜索数据时候,再根据权重将结果排名、打分 将返回结果呈现给用户 Elasticsearch可以用于搜索各种文档。它提供可扩展的搜索,具有接近实时的搜索,并支持多租户。
相关文章
|
12天前
|
API 索引
Elasticsearch Index Shard Allocation 索引分片分配策略
Elasticsearch Index Shard Allocation 索引分片分配策略
82 1
|
7月前
|
数据处理 数据库
08MyCat - 概念 - 分片节点、分片规则、全局序列号
08MyCat - 概念 - 分片节点、分片规则、全局序列号
24 0
|
7月前
|
NoSQL Redis 容器
Redis集群报错cluster_state:fail,如何解决并重新恢复集群(IP问题/ slot未完全分配问题)
Redis集群报错cluster_state:fail,如何解决并重新恢复集群(IP问题/ slot未完全分配问题)
133 0
|
存储 NoSQL MongoDB
路由节点进行分片操作|学习笔记
快速学习路由节点进行分片操作
211 0
路由节点进行分片操作|学习笔记
|
jenkins 持续交付 Perl
将pod 指定部署到特定节点(master)上的一种方法记录
将pod 指定部署到特定节点(master)上的一种方法记录 -nodeSelector 需求是这样的,我要搭一个集群,这个集群要走 CI/CD 流程,还要管理 CI/CD 流程的产出物,将其保存在 Harbor 中,让后让产出物在另外两个节点上运行起来。
3920 0
|
监控
日志服务支持Shard自动分裂
日志服务提供Shard自动分裂功能,实时监控Shard流量,自动进行shard分裂以应对流量上涨,解决流量估算不准、随时可能上涨又难以及时处理的问题,保障数据完整。
8902 0