修改索引的最大查询数量
不推荐使用该方案，因为治标不治本，虽然可以超过1W条，但是当数据量特别大的时候会直接报错。
PUT your_index_name
{
"settings": {
"max_result_window": "100000"
}
}
Scroll滚动查询
底层原理是快照，可以用于超过1W条导出的场景。
#scroll 快照数据不具备实时性 不支持跳页 每一个页面对应一个scroll_ID 可以用于导出
#scroll查询
GET analert/_search?scroll=5m
{
"from": 0,
"size": 10000,
"query": {
"match_all": {}
}
}
#scroll历史数据
GET _search/scroll
{
"scroll_id" : "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAEOE4Fll4RUxFSmpVU0VTamFoWVpKMGYwR3cAAAAAABDhOhZZeEVMRUpqVVNFU2phaFlaSjBmMEd3AAAAAAAQ4TkWWXhFTEVKalVTRVNqYWhZWkowZjBHdwAAAAAAEOE7Fll4RUxFSmpVU0VTamFoWVpKMGYwR3cAAAAAABDhNxZZeEVMRUpqVVNFU2phaFlaSjBmMEd3",
"scroll" : "5m"
}
SearchAfter深度分页查询
底层原理是游标，作用于1W条数据之后深度分页下的实时查询，比较推荐。
#search after 深度分页 无法跳页请求，索引有实时的增删改可以快速查阅
#每次修改查询条件之后需要重新查询1000条
#需要维护一个 search_after ID
GET analert/_search
{
"from": 0,
"size": 999,
"query": {
"match_all": {}
},
"sort": [
{
"_id": {
"order": "desc"
}
}
]
}
#search after实时数据
GET analert/_search
{
"from": 0,
"size": 1000,
"query": {
"match_all": {}
},
"search_after": ["zB7MKIgBnVHJ1eC2Vl9S"],
"sort": [
{
"_id": {
"order": "desc"
}
}
]
}
不能够修改query查询条件，不能够修改页数
存在的问题 : size * total 大小超过1W条，
最终落地接口设计
请求方式: POST请求
接口地址: xxx/xxx
入参: 请求体 JSON类型
{
"page": 1000,
"limit": 10,
"searchTimeType": "week",
"pageSearchAfterMap": {},
"lastSearchAfter": [],
"nextSearchAfter": [],
"pageAction": "skip"
}
返参:
{
"msg": "",
"code": "0",
"pageSearchAfterMap": {
"1001": [
1684722949,
"695QQYgBnVHJ1eC2PpZF"
]
},
"lastSearchAfter": [],
"data": [
{
"id": "6N5QQYgBnVHJ1eC2Wsfu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "6N5QQYgBnVHJ1eC2Wsbu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "6N5QQYgBnVHJ1eC2PpZF",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "6N5QQYgBnVHJ1eC2PpVE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "6N5QQYgBnVHJ1eC2PpRE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "6N5QQYgBnVHJ1eC2PpNE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "6N5QQYgBnVHJ1eC2PpJE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "695QQYgBnVHJ1eC2Wsfu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "695QQYgBnVHJ1eC2Wsbu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "695QQYgBnVHJ1eC2PpZF",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
}
],
"count": 57003,
"nextSearchAfter": [
1684722949,
"695QQYgBnVHJ1eC2PpZF"
]
}
第二次入参
{
"page": 1001,
"limit": 10,
"searchTimeType": "week",
"pageSearchAfterMap": {
"1001": [
1684722949,
"695QQYgBnVHJ1eC2PpZF"
]
},
"lastSearchAfter": [],
"nextSearchAfter": [
1684722949,
"695QQYgBnVHJ1eC2PpZF"
],
"pageAction": "page-after"
}
第二次返参
{
"msg": "",
"code": "0",
"pageSearchAfterMap": {
"1002": [
1684722949,
"5t5QQYgBnVHJ1eC2PpNE"
],
"1001": [
1684722949,
"695QQYgBnVHJ1eC2PpZF"
]
},
"lastSearchAfter": [
1684722949,
"695QQYgBnVHJ1eC2PpZF"
],
"data": [
{
"id": "695QQYgBnVHJ1eC2PpVE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "695QQYgBnVHJ1eC2PpRE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "695QQYgBnVHJ1eC2PpNE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "695QQYgBnVHJ1eC2PpJE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5t5QQYgBnVHJ1eC2Wsfu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5t5QQYgBnVHJ1eC2Wsbu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5t5QQYgBnVHJ1eC2PpZF",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5t5QQYgBnVHJ1eC2PpVE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5t5QQYgBnVHJ1eC2PpRE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5t5QQYgBnVHJ1eC2PpNE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
}
],
"count": 57003,
"nextSearchAfter": [
1684722949,
"5t5QQYgBnVHJ1eC2PpNE"
]
}
第三次入参
{
"page": 1002,
"limit": 10,
"searchTimeType": "week",
"pageSearchAfterMap": {
"1001": [
1684722949,
"695QQYgBnVHJ1eC2PpZF"
],
"1002": [
1684722949,
"5t5QQYgBnVHJ1eC2PpNE"
]
},
"lastSearchAfter": [
1684722949,
"695QQYgBnVHJ1eC2PpZF"
],
"nextSearchAfter": [
1684722949,
"5t5QQYgBnVHJ1eC2PpNE"
],
"pageAction": "page-after",
"_": 1684826142153,
"nonce": 48146043,
"sign": "fb5469d67b6a7c208f9604b4862f6c6b"
}
第三次返参
{
"msg": "",
"code": "0",
"pageSearchAfterMap": {
"1003": [
1684722949,
"5N5QQYgBnVHJ1eC2Wsbu"
],
"1002": [
1684722949,
"5t5QQYgBnVHJ1eC2PpNE"
],
"1001": [
1684722949,
"695QQYgBnVHJ1eC2PpZF"
]
},
"lastSearchAfter": [
1684722949,
"5t5QQYgBnVHJ1eC2PpNE"
],
"data": [
{
"id": "5t5QQYgBnVHJ1eC2PpJE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5d5QQYgBnVHJ1eC2Wsfu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5d5QQYgBnVHJ1eC2Wsbu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5d5QQYgBnVHJ1eC2PpZF",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5d5QQYgBnVHJ1eC2PpVE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5d5QQYgBnVHJ1eC2PpRE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5d5QQYgBnVHJ1eC2PpNE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5d5QQYgBnVHJ1eC2PpJE",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5N5QQYgBnVHJ1eC2Wsfu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
},
{
"id": "5N5QQYgBnVHJ1eC2Wsbu",
"srcIp": "4.41.51.216",
"area": "US-美国",
"alertTime": "2023-05-22 10:35:49",
"destIp": "2.2.2.2",
"repeatNum": "1",
"successRepeatNum": "1",
"eventName": "SYNFLOOD攻击",
"deviceId": "1"
}
],
"count": 57003,
"nextSearchAfter": [
1684722949,
"5N5QQYgBnVHJ1eC2Wsbu"
]
}
方案点评
该方案其实比较适合滚动查询的场景，比如说淘宝这种可以一直向下滚动的电商网站，或者当作一个临时方案来进行使用，在之后我们可以加入其他组件，比如Hbase使用rowkey关联的场景来查询其他更多的超过一万条的数据。
