搜索引擎蜘蛛给网站带来的危害，有效指引爬虫对应的措施[方法篇](禁止YisouSpider|EasouSpider|EtaoSpider)
上一遍文章《搜索引擎蜘蛛给网站带来的危害，有效指引爬虫对应的措施(最准确搜索引擎蜘蛛名称)》
下一篇“如何更高效发现&屏蔽搜索引擎蜘蛛”实例篇。
现在接着详细介绍该方法，包括如何发现蜘蛛，从网站的日志里统计蜘蛛，发现未知的蜘蛛。针对不同的蜘蛛，屏蔽蜘蛛、禁止爬虫怎么样更高效，更快捷。并且通过实例来介绍。
收集最新最准确搜索引擎蜘蛛IP
信息来源线上：最新最准确
根据线上空间的访问日志来整理常见的蜘蛛名称，不求最全，但力求爬虫信息资料最新最准确。以下搜索引擎蜘蛛名称都是根据线上空间日志亲手提取。
各种搜索引擎的蜘蛛爬虫会不断地访问抓取我们站点的内容，也会消耗站点的一定流量，有时候就需要屏蔽某些蜘蛛访问我们的站点。
其实有效常用的搜索引擎就那么几个，只要在robots.txt文件里把常用的几个搜索引擎蜘蛛允许放行就好了，其它的爬虫统统通过通配符（*）禁止掉，屏蔽某些蜘蛛。
YisouSpider,这货是哪个的蜘蛛？怎么封掉它？
答案：国内最疯狂的蜘蛛，能让你的网站挂掉。VPS、小内存虚拟机就别说了，它来十遭殃。
直接来有质量的内容
1.怎么抓住这个YisouSpider爬虫？
2.直接封YisouSpider方法。
3.检测配置有效性
1.怎么抓住这个YisouSpider爬虫？
Nginx为例，日志文件为
|
1
2
3
4
|
access.
log
cat
grep
access.
log
命令
cat access.
log
|
grep
-i
"YisouSpider"
| awk
'{print $1}'
> iplog.txt
|
将他的抓取ip日志保存起来。
分析统计，连续的IP，C段IP计算，这蜘蛛它的IP资源挺多啊！
|
1
|
cat iplog.txt | awk -F
'.'
'{print $1"."$2"."$3".0"}'
|
sort
| uniq -c |
sort
-r -n > stat_ip.
log
|
以下是搜索引擎蜘蛛爬虫IP地址，这里只是部分蜘蛛IP，不完整。
YisouSpider搜索引擎蜘蛛
iplog.txt
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
|
实例IP统计:
15370 110.75.176.51
15310 110.75.176.52
15100 110.75.176.53
15050 110.75.176.50
15040 110.75.176.54
14880 110.75.176.49
14810 118.195.65.244
14310 118.195.65.245
13560 118.195.65.242
13460 125.39.66.149
12760 125.39.66.148
12670 125.39.66.150
11300 211.138.121.37
10790 211.138.121.38
10610 125.39.66.146
10100 211.138.121.36
8880 221.176.88.90
8810 125.39.66.147
8470 118.195.65.246
6730 211.162.121.246
5560 221.176.14.6
4880 221.176.88.87
4540 118.195.65.243
2730 122.228.66.130
2560 122.228.66.134
2410 122.228.66.132
2370 221.176.88.106
2250 122.228.66.133
2030 221.176.88.78
1980 110.75.172.81
1960 221.176.88.104
1910 221.176.88.82
1880 221.176.88.86
1850 110.75.172.80
1840 122.228.66.135
1770 110.75.172.82
1650 221.176.88.89
1640 221.176.88.101
1640 110.75.172.77
1620 110.75.172.78
1600 221.176.88.81
1570 110.75.172.79
1560 221.176.88.98
1560 221.176.88.100
1540 221.176.88.88
1430 221.176.88.79
1110 119.233.255.60
1000 122.198.80.12
790 183.129.198.78
710 183.129.198.79
640 27.24.158.140
630 27.24.158.139
580 122.228.66.165
530 183.129.198.69
480 183.129.198.77
480 122.228.66.162
460 110.75.173.195
450 27.24.158.156
450 122.228.66.167
420 183.129.198.80
420 122.228.66.166
410 122.228.66.164
390 67.55.71.208
390 27.24.158.152
390 122.228.66.163
340 221.130.18.152
310 221.130.18.148
290 202.171.253.105
270 110.75.173.196
250 202.171.253.104
200 110.75.173.199
190 110.75.173.197
180 221.130.18.121
170 202.171.253.108
170 110.75.176.19
170 110.75.173.200
170 110.75.173.198
160 221.181.192.23
160 110.75.176.20
150 202.171.253.103
150 173.208.185.2
150 110.75.174.227
150 110.75.174.185
130 110.75.174.186
130 110.75.174.184
130 110.75.173.7
130 110.75.173.201
120 221.130.18.122
120 120.203.215.10
120 110.75.175.24
120 110.75.174.228
110 221.130.18.120
110 120.203.214.129
110 110.75.174.230
110 110.75.174.225
110 110.75.173.204
110 110.75.172.109
100 120.203.215.8
100 118.244.165.184
100 110.75.176.27
100 110.75.176.23
100 110.75.174.55
100 110.75.174.244
100 110.75.174.243
100 110.75.174.226
100 110.75.173.221
100 110.75.173.175
90 67.210.103.46
90 110.75.175.23
90 110.75.175.19
90 110.75.174.248
90 110.75.174.229
90 110.75.174.183
90 110.75.173.51
90 110.75.173.206
90 110.75.173.203
90 110.75.173.202
80 221.130.18.123
80 110.75.176.22
80 110.75.175.45
80 110.75.174.59
80 110.75.174.245
80 110.75.173.222
80 110.75.173.10
80 110.75.172.107
70 221.130.18.155
70 110.75.176.29
70 110.75.176.21
70 110.75.175.13
70 110.75.174.57
70 110.75.174.56
70 110.75.174.170
70 110.75.174.162
70 110.75.173.9
60 120.203.215.18
60 120.203.214.130
60 110.75.175.15
60 110.75.174.187
60 110.75.174.164
60 110.75.174.160
60 110.75.173.8
60 110.75.173.60
60 110.75.173.52
60 110.75.173.40
60 110.75.173.29
60 110.75.173.224
60 110.75.173.219
60 110.75.173.205
60 110.75.173.19
60 110.75.173.146
50 124.83.190.64
50 124.83.190.62
50 110.75.176.60
50 110.75.176.30
50 110.75.176.26
50 110.75.175.20
50 110.75.175.14
50 110.75.174.246
50 110.75.174.169
50 110.75.174.166
50 110.75.174.165
50 110.75.174.161
50 110.75.173.58
50 110.75.173.54
50 110.75.173.50
50 110.75.173.220
50 110.75.173.164
50 110.75.173.11
40 221.130.18.88
40 110.75.176.57
40 110.75.176.25
40 110.75.175.30
40 110.75.175.21
40 110.75.174.60
40 110.75.174.58
40 110.75.174.233
40 110.75.174.195
40 110.75.174.188
40 110.75.174.167
40 110.75.174.159
40 110.75.173.59
40 110.75.173.57
40 110.75.173.56
40 110.75.173.42
40 110.75.173.41
40 110.75.173.39
40 110.75.173.38
40 110.75.173.163
40 110.75.173.129
40 110.75.173.12
40 110.75.172.110
30 124.83.190.63
30 120.203.215.9
30 110.75.176.55
30 110.75.175.27
30 110.75.175.17
30 110.75.175.16
30 110.75.174.247
30 110.75.174.231
30 110.75.174.168
30 110.75.173.55
30 110.75.173.37
30 110.75.173.25
30 110.75.173.24
30 110.75.173.161
30 110.75.172.97
30 110.75.172.85
30 110.75.172.84
30 110.75.172.83
30 110.75.172.108
20 221.130.18.89
20 221.130.18.119
20 120.203.215.15
20 110.75.176.59
20 110.75.176.28
20 110.75.176.24
20 110.75.175.46
20 110.75.175.29
20 110.75.175.28
20 110.75.175.26
20 110.75.175.25
20 110.75.175.22
20 110.75.175.18
20 110.75.174.236
20 110.75.174.234
20 110.75.174.163
20 110.75.173.49
20 110.75.173.30
20 110.75.173.28
20 110.75.173.23
20 110.75.173.223
20 110.75.173.162
20 110.75.173.160
20 110.75.173.141
20 110.75.172.88
20 110.75.172.87
20 110.75.172.86
20 110.75.172.113
20 110.75.172.111
20 110.75.171.96
20 110.75.171.81
10 60.223.237.167
10 58.22.138.18
10 110.75.176.58
10 110.75.176.56
10 110.75.176.5
10 110.75.176.36
10 110.75.176.2
10 110.75.176.17
10 110.75.176.1
10 110.75.175.58
10 110.75.175.47
10 110.75.175.33
10 110.75.174.53
10 110.75.174.50
10 110.75.174.235
10 110.75.174.22
10 110.75.174.200
10 110.75.174.20
10 110.75.174.199
10 110.75.174.197
10 110.75.174.178
10 110.75.174.136
10 110.75.174.10
10 110.75.173.27
10 110.75.173.26
10 110.75.173.21
10 110.75.173.20
10 110.75.173.193
|