在企业级数据采集与分布式爬虫架构中,最隐蔽的线上故障并非进程崩溃,而是“进程正常存活、日志零报错,但主力目标域名的采集成功率已在后台悄然腰斩”。本文基于 Python aiohttp 与 prometheus_client 构建生产级异步埋点引擎,结合 Prometheus + Grafana 实现按域名拆分的实时成功率(首包与终态双口径)、P95 首字节延迟监控及 Alertmanager 告警闭环,提供单文件 100% 可直接运行的完整代码与排障手册。
一、环境依赖与基准清单
为确保代码与 PromQL 查询结果可直接复现,本文实战基于以下标准环境验证:
- 操作系统:Ubuntu 22.04 LTS / macOS 14+(生产环境建议提前执行
ulimit -n 65535放开文件描述符上限)。 - Python 版本:Python 3.11.8。
- 核心依赖安装:
pip install aiohttp==3.10.11 prometheus-client==0.21.0 - 可观测组件版本:Prometheus v2.53.x + Grafana v11.x(可通过 Docker 一键拉起)。
- 前置网络基准环境:为排除免费代理高丢包和节点假死对延迟分位数的失真干扰,本实战统一接入爬虫加强版隧道代理网关(固定入口
tn.16yun.cn,端口9020)。本地运行前只需配置环境变量TUNNEL_USER与TUNNEL_PASS,或替换代码中的默认测试凭据即可直接启动。
二、核心机制解析:为什么传统监控抓不到静默退化?
在实际生产排障中,传统基于日志文件或总成功率的统计往往存在三个结构性盲区:
- 重试机制掩盖真实链路恶化:只统计
最终成功任务数 / 总任务数会掩盖代理节点质量下降的事实。必须将指标拆解为首包成功率(不含重试的首次请求命中率)与终态成功率(重试兜底后的交付率)。两者之差即“重试挽救率”,一旦持续超过 10%,说明系统正在用翻倍的请求开销硬扛风控拦截。 - HTTP 200 假成功泡沫:部分目标站触发风控后返回状态码 200 配合验证提示页面。必须在埋点阶段执行“状态码 200 + 前 4KB 关键字拦截检测 + 核心载荷校验”三层过滤。
- 均值耗时掩盖 P95 长尾阻塞与分桶失真:利用
aiohttp.TraceConfig在请求生命周期钩子中精准拆出 TCP 建连时间与首字节耗时(TTFB)。同时针对隧道代理 TTFB 多分布在 0.8 秒至 3.5 秒的特性,必须在 Histogram 中加密该区间分桶,避免histogram_quantile(0.95, ...)插值结果失真。
在标签设计上严格遵守低基数原则,只保留清洗后的主域名 domain、attempt_type 和按处置动作划分的七类 err_type(timeout、conn_reset、tls_error、auth_error、http_4xx、http_5xx、blocked),严禁将完整 URL 或动态代理 IP 放入 Label。
三、完整工程实战(单文件可运行代码与配置)
3.1 Python 异步监控埋点与 /metrics 服务(crawler_health_exporter.py)
将下列代码保存为 crawler_health_exporter.py,配置隧道代理凭据后执行 python crawler_health_exporter.py 即可在 9108 端口暴露 Prometheus 指标流:
import asyncio
import os
import time
from typing import Optional
from urllib.parse import urlparse
import aiohttp
from aiohttp import web
from prometheus_client import CONTENT_TYPE_LATEST, Counter, Gauge, Histogram, generate_latest
# ================= 代理网关与服务配置(16YUN爬虫代理) =================
PROXY_HOST = os.getenv("TUNNEL_HOST", "tn.16yun.cn")
PROXY_PORT = int(os.getenv("TUNNEL_PORT", "9020"))
PROXY_USER = os.getenv("TUNNEL_USER", "your_username")
PROXY_PASS = os.getenv("TUNNEL_PASS", "your_password")
PROXY_URL = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
METRICS_PORT = int(os.getenv("METRICS_PORT", "9108"))
# ================= Prometheus 指标声明 =================
REQ_TOTAL = Counter(
"crawler_requests_total",
"Total crawler requests by domain, attempt_type, and result",
["domain", "attempt_type", "result"],
)
TTFB_HISTOGRAM = Histogram(
"crawler_ttfb_seconds",
"Time to first byte in seconds by domain",
["domain"],
buckets=[0.1, 0.3, 0.5, 0.8, 1.0, 1.2, 1.5, 2.0, 2.5, 3.0, 4.0, 6.0, 10.0, 15.0],
)
TCP_HISTOGRAM = Histogram(
"crawler_proxy_tcp_seconds",
"TCP connection setup time to proxy gateway in seconds",
["domain"],
buckets=[0.01, 0.03, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0],
)
ERR_TOTAL = Counter(
"crawler_errors_total",
"Total crawler errors classified by actionable err_type",
["domain", "err_type"],
)
INFLIGHT_GAUGE = Gauge(
"crawler_inflight_requests",
"Current in-flight requests per domain",
["domain"],
)
def is_real_success(status: int, body: bytes, url: str) -> bool:
"""三层校验:HTTP 状态码 -> 反爬特征关键字 -> 业务接口格式"""
if status != 200 or not body:
return False
text = body[:4096].decode("utf-8", errors="ignore").lower()
block_keywords = (
"操作频繁",
"请稍后再试",
"访问验证",
"滑块",
"captcha",
"robot check",
"unusual traffic",
)
if any(kw in text for kw in block_keywords):
return False
if "/api/" in url and "{" not in text:
return False
return True
def build_trace_config(timing_bag: dict) -> aiohttp.TraceConfig:
"""每次请求构造独立 TraceConfig,捕获 TCP 建连耗时与 TTFB"""
tc = aiohttp.TraceConfig()
async def on_request_start(session, ctx, params):
ctx.req_start = time.perf_counter()
async def on_conn_start(session, ctx, params):
ctx.conn_start = time.perf_counter()
async def on_conn_end(session, ctx, params):
if hasattr(ctx, "conn_start"):
timing_bag["tcp_sec"] = time.perf_counter() - ctx.conn_start
async def on_headers_received(session, ctx, params):
if hasattr(ctx, "req_start"):
timing_bag["ttfb_sec"] = time.perf_counter() - ctx.req_start
tc.on_request_start.append(on_request_start)
tc.on_connection_create_start.append(on_conn_start)
tc.on_connection_create_end.append(on_conn_end)
tc.on_response_headers_received.append(on_headers_received)
return tc
async def fetch_with_metrics(
session: aiohttp.ClientSession,
url: str,
max_retry: int = 2,
timeout_sec: int = 15,
) -> Optional[bytes]:
"""带全链路指标埋点与退避重试的异步抓取函数"""
domain = urlparse(url).netloc or "unknown"
for attempt in range(1, max_retry + 1):
attempt_type = "first" if attempt == 1 else "retry"
timing_bag = {
}
trace_cfg = build_trace_config(timing_bag)
INFLIGHT_GAUGE.labels(domain=domain).inc()
try:
async with session.get(
url,
proxy=PROXY_URL,
timeout=aiohttp.ClientTimeout(total=timeout_sec),
trace_configs=[trace_cfg],
headers={
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"},
) as resp:
body = await resp.read()
if "tcp_sec" in timing_bag:
TCP_HISTOGRAM.labels(domain=domain).observe(timing_bag["tcp_sec"])
if "ttfb_sec" in timing_bag:
TTFB_HISTOGRAM.labels(domain=domain).observe(timing_bag["ttfb_sec"])
if is_real_success(resp.status, body, url):
REQ_TOTAL.labels(domain=domain, attempt_type=attempt_type, result="success").inc()
return body
err_type = "blocked" if resp.status == 200 else f"http_{resp.status // 100}xx"
REQ_TOTAL.labels(domain=domain, attempt_type=attempt_type, result="fail").inc()
ERR_TOTAL.labels(domain=domain, err_type=err_type).inc()
if resp.status == 429:
await asyncio.sleep(min(2 ** attempt, 8))
except asyncio.TimeoutError:
REQ_TOTAL.labels(domain=domain, attempt_type=attempt_type, result="fail").inc()
ERR_TOTAL.labels(domain=domain, err_type="timeout").inc()
except aiohttp.ClientProxyConnectionError:
REQ_TOTAL.labels(domain=domain, attempt_type=attempt_type, result="fail").inc()
ERR_TOTAL.labels(domain=domain, err_type="auth_error").inc()
break
except aiohttp.ClientSSLError:
REQ_TOTAL.labels(domain=domain, attempt_type=attempt_type, result="fail").inc()
ERR_TOTAL.labels(domain=domain, err_type="tls_error").inc()
except aiohttp.ClientConnectorError:
REQ_TOTAL.labels(domain=domain, attempt_type=attempt_type, result="fail").inc()
ERR_TOTAL.labels(domain=domain, err_type="conn_reset").inc()
except Exception as exc:
REQ_TOTAL.labels(domain=domain, attempt_type=attempt_type, result="fail").inc()
ERR_TOTAL.labels(domain=domain, err_type=f"other_{type(exc).__name__}").inc()
finally:
INFLIGHT_GAUGE.labels(domain=domain).dec()
return None
async def metrics_handler(request: web.Request) -> web.Response:
"""异步暴露 /metrics 给 Prometheus 拉取"""
return web.Response(body=generate_latest(), headers={
"Content-Type": CONTENT_TYPE_LATEST})
async def main():
app = web.Application()
app.router.add_get("/metrics", metrics_handler)
runner = web.AppRunner(app)
await runner.setup()
site = web.TCPSite(runner, "0.0.0.0", METRICS_PORT)
await site.start()
print(f"[Exporter 启动] 指标端口已监听 http://127.0.0.1:{METRICS_PORT}/metrics")
test_urls = [
"https://httpbin.org/ip",
"https://httpbin.org/headers",
"https://httpbin.org/delay/1",
]
connector = aiohttp.TCPConnector(limit=100, ttl_dns_cache=300)
async with aiohttp.ClientSession(connector=connector) as session:
for round_idx in range(1, 10):
results = await asyncio.gather(*(fetch_with_metrics(session, u) for u in test_urls))
ok_count = sum(1 for item in results if item is not None)
print(f"[采集轮次 {round_idx}] 成功 {ok_count}/{len(test_urls)}")
await asyncio.sleep(5)
await runner.cleanup()
if __name__ == "__main__":
asyncio.run(main())
3.2 Prometheus 抓取配置(prometheus.yml)与告警规则(rules.yml)
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "rules.yml"
scrape_configs:
- job_name: "crawler_health"
metrics_path: "/metrics"
static_configs:
- targets: ["host.docker.internal:9108"]
对应的 rules.yml 核心生产告警规则(精简至四条高信噪比规则):
groups:
- name: crawler_health_alerts
rules:
- alert: CrawlerProxyAuthError
expr: increase(crawler_errors_total{
err_type="auth_error"}[5m]) > 0
for: 0m
labels:
severity: critical
annotations:
summary: "代理网关鉴权或并发异常 ({
{ $labels.domain }})"
description: "检测到 auth_error,请检查白名单、账密或套餐并发上限。"
- alert: CrawlerFirstSuccessRateLow
expr: >
sum by (domain) (rate(crawler_requests_total{attempt_type="first", result="success"}[5m]))
/
sum by (domain) (rate(crawler_requests_total{attempt_type="first"}[5m])) < 0.90
for: 5m
labels:
severity: critical
annotations:
summary: "域名首包成功率跌破 90% ({
{ $labels.domain }})"
- alert: CrawlerRetrySaveRatioHigh
expr: >
(sum by (domain) (rate(crawler_requests_total{result="success"}[10m]))
- sum by (domain) (rate(crawler_requests_total{attempt_type="first", result="success"}[10m])))
/
sum by (domain) (rate(crawler_requests_total{attempt_type="first"}[10m])) > 0.10
for: 10m
labels:
severity: warning
annotations:
summary: "重试挽救率超过 10% ({
{ $labels.domain }})"
- alert: CrawlerP95LatencyHigh
expr: histogram_quantile(0.95, sum by (le, domain) (rate(crawler_ttfb_seconds_bucket[5m]))) > 4.5
for: 10m
labels:
severity: warning
annotations:
summary: "域名 P95 首字节延迟超过 4.5s ({
{ $labels.domain }})"
3.3 Grafana 看板核心 PromQL 面板
- 各域名首包成功率:
sum by (domain) (rate(crawler_requests_total{attempt_type="first", result="success"}[5m])) / sum by (domain) (rate(crawler_requests_total{attempt_type="first"}[5m])) - 各域名终态成功率:
sum by (domain) (rate(crawler_requests_total{result="success"}[5m])) / sum by (domain) (rate(crawler_requests_total{attempt_type="first"}[5m])) - 各域名 P95 首字节延迟(单位毫秒):
histogram_quantile(0.95, sum by (le, domain) (rate(crawler_ttfb_seconds_bucket[5m]))) * 1000 - 各域名异常分类速率:
sum by (domain, err_type) (rate(crawler_errors_total[5m]))
四、常见报错与生产排障
- Grafana 面板无数据或分位数计算返回
NaN:优先检查rate(...[5m])窗口内是否有足够请求样本,当 QPS 极低(例如每分钟仅 1 至 2 次请求)时,[5m]窗口可能采不到两个点导致rate为 0。低频任务建议改用increase(...[15m])或调大抓取评估窗口。 - Prometheus 内存持续飙升甚至 OOM:检查是否误将包含 Query 参数的完整 URL、任务 UUID 或代理节点 IP 写入了指标标签。上线前可通过访问
/metrics接口核对暴露的序列行数,正常情况下单机指标总行数应控制在几百行以内。 - 多进程爬虫模式下部分指标跳变丢失:
prometheus_client默认将计数保存在单进程内存中。如果采用multiprocessing启动多个 Worker,必须在进程启动前设置环境变量PROMETHEUS_MULTIPROC_DIR并使用MultiProcessCollector,或者让每个 Worker 绑定独立递增端口交由 Prometheus 服务发现拉取。 - TTFB 曲线异常偏高或与
total_ms倒挂:检查是否在全局复用了同一个aiohttp.TraceConfig实例。高并发协程共享同一个TraceConfig会覆盖ctx.req_start时间戳,务必保证每次session.get都通过工厂函数新建独立实例。
总结与架构权衡
Prometheus + Grafana 搭建爬虫健康度监控的关键,在于将观测维度收口到“低基数域名标签 + 首包/终态双口径成功率 + 加密分桶 P95 TTFB”。在实际选型时,中小规模异步任务推荐采用文中同事件循环暴露 /metrics 的零额外开销架构;当集群规模扩展至跨机房多 Worker 时,建议结合服务发现或 VictoriaMetrics 长期存储底座,保障高并发下的持续可观测性。