CloudLens 基于日志服务构建统一的云产品可观测能力,通过日志、指标、配置计量等数据的关联分析,提供阿里云产品的用量分析、性能监控、安全分析、数据保护、异常检测、访问分析等服务。从成本、性能、安全、数据保护、稳定性、访问分析六个维度,助力企业快速构建云产品的可观测性,更好地使用云产品。
CloudLens for OSS介绍
日志服务联合阿里云OSS推出CloudLens for OSS,支持Bucket粒度的统一管理视图,支持资源用量、访问分析、异常检测、安全分析等可视化分析能力,提供场景化运维管理,实现Bucket资产的可观测性。主要功能如下:
- 接入管理:支持集中管理所有Bucket访问明细日志的采集状态;支持集中管理所有的日志存储库
- 查询分析:支持实时查询与分析OSS访问明细日志和计量日志。
- 异常监测:支持按照Bucket维度统一配置关键的异常检测告警。
- 报表中心:支持内置Bucket核心指标的可视化报表。
- 自定义仪表盘:支持集中管理Project下自定义仪表盘。
应用入口
CloudLens for OSS 当前已集成到阿里云控制台首页 与 OSS控制台内,您可以分别通过OSS、SLS或者控制台首页三个入口进入。
控制台首页
阿里云控制台首页入口:登录控制台首页后在云产品可观测tab查看,接入管理支持日志的开通,首页链接
OSS控制台
OSS控制台入口:点击OSS控制台左侧资源用量tab即可直接使用,控制台链接
SLS 控制台
SLS控制台入口:日志应用->云产品Lens->CloudLens for OSS,入口链接
日志类型
CloudLens默认免费开通中心化计量日志及时序监控数据,自开通之日起,可免费查询60天的数据;访问明细日志(最近7天免费,限额900GB/天的日志写入额度(如果一条日志为1KB,约为9亿条))的查询分析。
日志类型 |
说明 |
访问明细日志 |
记录相关OSS Bucket的所有访问日志及批量删除日志时具体的删除信息,实时采集,同时也包含计量日志。 |
计量日志 |
记录特定OSS Bucket每小时累计的计量数据。延迟时间为小时级别,用于辅助分析,默认免费开通。 |
时序监控数据 |
记录10秒粒度的请求、延迟、流量等指标及5分钟粒度的请求、流量等指标。默认免费开通。 |
报表中心全新升级
基于三大类日志孵化出资源用量、访问分析、安全分析三类报表,对应报表项均可预览查询语句,方便构建自定义报表、监控。
资源用量
报表项 |
依赖日志 |
报表说明 |
OSS用量总览 |
计量日志、时序监控数据 |
提供Bucket存储、流量等用量总览、趋势以及TOP统计分析 |
存储用量详情 |
计量日志 |
提供计量相关的Bucket用量趋势、地域分布 |
流量带宽用量详情 |
时序监控数据 |
提供Bucket流量类趋势、TOP统计 |
请求用量 |
提供Bucket读写请求趋势、地域分布以及TOP统计 |
访问分析
报表项 |
依赖日志 |
报表说明 |
Top分析 |
时序监控数据 |
提供Bucket上传/下载/删除数据量Top统计、流量、请求分析 |
访问分析 |
访问明细日志 |
提供Bucket内外网流量分布 |
安全分析
安全分析报表中心基于OSS Bucket资产数据 结合访问明细日志对可能存在风险(公共读写、为配置数据加密、未开启防盗链等属性)Bucket进行标识,可视化统计文件删除等高危操作。
自定义报表集中管理
当内置报表中心报表满足不了业务需求时,可以使用自定义仪表盘选择账号下任意Project下仪表盘,导入后到CloudLens for OSS内集中管理。
全新API助力OSS日志隔离
上述介绍了基于CloudLens for OSS的基本功能,自定义仪表盘可以导入Region化Project内自定义构建的报表,如果有中心化需求或者数据隔离需求应该如何操作?统一接入API可以帮助您实现。
统一接入API基于规则策略实现自动化采集云产品日志,通过全选、属性、实例三种模式实现精细化日志采集控制,其高级功能上也支持数据中心化汇总、数据实例粒度隔离、数据跨境合规等需求。目前统一接入API已支持OSS 访问明细日志的规则配置,通过数据加工流转,使得不同bucket的访问日志按照业务属性划分投递到不同的logstore下,对于不同的业务团队,给予不同logstore的RAM访问权限,从而实现了访问日志的权限隔离。
- 关于统一接入API的更多功能可参考:统一接入API赋能开发者:自动高效、灵活编排的云产品日志采集方案
- 接口说明:API说明文档
访问明细日志AK安全审计实践
越来越多的企业对AK的安全访问提出要求,通过开启Bucket的访问明细日志可轻松构建阿里云账号下所有AK 对 Bucket/Object操作视图。构建AK安全审计报表之前需要先开通Bucket的访问明细日志,访问明细日志是按地域纬度存储在账号下oss-log-{uid}-{region} / oss-log-store 目标库下,可针对所需监控地域的目标库构建如下报表。(如需监控全地域审计报表或者多地域中心化报表,可通过统一接入API开通)
操作步骤
- 开通CloudLens for OSS
- 开通Bucket访问明细日志,两种方式任选
- 单Region操作:控制台操作
- CloudLens for OSS控制台左侧导航栏-> 接入管理 -> 勾选Bucket 访问明细日志开通
- 中心化需求:统一接入API -> UpsertCollectionPolicy 配置规则
- resourceMode: all,默认开启账号下所有Bucket访问明细日志
- centralizeEnable : true ,启用跨域中心投递
- centralizeConfig 中dstProject 配置中心化投递的目标库,建议logstore设置为oss-log-store
- 进入访问明细日志目标存储库
报表分为操作概览、查询审计、更新操作审计、删除操作审计四个部分
操作概览
操作概览提供Bucket纬度操作全览,基于Bucket资产构建天纬度的操作(读/更新/删除)趋势图,并列出所有Bucket内文件操作详细。过滤器配置好后可针对AK,Region,Bucket做到更精细化审计。
构建报表所需SQL如下:
- 公共读写Bucket数量
*|SELECTCOUNT(DISTINCT id)as"公共读写Bucket数量"from"resource.sls.cmdb.oss"WHERE released_flag ='0'and grant ='public-read-write'AND region LIKE'${{region|%}}'AND id LIKE'${{bucket|%}}'limit1000
- 公共读Bucket总数
*|SELECTCOUNT(DISTINCT id)as"公共读Bucket数量"from"resource.sls.cmdb.oss"WHERE released_flag ='0'and grant ='public-read'AND region LIKE'${{region|%}}'AND id LIKE'${{bucket|%}}'limit1000
- 未配置数据加密Bucket数量
*|SELECTCOUNT(DISTINCT id)as"未配置数据加密Bucket数量"from"resource.sls.cmdb.oss"WHERE released_flag ='0'and sse_algorithm =''AND region LIKE'${{region|%}}'AND id LIKE'${{bucket|%}}'limit1000
- 操作统计趋势图
*|select date_format(__time__,'%Y-%m-%d')astime, sum("查询")as"查询", sum("删除")as"删除", sum("更新")as"更新"from(select __time__, access_id, bucket,CASE WHEN operation in('DeleteObject','DeleteObjects') then 1 else 0 end as"删除",CASE WHEN http_method in('GET','HEAD') then 1 else 0 end as"查询",CASE WHEN http_method in('PUT','POST','DELETE') then 1 else 0 end as"更新"from log)where access_id like'${{access_id|%}}'and bucket LIKE'${{bucket|%}}'groupbytimeorderbytime
- 文件操作统计列表
(__topic__: oss_access_log or __topic__: oss_batch_delete_log)and(OPERATION:"DeleteObject"OR OPERATION:PutObject OR OPERATION:GetObject OR OPERATION:"DeleteObjects")AND(NOT object:-)AND http_status>=200AND http_status <300|SELECT bucket AS"Bucket", url_decode(object)AS"文件", date_format(from_unixtime(last_access_time),'%Y-%m-%d %H:%i:%S')AS"最近访问时间", concat(cast(op_cnt ASvarchar),' ( ',(CASE WHEN write_times =0 THEN '' ELSE concat(cast(write_times ASvarchar),'写 ') END),(CASE WHEN delete_times =0 THEN '' ELSE concat(cast(delete_times ASvarchar),'删 ') END),(CASE WHEN read_times =0 THEN '' ELSE concat(cast(read_times ASvarchar),'读 ') END),')')AS"操作总数 (写, 删, 读)"FROM(SELECT object, bucket , MAX(__time__)as last_access_time, Sum(CASE WHEN OPERATION ='GetObject' THEN 1 ELSE 0 END)AS read_times , Sum(CASE WHEN OPERATION ='PutObject' THEN 1 ELSE 0 END)AS write_times , Sum(CASE WHEN OPERATION ='DeleteObject'OR OPERATION ='DeleteObjects' THEN 1 ELSE 0 END)AS delete_times ,count(*)AS op_cnt FROM log WHERE bucket LIKE'${{bucket|%}}'and access_id like'${{access_id|%}}'GROUPBY object, bucket ORDERBY delete_times DESC, write_times DESC, op_cnt DESCLIMIT1000)
查询审计
查询审计展示了AK在1天内(时间可调整)访问Bucket总数,其中公共读写Bucket、公共读Bucket、未配置数据加密的Bucket中有多少有被查询,并针对这几种场景通过列表展示各Bucket的查询次数。
构建报表所需SQL如下:
- 有查询请求Bucket总数
(__topic__: oss_access_log OR __topic__: oss_batch_delete_log)AND(NOT object:-)AND http_status>=200AND http_status <300|selectcount(1)from(select bucket ,count(1)ascountfrom log where http_method in('GET','HEAD')and access_id like'${{access_id|%}}'AND bucket LIKE'${{bucket|%}}'groupby bucket)wherecount>0
- 公共读写Bucket中有查询Bucket数量
(__topic__: oss_access_log OR __topic__: oss_batch_delete_log)AND(NOT object:-)AND http_status>=200AND http_status <300|selectcount(1)as bucket_num from(select l.idas bucket, COALESCE(r.count,0)ascountfrom"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('GET','HEAD')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.grant='public-read-write'AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)wherecount>0
- 公共读Bucket中有查询的Bucket数
(__topic__: oss_access_log OR __topic__: oss_batch_delete_log)AND(NOT object:-)AND http_status>=200AND http_status <300|selectcount(1)as bucket_num from(select l.idas bucket, COALESCE(r.count,0)ascountfrom"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('GET','HEAD')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.grant='public-read'AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)wherecount>0
- 未配置数据加密Bucket中有查询的Bucket数
(__topic__: oss_access_log OR __topic__: oss_batch_delete_log)AND(NOT object:-)AND http_status>=200AND http_status <300|selectcount(1)as bucket_num from(select l.idas bucket, COALESCE(r.count,0)ascountfrom"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('GET','HEAD')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.sse_algorithm=''AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)wherecount>0
- 公共读写Bucket查询统计
(__topic__: oss_access_log OR __topic__: oss_batch_delete_log)AND(NOT object:-)AND http_status>=200AND http_status <300|select*from(select l.idas Bucket, COALESCE(r.count,0)as"查询请求数"from"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('GET','HEAD')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.grant='public-read-write'AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)where"查询请求数">0orderby"查询请求数"desc
- 公共读Bucket查询统计
(__topic__: oss_access_log OR __topic__: oss_batch_delete_log)AND(NOT object:-)AND http_status>=200AND http_status <300|select*from(select l.idas Bucket, COALESCE(r.count,0)as"查询请求数"from"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('GET','HEAD')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.grant='public-read'AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)where"查询请求数">0orderby"查询请求数"desc
- 未配置数据加密Bucket查询统计
(__topic__: oss_access_log OR __topic__: oss_batch_delete_log)AND(NOT object:-)AND http_status>=200AND http_status <300|select*from(select l.idas Bucket, COALESCE(r.count,0)as"查询请求数"from"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('GET','HEAD')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.sse_algorithm=''AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)where"查询请求数">0orderby"查询请求数"desc
- 文件查询统计列表
(__topic__: oss_access_log or __topic__: oss_batch_delete_log)and(http_method:GET or http_method: HEAD)AND(NOT object:-)AND http_status>=200AND http_status <300|SELECT bucket AS"Bucket", url_decode(object)AS"文件", date_format(from_unixtime(last_access_time),'%Y-%m-%d %H:%i:%S')AS"最近访问时间", read_times as"读取次数"FROM(SELECT object, bucket , MAX(__time__)as last_access_time, Sum(CASE WHEN OPERATION ='GetObject' THEN 1 ELSE 0 END)AS read_times FROM log WHERE bucket LIKE'${{bucket|%}}'and access_id like'${{access_id|%}}'GROUPBY object, bucket ORDERBY read_times DESCLIMIT1000)
更新操作审计
更新操作审计展示了账号下有哪些Bucket、文件有更新操作,以及不同文件被修改次数统计。方便您查看是否有高危的操作。
构建报表所需SQL如下:
- 有更新请求Bucket总数
__topic__: oss_access_log and(http_method: PUT or http_method: POST or http_method:DELETE)AND(NOT object:-)AND http_status>=200AND http_status <300|selectcount(1)from(select bucket ,count(1)ascountfrom log where http_method in('PUT','POST','DELETE')and access_id like'${{access_id|%}}'AND bucket LIKE'${{bucket|%}}'groupby bucket)wherecount>0
- 修改文件个数
__topic__: oss_access_log and(http_method: PUT or http_method: POST or http_method:DELETE)AND(NOT object:-)AND http_status>=200AND http_status <300|SELECT approx_distinct(concat(bucket, object))AS writeCount from log WHERE bucket LIKE'${{bucket|%}}'and access_id like'${{access_id|%}}'
- 公共读Bucket中有更新的Bucket数
__topic__: oss_access_log and(http_method: PUT or http_method: POST or http_method:DELETE)AND(NOT object:-)AND http_status>=200AND http_status <300|selectcount(1)as bucket_num from(select l.idas bucket, COALESCE(r.count,0)ascountfrom"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('PUT','POST','DELETE')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.grant='public-read'AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)wherecount>0
- 公共读写Bucket中有更新Bucket数量
__topic__: oss_access_log and(http_method: PUT or http_method: POST or http_method:DELETE)AND(NOT object:-)AND http_status>=200AND http_status <300|selectcount(1)as bucket_num from(select l.idas bucket, COALESCE(r.count,0)ascountfrom"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('PUT','POST','DELETE')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.grant='public-read-write'AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)wherecount>0
- 未配置数据加密Bucket中有更新的Bucket数
__topic__: oss_access_log and(http_method: PUT or http_method: POST or http_method:DELETE)AND(NOT object:-)AND http_status>=200AND http_status <300|selectcount(1)as bucket_num from(select l.idas bucket, COALESCE(r.count,0)ascountfrom"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('PUT','POST','DELETE')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.sse_algorithm=''AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)wherecount>0
- 公共读Bucket更新统计
__topic__: oss_access_log and(http_method: PUT or http_method: POST or http_method:DELETE)AND(NOT object:-)AND http_status>=200AND http_status <300|select*from(select l.idas Bucket, COALESCE(r.count,0)as"修改请求数"from"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('PUT','POST','DELETE')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.grant='public-read'AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)where"修改请求数">0orderby"修改请求数"desc
- 未配置数据加密Bucket更新统计
__topic__: oss_access_log and(http_method: PUT or http_method: POST or http_method:DELETE)AND(NOT object:-)AND http_status>=200AND http_status <300|select*from(select l.idas Bucket, COALESCE(r.count,0)as"修改请求数"from"resource.sls.cmdb.oss"as l left join(select bucket ,count(1)ascountfrom log where http_method in('PUT','POST','DELETE')and access_id like'${{access_id|%}}'groupby bucket)as r on l.id= r.bucketWHERE l.released_flag='0'and l.sse_algorithm=''AND l.regionLIKE'${{region|%}}'AND l.idLIKE'${{bucket|%}}'groupby l.id, r.count)where"修改请求数">0orderby"修改请求数"desc
- 文件修改统计列表
__topic__: oss_access_log and(http_method: PUT or http_method: POST or http_method:DELETE)AND(NOT object:-)AND http_status>=200AND http_status <300|SELECT bucket AS"Bucket", url_decode(object)AS"文件", date_format(from_unixtime(last_access_time),'%Y-%m-%d %H:%i:%S')AS"最近访问时间", update_times as"修改次数"FROM(SELECT object, bucket , MAX(__time__)as last_access_time, Sum(CASE WHEN http_method in('PUT','POST','DELETE') THEN 1 ELSE 0 END)AS update_times FROM log WHERE bucket LIKE'${{bucket|%}}'and access_id like'${{access_id|%}}'GROUPBY object, bucket ORDERBY update_times DESCLIMIT1000)
删除操作审计
删除操作审计列举了Bucket纬度文件删除个数,以及文件删除时间。
构建报表所需SQL如下:
- Bucket删除文件统计
((__topic__: oss_access_log or __topic__: oss_batch_delete_log)and(OPERATION:"DeleteObject"OR OPERATION:"DeleteObjects")AND http_status>=200AND http_status <300)|SELECT bucket as"Bucket", sum(if(object !='-',1,0))as"删除文件个数", sum(if(delta_data_size <0,-delta_data_size, delta_data_size))as"删除总空间"from log WHERE bucket LIKE'${{bucket|%}}'and access_id like'${{access_id|%}}'GROUPBY bucket ORDERBY"删除文件个数"DESC
- 文件删除详情
((__topic__: oss_access_log or __topic__: oss_batch_delete_log)and(OPERATION:"DeleteObject"OR OPERATION:"DeleteObjects")AND(NOT object:-)AND http_status>=200AND http_status <300)|SELECT bucket as"Bucket", url_decode(object)as"文件", date_format(from_unixtime(__time__),'%Y-%m-%d %H:%i:%S')"文件删除时间"from log WHERE bucket LIKE'${{bucket|%}}'and access_id like'${{access_id|%}}'ORDERBYtimeDESCLIMIT1000
自定义仪表盘导入
构建好报表后可通过CloudLens for OSS自定义仪表盘导入功能,在Lens内做统一管理。