开发者社区> 问答> 正文

在e-mapreduce跑hadoop mr报错,com.aliyun.oss.OSSException: AccessDenied

Error: java.io.IOException: java.lang.reflect.InvocationTargetException
    at com.aliyun.fs.oss.utils.OSSClientAgent.handleException(OSSClientAgent.java:397)
    at com.aliyun.fs.oss.utils.OSSClientAgent.deleteObject(OSSClientAgent.java:217)
    at com.aliyun.fs.oss.nat.JetOssNativeFileSystemStore.delete(JetOssNativeFileSystemStore.java:391)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy14.delete(Unknown Source)
    at com.aliyun.fs.oss.nat.NativeOssFileSystem.rename(NativeOssFileSystem.java:522)
    at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:445)
    at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:425)
    at org.apache.hadoop.mapred.Task.commit(Task.java:1163)
    at org.apache.hadoop.mapred.Task.done(Task.java:1025)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:397)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.aliyun.fs.oss.utils.OSSClientAgent.deleteObject(OSSClientAgent.java:215)
    ... 19 more
Caused by: com.aliyun.oss.OSSException: AccessDenied
[ErrorCode]: AccessDenied
[RequestId]: 57109923941E5EA00A8CC20A
[HostId]: emr-logs2.oss-cn-hangzhou-internal.aliyuncs.com
[ResponseError]:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AccessDenied</Code>
  <Message>AccessDenied</Message>
  <RequestId>57109923941E5EA00A8CC20A</RequestId>
  <HostId>emr-logs2.oss-cn-hangzhou-internal.aliyuncs.com</HostId>
</Error>
    at com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99)
    at com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:72)
    at com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:242)
    at com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:124)
    at com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:67)
    at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:92)
    at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:140)
    at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:111)
    at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:106)
    at com.aliyun.oss.internal.OSSObjectOperation.deleteObject(OSSObjectOperation.java:386)
    at com.aliyun.oss.OSSClient.deleteObject(OSSClient.java:644)
    at com.aliyun.oss.OSSClient.deleteObject(OSSClient.java:638)
    ... 24 more

展开
收起
封神 2016-04-15 15:42:11 5423 0
1 条回答
写回答
取消 提交回答
  • 专注在大数据分布式计算、数据库及存储领域,拥有13+年大数据引擎、数据仓库、宽表引擎、平台研发经验,6年云智能大数据产品技术一号位经验,10年技术团队管理经验;云智能技术架构/云布道师; 研发阿里历代的大数据技术产品包括ODPS、DLA、ADB,最近五年主导宽表引擎研发、DLA、ADB湖仓研发;

    这个问题看起来就是 访问OSS的权限不足引起的。
    在此case中,在deleteObject处发生错误。
    一般是是在RAM中,配置策略出现了错误,缺少oss:DeleteObject权限。

    全部的权限应该是:

    {
      "Version": "1",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "oss:ListBuckets"
          ],
          "Resource": [
            "acs:oss:*:*:*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
            "oss:Listobjects",
            "oss:GetObject",
            "oss:PutObject",
            "oss:DeleteObject"
          ],
          "Resource": [
            "acs:oss:*:*:emr-logs2",
            "acs:oss:*:*:emr-logs2/*"
          ]
        }
      ]
    }

    可以参考:https://help.aliyun.com/document_detail/emr/trouble-shooting/oss-ram-control.html?spm=5176.docemr/best-practice/HBase.6.198.BVMujg

    2019-07-17 18:43:21
    赞同 2 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
《构建Hadoop生态批流一体的实时数仓》 立即下载
零基础实现hadoop 迁移 MaxCompute 之 数据 立即下载
CIO 指南:如何在SAP软件架构中使用Hadoop 立即下载