开发者社区 问答 正文

scrapy框架中如何实现大文件的下载?

scrapy框架中如何实现大文件的下载?

展开
收起
珍宝珠 2019-11-22 13:57:16 2300 分享
分享
版权
举报
1 条回答
写回答
取消 提交回答
  • from twisted.web.client import Agent, getPage, ResponseDone, PotentialDataLoss
    from twisted.internet import defer, reactor, protocol
    from twisted.web._newclient import Response
    from io import BytesIO
    
    class _ResponseReader(protocol.Protocol):
        def __init__(self, finished, txresponse, file_name):
            self._finished = finished
            self._txresponse = txresponse
            self._bytes_received = 0
            self.f = open(file_name, mode='wb')
        def dataReceived(self, bodyBytes):
            self._bytes_received += len(bodyBytes)
            # 一点一点的下载
            self.f.write(bodyBytes)
            self.f.flush()
        def connectionLost(self, reason):
            if self._finished.called:
                return
            if reason.check(ResponseDone):
                # 下载完成
                self._finished.callback((self._txresponse, 'success'))
            elif reason.check(PotentialDataLoss):
                # 下载部分
                self._finished.callback((self._txresponse, 'partial'))
            else:
                # 下载异常
                self._finished.errback(reason)
            self.f.close()
    
    2019-11-22 13:57:27 举报
    赞同 评论

    评论

    全部评论 (0)

    登录后可评论
AI助理

你好,我是AI助理

可以解答问题、推荐解决方案等