开发者社区> 问答> 正文

如何通过scrapy请求发送cookie数据?

我一直在尝试将标头中的cookie发送到url并刮掉json响应。要发送的Cookie是cookie:"tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"

在使用python的请求库时,我得到了所需的json数据。这是我的Python代码。

import requests url = "https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3" cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;" headers = {'cookie':cookie} requests.get(url, headers=headers).json() 这是我的spider代码:

def parse(self, response): cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;" url = "https://amazon.zappos.com/mobileapif/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3" request = scrapy.Request(url=url, cookies ={'cookie': cookie},callback=self.parsejson) yield request

def parsejson(self, response):
    jsonresponse = json.loads(response.text)
    print(jsonresponse)

但spider给出了这个错误。

2019-07-16 15:59:59 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://amazon.zappos.com/c/404> from <GET https://amazon.zappos.com/mobileapif/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3> 2019-07-16 15:59:59 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://amazon.zappos.com/c/404> Cookie: cookie=tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg

2019-07-16 16:00:00 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://amazon.zappos.com/c/404> (referer: https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3) 2019-07-16 16:00:00 [scrapy.core.scraper] ERROR: Spider error processing <GET https://amazon.zappos.com/c/404> (referer: https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3) Traceback (most recent call last): File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/Users/mac/scrapy/projects/zappos/zappos/spiders/zappo.py", line 18, in parsejson jsonresponse = json.loads(response.text) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py", line 338, in loads return _default_decoder.decode(s) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded 2019-07-16 16:00:00 [scrapy.core.engine] INFO: Closing spider (finished)

展开
收起
游客6qcs5bpxssri2 2019-09-14 22:24:41 1108 0
1 条回答
写回答
取消 提交回答
  • 您需要使用以下key: value格式cookies:

    def parse(self, response):

         cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"
        url = "https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3"
    
        request = scrapy.Request(url=url, cookies ={'tid': "fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg"},callback=self.parsejson)
        yield request
    
    2019-09-14 22:25:32
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
低代码开发师(初级)实战教程 立即下载
冬季实战营第三期:MySQL数据库进阶实战 立即下载
阿里巴巴DevOps 最佳实践手册 立即下载