我一直在尝试将标头中的cookie发送到url并刮掉json响应。要发送的Cookie是cookie:"tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"
在使用python的请求库时,我得到了所需的json数据。这是我的Python代码。
import requests url = "https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3" cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;" headers = {'cookie':cookie} requests.get(url, headers=headers).json() 这是我的spider代码:
def parse(self, response): cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;" url = "https://amazon.zappos.com/mobileapif/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3" request = scrapy.Request(url=url, cookies ={'cookie': cookie},callback=self.parsejson) yield request
def parsejson(self, response):
jsonresponse = json.loads(response.text)
print(jsonresponse)
但spider给出了这个错误。
2019-07-16 15:59:59 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://amazon.zappos.com/c/404> from <GET https://amazon.zappos.com/mobileapif/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3> 2019-07-16 15:59:59 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://amazon.zappos.com/c/404> Cookie: cookie=tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg
2019-07-16 16:00:00 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://amazon.zappos.com/c/404> (referer: https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3) 2019-07-16 16:00:00 [scrapy.core.scraper] ERROR: Spider error processing <GET https://amazon.zappos.com/c/404> (referer: https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3) Traceback (most recent call last): File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/Users/mac/scrapy/projects/zappos/zappos/spiders/zappo.py", line 18, in parsejson jsonresponse = json.loads(response.text) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py", line 338, in loads return _default_decoder.decode(s) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded 2019-07-16 16:00:00 [scrapy.core.engine] INFO: Closing spider (finished)
您需要使用以下key: value格式cookies:
def parse(self, response):
cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"
url = "https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3"
request = scrapy.Request(url=url, cookies ={'tid': "fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg"},callback=self.parsejson)
yield request
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。