正常运行爬虫,在公司电脑上没问题,回到家自己电脑上就出现了下面的报错,百思不得解
Traceback (most recent call last): File "D:\.virtualenvs\spider\lib\site-packages\twisted\internet\defer.py", line 1384, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "D:\.virtualenvs\spider\lib\site-packages\twisted\python\failure.py", line 422, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "D:\.virtualenvs\spider\lib\site-packages\scrapy\core\downloader\middleware.py", line 43, in process_request defer.returnValue((yield download_func(request=request,spider=spider))) File "D:\.virtualenvs\spider\lib\site-packages\scrapy\utils\defer.py", line 45, in mustbe_deferred result = f(*args, **kw) File "D:\.virtualenvs\spider\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 65, in download_request return handler.download_request(request, spider) File "D:\.virtualenvs\spider\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 60, in download_request return agent.download_request(request) File "D:\.virtualenvs\spider\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 285, in download_request method, to_bytes(url, encoding='ascii'), headers, bodyproducer) File "D:\.virtualenvs\spider\lib\site-packages\twisted\web\client.py", line 1657, in request parsedURI.originForm) File "D:\.virtualenvs\spider\lib\site-packages\twisted\web\client.py", line 1435, in _requestWithEndpoint d = self._pool.getConnection(key, endpoint) File "D:\.virtualenvs\spider\lib\site-packages\twisted\web\client.py", line 1320, in getConnection return self._newConnection(key, endpoint) File "D:\.virtualenvs\spider\lib\site-packages\twisted\web\client.py", line 1332, in _newConnection return endpoint.connect(factory) File "D:\.virtualenvs\spider\lib\site-packages\twisted\internet\endpoints.py", line 924, in connect EndpointReceiver, self._hostText, portNumber=self._port File "D:\.virtualenvs\spider\lib\site-packages\twisted\internet\_resolver.py", line 189, in resolveHostName onAddress = self._simpleResolver.getHostByName(hostName) File "D:\.virtualenvs\spider\lib\site-packages\scrapy\resolver.py", line 21, in getHostByName d = super(CachingThreadedResolver, self).getHostByName(name, timeout) File "D:\.virtualenvs\spider\lib\site-packages\twisted\internet\base.py", line 276, in getHostByName timeoutDelay = sum(timeout) TypeError: 'float' object is not iterable
网上查到是scrapy与twisted版本不兼容的问题
当前版本:
Scrapy==1.1.2 Twisted==18.4.0
卸载twisted重装
pip uninstall twisted pip install twisted==16.6.0
参考
scrapy domz报错TypeError: ‘float’ object is not iterable
Scapry does not work with Twisted version 17.1.0