开发者社区> 程序猿v> 正文
阿里云
为了无法计算的价值
打开APP
阿里云APP内打开

Python爬虫:scrapy中间件及一些参数

简介: Python爬虫:scrapy中间件及一些参数
+关注继续查看

scrapy中间件

from scrapy.settings import default_settings
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100,
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300,
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware': 350,
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 400,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 500,
'scrapy.downloadermiddlewares.retry.RetryMiddleware': 550,
'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560,
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590,
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600,
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
'scrapy.downloadermiddlewares.stats.DownloaderStats': 850,
'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
from scrapy.downloadermiddlewares.robotstxt import RobotsTxtMiddleware
from scrapy.downloadermiddlewares.httpauth import HttpAuthMiddleware
from scrapy.downloadermiddlewares.downloadtimeout import DownloadTimeoutMiddleware
from scrapy.downloadermiddlewares.defaultheaders import DefaultHeadersMiddleware
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
from scrapy.downloadermiddlewares.retry import RetryMiddleware
from scrapy.downloadermiddlewares.ajaxcrawl import AjaxCrawlMiddleware
from scrapy.downloadermiddlewares.redirect import MetaRefreshMiddleware
from scrapy.downloadermiddlewares.httpcompression import HttpCompressionMiddleware
from scrapy.downloadermiddlewares.redirect import RedirectMiddleware
from scrapy.downloadermiddlewares.cookies import CookiesMiddleware
from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware
from scrapy.downloadermiddlewares.stats import DownloaderStats
from scrapy.downloadermiddlewares.httpcache import HttpCacheMiddleware

参数

# settings
USER_AGENT
ROBOTSTXT_OBEY {bool}
DOWNLOAD_TIMEOUT {float}
DEFAULT_REQUEST_HEADERS
RETRY_ENABLED {bool}
RETRY_TIMES {int}
RETRY_HTTP_CODES
RETRY_PRIORITY_ADJUST
AJAXCRAWL_ENABLED {bool}
AJAXCRAWL_MAXSIZE {int} 32768
REDIRECT_MAX_METAREFRESH_DELAY {int}
METAREFRESH_MAXDELAY {int}
COMPRESSION_ENABLED {bool}
COOKIES_ENABLED {bool}
COOKIES_DEBUG {bool}
HTTPPROXY_ENABLED
HTTPPROXY_AUTH_ENCODING
DOWNLOADER_STATS {bool}
HTTPCACHE_ENABLED {bool}
HTTPCACHE_POLICY
HTTPCACHE_STORAGE
HTTPCACHE_IGNORE_MISSING
CONCURRENT_REQUESTS 同时发出的请求
DOWNLOAD_DELAY 下载延时
HTTPERROR_ALLOWED_CODES
# request
dont_filter {bool}
# request.meta
dont_obey_robotstxt {bool}
download_timeout {float}
dont_retry {bool} False
retry_times {} 0
max_retry_times 
ajax_crawlable {bool}
dont_redirect {bool} False
dont_merge_cookies  {bool} False
cookiejar
proxy
dont_cache {bool} False
# request.headers
User-Agent
Cookie
Proxy-Authorization
# response.headers
Set-Cookie
# spider
http_user {str}
http_pass {str}
download_timeout {float}
user_agent
handle_httpstatus_list []


版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
python爬虫之微打赏(scrapy版)
上次写到单线程的微打赏爬虫,知道微打赏需要用post请求,那今天看看如何用scrapy完成post请求。 创建项目 打开cmd,输入以下代码即可创建scrapy项目。
562 0
Python爬虫之scrapy从入门到忘记
一、初窥scrapy scrapy中文文档 Scrapy是一个为了爬取网站数据,提取结构性数据而编写的应用框架。 可以应用在包括数据挖掘,信息处理或存储历史数据等一系列的程序中。
740 0
python爬虫:scrapy可视化管理工具spiderkeeper部署
python爬虫:scrapy可视化管理工具spiderkeeper部署
0 0
Python爬虫:关于scrapy模块的请求头
Python爬虫:关于scrapy模块的请求头
0 0
Python爬虫:scrapy查看Cookie值
Python爬虫:scrapy查看Cookie值
0 0
Python爬虫:scrapy直接运行爬虫
Python爬虫:scrapy直接运行爬虫
0 0
Python爬虫:scrapy管理服务器返回的cookie
Python爬虫:scrapy管理服务器返回的cookie
0 0
Python爬虫:python2使用scrapy输出unicode乱码
Python爬虫:python2使用scrapy输出unicode乱码
0 0
Python爬虫:scrapy辅助功能实用函数
Python爬虫:scrapy辅助功能实用函数
0 0
+关注
文章
问答
文章排行榜
最热
最新
相关电子书
更多
Python第五讲——关于爬虫如何做js逆向的思路
立即下载
Python 系列直播——深入Python与日志服务,玩转大规模数据分析处理实战第二讲
立即下载
Python系列直播第一讲——Python中的一切皆对象
立即下载