我正在制作4chan刮板,以从线程下载图像。一切正常,我能够抓取图像链接。图像管道似乎也正常工作,这是我得到的响应。
thisisppn@thisisppn-HP-15-Notebook-PC:~/Work/ScrapyTests/FourChan/FourChan$ sudo scrapy crawl imageSpider
2016-01-23 13:45:31 [scrapy] INFO: Scrapy 1.0.4 started (bot: FourChan)
2016-01-23 13:45:31 [scrapy] INFO: Optional features available: ssl, http11
2016-01-23 13:45:31 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'FourChan.spiders', 'SPIDER_MODULES': ['FourChan.spiders'], 'BOT_NAME': 'FourChan'}
2016-01-23 13:45:31 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2016-01-23 13:45:32 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-01-23 13:45:32 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-23 13:45:32 [scrapy] INFO: Enabled item pipelines: ImagesPipeline
2016-01-23 13:45:32 [scrapy] INFO: Spider opened
2016-01-23 13:45:32 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-01-23 13:45:32 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-23 13:45:33 [scrapy] DEBUG: Crawled (200) <GET http://boards.4chan.org/a/thread/136492097/would-you-still-stay-with-you-waifu-if-she-was> (referer: None)
2016-01-23 13:45:33 [scrapy] DEBUG: File (uptodate): Downloaded image from <GET http://i.4cdn.org/a/1453536312441.jpg> referred in <None>
2016-01-23 13:45:33 [scrapy] DEBUG: Scraped from <200 http://boards.4chan.org/a/thread/136492097/would-you-still-stay-with-you-waifu-if-she-was>
{'fileName': u'1453536312441.jpg',
'image_urls': [u'http://i.4cdn.org/a/1453536312441.jpg']}
2016-01-23 13:45:33 [scrapy] INFO: Closing spider (finished)
2016-01-23 13:45:33 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 279,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 5639,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'file_count': 1,
'file_status_count/uptodate': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 1, 23, 8, 15, 33, 468893),
'item_scraped_count': 1,
'log_count/DEBUG': 4,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2016, 1, 23, 8, 15, 32, 44913)}
2016-01-23 13:45:33 [scrapy] INFO: Spider closed (finished)
这是来自settings.py文件的图像管道设置
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = '/Work/ScrapyTests/FourChan/FourChan/downloads/
'```
该过程结束后,我去检查是否在给定目录中下载了图像,但是文件夹为空。有人请看看。
这是主要的Spider代码供参考
```js
import scrapy
import FourChan.items as items
class DmozSpider(scrapy.Spider):
name = "imageSpider"
allowed_domains = ["4chan.org"]
# thread = raw_input("Enter thread link: ")
start_urls = [
# thread
"http://boards.4chan.org/a/thread/136492097/would-you-still-stay-with-you-waifu-if-she-was",
]
def parse(self, response):
page = response
item = items.FourchanItem()
testCss = page.selector.css('.fileThumb::attr(href)').extract()
for test in testCss:
url = "http:"+test
# url = test[2:]
filename = url.split("/")[-1]
# print filename
item['fileName'] = filename
item['image_urls'] = [url] #This needs to be a list, in order for the image pipeline to work, can't be a string
yield item
这真是愚蠢,错误在于
IMAGES_STORE = '/Work/ScrapyTests/FourChan/FourChan/downloads/'
应该是 IMAGES_STORE = 'Work/ScrapyTests/FourChan/FourChan/downloads/'
不需要开始的正斜杠。
版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。