开发者社区> 问答> 正文

无法从Scrapy python框架中找到下载的图像

我正在制作4chan刮板,以从线程下载图像。一切正常,我能够抓取图像链接。图像管道似乎也正常工作,这是我得到的响应。

thisisppn@thisisppn-HP-15-Notebook-PC:~/Work/ScrapyTests/FourChan/FourChan$ sudo scrapy crawl imageSpider
2016-01-23 13:45:31 [scrapy] INFO: Scrapy 1.0.4 started (bot: FourChan)
2016-01-23 13:45:31 [scrapy] INFO: Optional features available: ssl, http11
2016-01-23 13:45:31 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'FourChan.spiders', 'SPIDER_MODULES': ['FourChan.spiders'], 'BOT_NAME': 'FourChan'}
2016-01-23 13:45:31 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2016-01-23 13:45:32 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-01-23 13:45:32 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-23 13:45:32 [scrapy] INFO: Enabled item pipelines: ImagesPipeline
2016-01-23 13:45:32 [scrapy] INFO: Spider opened
2016-01-23 13:45:32 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-01-23 13:45:32 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-23 13:45:33 [scrapy] DEBUG: Crawled (200) <GET http://boards.4chan.org/a/thread/136492097/would-you-still-stay-with-you-waifu-if-she-was> (referer: None)
2016-01-23 13:45:33 [scrapy] DEBUG: File (uptodate): Downloaded image from <GET http://i.4cdn.org/a/1453536312441.jpg> referred in <None>
2016-01-23 13:45:33 [scrapy] DEBUG: Scraped from <200 http://boards.4chan.org/a/thread/136492097/would-you-still-stay-with-you-waifu-if-she-was>
{'fileName': u'1453536312441.jpg',
 'image_urls': [u'http://i.4cdn.org/a/1453536312441.jpg']}
2016-01-23 13:45:33 [scrapy] INFO: Closing spider (finished)
2016-01-23 13:45:33 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 279,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 5639,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'file_count': 1,
 'file_status_count/uptodate': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 1, 23, 8, 15, 33, 468893),
 'item_scraped_count': 1,
 'log_count/DEBUG': 4,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2016, 1, 23, 8, 15, 32, 44913)}
2016-01-23 13:45:33 [scrapy] INFO: Spider closed (finished)

这是来自settings.py文件的图像管道设置

ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = '/Work/ScrapyTests/FourChan/FourChan/downloads/
'```  

该过程结束后,我去检查是否在给定目录中下载了图像,但是文件夹为空。有人请看看。

这是主要的Spider代码供参考

```js
import scrapy
import FourChan.items as items
class DmozSpider(scrapy.Spider):
    name = "imageSpider"
    allowed_domains = ["4chan.org"]
    # thread = raw_input("Enter thread link: ")
    start_urls = [
        # thread
        "http://boards.4chan.org/a/thread/136492097/would-you-still-stay-with-you-waifu-if-she-was",
    ]


    def parse(self, response):
        page = response
        item = items.FourchanItem()
        testCss = page.selector.css('.fileThumb::attr(href)').extract()
        for test in testCss:
            url = "http:"+test
            # url = test[2:]
            filename = url.split("/")[-1]
            # print filename
            item['fileName'] = filename
            item['image_urls'] = [url] #This needs to be a list, in order for the image pipeline to work, can't be a string
            yield item

展开
收起
祖安文状元 2020-02-23 16:18:00 1130 0
1 条回答
写回答
取消 提交回答
  • 这真是愚蠢,错误在于

    IMAGES_STORE = '/Work/ScrapyTests/FourChan/FourChan/downloads/'

    应该是 IMAGES_STORE = 'Work/ScrapyTests/FourChan/FourChan/downloads/'

    不需要开始的正斜杠。

    2020-02-23 16:18:08
    赞同 展开评论 打赏
问答排行榜
最热
最新

相关电子书

更多
From Python Scikit-Learn to Sc 立即下载
Data Pre-Processing in Python: 立即下载
双剑合璧-Python和大数据计算平台的结合 立即下载