1clickcrawl
packaged configuration web crawler
Last updated a year ago by rphluo .
ISC · Original npm · Tarball · package.json
$ cnpm install 1clickcrawl 
SYNC missed versions from official npm registry.

1ClickCrawler

Simple quick webcrawler for nodejs. Provide your own visit function and the crawler will do the rest.

Installation

$ npm install 1clickcrawler

Usage

var Crawler = require('1clickcrawler');
var crawler = new Crawler(delay, seeds, visit);
crawler.start()

Parameters

delay: milliseconds to wait between requests (ex: 200) seeds: array of urls to start the crawl (ex: ['https://example.com']) visit: a function that will process the data with params url, res, html (ex: function(url,res,html) {})

Current Tags

  • 1.0.1                                ...           latest (a year ago)

2 Versions

  • 1.0.1                                ...           a year ago
  • 1.0.0                                ...           a year ago
Maintainers (1)
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 2
Last Month 10
Dependencies (2)
Dev Dependencies (0)
None
Dependents (0)
None

Copyright 2014 - 2017 © taobao.org |