dagmar
deadly simple crawling/scraping package for Node.
Last updated 4 years ago by chriscx .
ISC · Original npm · Tarball · package.json
$ cnpm install dagmar 
SYNC missed versions from official npm registry.

Build Status Dependency Status devDependency Status

Dagmar

Dagmar is a deadly simple crawling/scraping package for Node.

It features:

  • A clean, simple API
  • Possible use of server-side DOM & automatic jQuery insertion with Cheerio
  • node 0.10+ support

How to install

$ npm install dagmar

Crash course


var crawler = new Crawler();

crawler.forEach(function(error, response, body) {
  if (error || response.statusCode !== 200) {
    console.log(error);
  } else {
    console.log(body);
  }
});

crawler.end(function() {
  console.log('Done.');
});

crawler.queue("http://www.google.com");
crawler.queue("http://www.yahoo.com");
crawler.queue("http://www.apple.com");
crawler.queue("http://www.twitter.com");
crawler.queue("http://www.facebook.com");

crawler.start();

Using Cheerio


var crawler = new Crawler();

crawler.forEach(function(error, response, body) {
  var $, list;
  if (!error && response.statusCode === 200) {
    $ = cheerio.load(body);
    list = $('ul', '<ul id="fruits">...</ul>');
    console.log(list);
  } else {
    console.log(error);
  }
});

crawler.end(function() {
  console.log('Done.');
});

crawler.queue("http://www.fruits.org");

crawler.start();

Full crawler retrieving href and adding to queue

var crawler = new Crawler();

crawler.forEach(function(error, response, body) {
  if (!error && response.statusCode === 200) {
    var $ = cheerio.load(body);
    return $('a').each(function(index, a) {
      var url = $(a).attr('href');
      crawler.queue(url);
    });
  } else {
    console.log(error);
  }
});

crawler.end(function() {
  console.log('Done.');
});

crawler.queue("http://www.fruits.org");

crawler.start();

Current Tags

  • 0.1.1                                ...           latest (4 years ago)

3 Versions

  • 0.1.1                                ...           4 years ago
  • 0.1.0                                ...           4 years ago
  • 0.0.2                                ...           5 years ago
Maintainers (1)
Downloads
Today 0
This Week 1
This Month 1
Last Day 0
Last Week 0
Last Month 1
Dependencies (4)
Dev Dependencies (3)
Dependents (2)

Copyright 2014 - 2016 © taobao.org |