arania
Node.js screen scraping and web crawling module
Last updated 6 years ago by dreyacosta .
MIT · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install arania 
SYNC missed versions from official npm registry.

Arania Build Status

Node.js screen scraping and web crawling module heavily inspired by Yoi project.

Installation

$ npm install arania --save

Usage

Extends crawler 'Class'

First of all you have to require arania and extends the class with two mandatory methods. Also you could export your extension as a module.

See one example

Use your crawler

Now you can import your crawler and pass some configurations:

'use strict'

RedditCrawler = require './examples/reddit.coffee'

# Options that you can pass to your crawler:
#   - cronTime: schedule crawler to run periodically
#   - requestsToStopper: timeout your crawler every X requests
#   - stopperTimeout: milliseconds for the crawler stopper
redditCrawler = new RedditCrawler
  cronTime: '00 38 * * * *'
  requestsToStopper: 100
  stopperTimeout: 30000

Current Tags

  • 0.1.2                                ...           latest (6 years ago)

3 Versions

  • 0.1.2                                ...           6 years ago
  • 0.1.1                                ...           6 years ago
  • 0.1.0                                ...           6 years ago
Maintainers (1)
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 0
Last Month 1
Dependencies (5)
Dev Dependencies (3)
Dependents (0)
None

Copyright 2014 - 2016 © taobao.org |