@crawlify.io/robots
Parses robots.txt files to provide meaningful, useful output as well as reporting syntax errors.
Last updated a year ago by _j4wx .
MIT · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install @crawlify.io/robots 
SYNC missed versions from official npm registry.

Crawlify/Robots

A robots.txt parser for node.js

This package comes with two options for parsing robots.txt files, either fetch - which retrieves the file from the URL provided and parses the response, or parse - which simply parses the textual response provided.

let RobotFetch = require('robot');
RobotFetch.fetch('https://reckless.agency/robots.txt', function() {
  console.log(RobotFetch.rulesets);
  console.log(RobotFetch.sitemaps);
});

let RobotParse = require('robot');
RobotParse.parse(someRobotsContent, function() {
    console.log(RobotParse.rulesets);
    console.log(RobotParse.sitemaps);
});

If any lines of the robots.txt cannot be understood by the parser, they will be returned in Robot.unknown.

More features will be added as we move forward.

Current Tags

  • 1.0.1                                ...           latest (a year ago)

2 Versions

  • 1.0.1                                ...           a year ago
  • 1.0.0                                ...           a year ago
Maintainers (2)
Downloads
Today 0
This Week 0
This Month 2
Last Day 0
Last Week 2
Last Month 1
Dependencies (1)
Dev Dependencies (0)
None
Dependents (0)
None

Copyright 2014 - 2016 © taobao.org |