generate-robotstxt
Awesome generator robots.txt
Last updated 3 years ago by itgalaxy .
MIT · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install generate-robotstxt 
SYNC missed versions from official npm registry.

generate-robotstxt

NPM version Travis Build Status dependencies Status devDependencies Status

Awesome generator robots.txt.

Installation

npm install --save-dev generate-robotstxt

Usage

import robotstxt from "generate-robotstxt";

robotstxt({
  policy: [
    {
      userAgent: "Googlebot",
      allow: "/",
      disallow: "/search",
      crawlDelay: 2,
    },
    {
      userAgent: "OtherBot",
      allow: ["/allow-for-all-bots", "/allow-only-for-other-bot"],
      disallow: ["/admin", "/login"],
      crawlDelay: 2,
    },
    {
      userAgent: "*",
      allow: "/",
      disallow: "/search",
      crawlDelay: 10,
      cleanParam: "ref /articles/",
    },
  ],
  sitemap: "http://example.com/sitemap.xml",
  host: "http://example.com",
})
  .then((content) => {
    console.log(content);

    return content;
  })
  .catch((error) => {
    throw error;
  });

File based configuration

robots-txt.config.js

module.exports = {
  policy: [
    {
      userAgent: "Googlebot",
      allow: "/",
      disallow: ["/search"],
      crawlDelay: 2,
    },
    {
      userAgent: "OtherBot",
      allow: ["/allow-for-all-bots", "/allow-only-for-other-bot"],
      disallow: ["/admin", "/login"],
      crawlDelay: 2,
    },
    {
      userAgent: "*",
      allow: "/",
      disallow: "/search",
      crawlDelay: 10,
      cleanParam: "ref /articles/",
    },
  ],
  sitemap: "http://example.com/sitemap.xml",
  host: "http://example.com",
};

CLI

Awesome generator robots.txt

  Usage generate-robotstxt [options] <dest>

  Options:
     --config  Path to a specific configuration file.

Contribution

Feel free to push your code if you agree with publishing under the MIT license.

Changelog

License

Current Tags

  • 8.0.3                                ...           latest (6 months ago)

28 Versions

  • 8.0.3                                ...           6 months ago
  • 8.0.2                                ...           8 months ago
  • 8.0.1                                ...           9 months ago
  • 8.0.0                                ...           a year ago
  • 7.1.0                                ...           a year ago
  • 7.0.0                                ...           a year ago
  • 6.0.1                                ...           2 years ago
  • 6.0.0                                ...           2 years ago
  • 5.0.7                                ...           3 years ago
  • 5.0.6                                ...           3 years ago
  • 5.0.5                                ...           3 years ago
  • 5.0.4                                ...           3 years ago
  • 5.0.3                                ...           3 years ago
  • 5.0.2                                ...           3 years ago
  • 5.0.1                                ...           3 years ago
  • 5.0.0                                ...           3 years ago
  • 4.0.4                                ...           3 years ago
  • 4.0.3                                ...           4 years ago
  • 4.0.2                                ...           4 years ago
  • 4.0.1                                ...           4 years ago
  • 4.0.0                                ...           4 years ago
  • 3.0.1                                ...           4 years ago
  • 3.0.0                                ...           4 years ago
  • 2.0.0                                ...           5 years ago
  • 1.0.3                                ...           6 years ago
  • 1.0.2                                ...           6 years ago
  • 1.0.1                                ...           6 years ago
  • 1.0.0                                ...           6 years ago
Maintainers (1)
Downloads
Today 7
This Week 14
This Month 334
Last Day 7
Last Week 67
Last Month 404
Dependencies (6)
Dev Dependencies (28)

Copyright 2014 - 2016 © taobao.org |