express-robots
Express middleware for generating your robots.txt
Last updated 4 years ago by ashaffer88 .
MIT · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install express-robots 
SYNC missed versions from official npm registry.

express-robots

Express middleware for generating a robots.txt or responding with an existing file.

Using a file

app.use(robots(__dirname + '/robots.txt'));

Using an object

Basic object

app.use(robots({UserAgent: '*', Disallow: '/'}))

Will produce:

UserAgent: *
Disallow: /

Crawl Delay

You can optionally pass a CrawlDelay in just like passing in Disallow

app.use(robots({UserAgent: '*', Disallow: '/', CrawlDelay: '5'}))

Will produce:

UserAgent: *
Disallow: /
Crawl-delay: 5

Or an array of objects

app.use(robots([
  {
    UserAgent: 'Googlebot',
    Disallow: '/no-google'
  },
  {
    UserAgent: 'Bingbot',
    Disallow: '/no-bing'
  }
]));

Will produce:

UserAgent: Googlebot
Disallow: /no-google
UserAgent: Bingbot
Disallow: /no-bing

Or either property (UserAgent | Disallow) as an array

app.use(robots([
  {
    UserAgent: [ 'Googlebot', 'Slurp' ],
    Disallow: [ '/no-google', '/no-yahoo' ]
  },
  {
    UserAgent: '*',
    Disallow: [ '/no-bots', '/still-no-bots' ]
  }
]));

Will produce:

UserAgent: Googlebot
UserAgent: Slurp
Disallow: /no-google
Disallow: /no-yahoo
UserAgent: *
Disallow: /no-bots
Disallow: /still-no-bots

Current Tags

  • 0.1.6                                ...           latest (4 years ago)

8 Versions

  • 0.1.6                                ...           4 years ago
  • 0.2.0                                ...           5 years ago
  • 0.1.5                                ...           6 years ago
  • 0.1.4                                ...           6 years ago
  • 0.1.3                                ...           6 years ago
  • 0.1.2                                ...           6 years ago
  • 0.1.1                                ...           6 years ago
  • 0.1.0                                ...           6 years ago
Maintainers (1)
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 4
Last Month 7
Dependencies (1)
  • as-array git://github.com/weo-edu/as-array.git
Dev Dependencies (4)

Copyright 2014 - 2016 © taobao.org |