abbijamal / robots-module

NuxtJS module for robots.txt

@nuxtjs/robots

A Nuxt.js module that injects a middleware to generate a robots.txt file

• 📖 Release Notes

Features

• Nuxt 3 and Nuxt Bridge support
• Generate robots.txt for static mode
• Add middleware for robots.txt

Setup

1. Add @nuxtjs/robots dependency to your project

yarn add @nuxtjs/robots # or npm install @nuxtjs/robots

2. Add @nuxtjs/robots to the modules section of nuxt.config.js

export default {
  modules: [
    // Simple usage
    '@nuxtjs/robots',

    // With options
    ['@nuxtjs/robots', { /* module options */ }]
  ]
}

Using top level options

export default {
  modules: [
    '@nuxtjs/robots'
  ],
  robots: {
    /* module options */
  }
}

Options

configPath

• Type: String
• Default: robots.config

rules

• Type: Object|Array
• Default:

{
  UserAgent: '*',
  Disallow: ''
}

Robots config

If you need to use function in any rule, you need to create a config file through the configPath option

export default [
  { UserAgent: '*' },
  { Disallow: '/' },
  { BlankLine: true },
  { Comment: 'Comment here' },
      
  // Be aware that this will NOT work on target: 'static' mode
  { Sitemap: (req) => `https://${req.headers.host}/sitemap.xml` }
]

output:

User-agent: *
Disallow: /

# Comment here
Sitemap: https://robots.nuxtjs.org/sitemap.xml

The keys and values available:

• UserAgent = User-agent
• CrawlDelay = Crawl-delay
• Disallow = Disallow
• Allow = Allow
• Host = Host
• Sitemap = Sitemap
• CleanParam = Clean-param
• Comment = # Comment
• BlankLine = Add blank line

Note: Don't worry, keys are parsed with case insensitivity and special characters.

Contributing

You can contribute to this module online with CodeSandBox:

Or locally:

1. Clone this repository
2. Install dependencies using pnpm install
3. Prepare development server using pnpm dev:prepare
4. Build module using pnpm build
5. Launch playground using pnpm dev

License

MIT License

Copyright (c) - Nuxt Community