nextjs - 💡(How to fix) Fix Support non-standard properties in `robots.ts` file [1 comments, 2 participants]

// @ts-expect-error not a standard property I would expect this non-standard directive to still be propagated. Obviously, it does not make sense to include it in the types, so I’m fine with suppressing the error, but it should still propagate to robots.txt. It is very useful when Seznam bots crawl the site too aggressively.

TL;DR

The Request-Rate directive is being ignored in the generated robots.txt file, likely due to it being a non-standard property.

Guidance

• The issue is likely caused by Next.js not supporting the non-standard Request-Rate property in the MetadataRoute.Robots type.
• To verify, check the generated robots.txt file to see if the Request-Rate directive is present.
• As a workaround, consider creating a custom robots.txt file with the desired Request-Rate directive, rather than relying on the MetadataRoute.Robots type.
• Alternatively, try using a different property name or syntax to see if Next.js will propagate the directive to the robots.txt file.

Example

// Example of a custom robots.txt file with the Request-Rate directive
// User-agent: SeznamBot
// Allow: /
// Request-Rate: 10/1m

Notes

The Request-Rate directive is not a standard property in the robots.txt specification, so it's unclear whether Next.js will ever support it. The workaround of creating a custom robots.txt file may be the most reliable solution.

Recommendation

Apply workaround: Create a custom robots.txt file with the desired Request-Rate directive, as this allows for more control over the file's contents and avoids relying on Next.js to support a non-standard property.