https://bugzilla.xfce.org/robots.txt

[Chris Bainbridge]

On 13 February 2015 at 10:09, Nick Schermer <nick at xfce.org> wrote:
>
> Yes, server capacity. CGI scripts are not very fast and search engines
tend to send a lot of requests (many deep links it will follow).
> That said, the bug database provides a search engine as well.
>
> Nick

You can change the Googlebot crawl rate

https://support.google.com/webmasters/answer/48620?hl=en

And there is crawl-delay in robots.txt for some other agents

http://stackoverflow.com/questions/17377835/robots-txt-what-is-the-proper-format-for-a-crawl-delay-for-multiple-user-agent
http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive

> I use Google to search for Xfce-related issues, just like everyone else.

Indeed. It would help users if the bugtracker content were indexed by the
search engines.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.xfce.org/pipermail/xfce4-dev/attachments/20150213/6f9bdb8e/attachment.html>