https://bugzilla.xfce.org/robots.txt

[André Miranda]

FWIW, there is an open bug for this:
https://bugzilla.xfce.org/show_bug.cgi?id=11404

On Fri Feb 13 2015 at 10:55:15 AM Chris Bainbridge <
chris.bainbridge at gmail.com> wrote:

> On 13 February 2015 at 10:09, Nick Schermer <nick at xfce.org> wrote:
> >
> > Yes, server capacity. CGI scripts are not very fast and search engines
> tend to send a lot of requests (many deep links it will follow).
> > That said, the bug database provides a search engine as well.
> >
> > Nick
>
> You can change the Googlebot crawl rate
>
> https://support.google.com/webmasters/answer/48620?hl=en
>
> And there is crawl-delay in robots.txt for some other agents
>
>
> http://stackoverflow.com/questions/17377835/robots-txt-what-is-the-proper-format-for-a-crawl-delay-for-multiple-user-agent
>
> http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive
>
>
> > I use Google to search for Xfce-related issues, just like everyone else.
>
> Indeed. It would help users if the bugtracker content were indexed by the
> search engines.
> _______________________________________________
> Xfce4-dev mailing list
> Xfce4-dev at xfce.org
> https://mail.xfce.org/mailman/listinfo/xfce4-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.xfce.org/pipermail/xfce4-dev/attachments/20150222/14b7118b/attachment.html>