climateprediction.net (CPDN) home page
Thread 'robots.txt missing'

Thread 'robots.txt missing'

Questions and Answers : Wish list : robots.txt missing
Message board moderation

To post messages, you must log in.

AuthorMessage
ProfileAnanas
Volunteer moderator

Send message
Joined: 31 Oct 04
Posts: 336
Credit: 3,316,482
RAC: 0
Message 9233 - Posted: 13 Feb 2005, 1:23:13 UTC

This is sure important for all BOINC based projects :

Currently the search engines index the result pages, result lists, work unit pages and so on.

From the traffic, those search engines produce on my own web site, I can imagine they produce a lot of queries on BOINC databases too.

Maybe someone should think about a robots.txt (or .htaccess) that at least excludes the pages with database queries from beeing indexed by search spiders.

(I posted this in the SETI@BOINC forum too)
ID: 9233 · Report as offensive     Reply Quote
ProfileKeck_Komputers
Avatar

Send message
Joined: 5 Aug 04
Posts: 426
Credit: 2,426,069
RAC: 0
Message 9268 - Posted: 13 Feb 2005, 19:52:31 UTC

The fix is in. The projects will have to update the server side software for it to take effect though.
BOINC WIKI

BOINCing since 2002/12/8
ID: 9268 · Report as offensive     Reply Quote

Questions and Answers : Wish list : robots.txt missing

©2024 cpdn.org