We recently found a robots.txt file on an NLM site that blocks all spiders except Google. Is the government allowed to do that? Does anyone know if this is common?
NSF blocks all indexing of the site between 7AM and 7PM ET, our peak traffic hours, for the convenience of our users. However, there is no block on the site from 7PM to 7AM ET. This is standard policy for most high traffic sites. The owner of [the Wayback Machine] need only comply with our policy in order to index our pages.
Could there be a similar explanation at the NLM?
Posted by
Peter Suber at 3/01/2007 06:03:00 PM.
The open access movement:
Putting peer-reviewed scientific and scholarly literature
on the internet. Making it available free of charge and
free of most copyright and licensing restrictions.
Removing the barriers to serious research.
