Hi has anyone got a sample robot text that disallows thumbs, random images, ecards, and so on. I only want the spiders to index comments,title, description, mini cms and images.
I've tried the plugin SEO but it doesn't work very well (I believe that I read it was still at beta stage)
You can't accomplish this using a robots.txt file, as you can only specify folders that you don't want to see spidered there. You'll have a rel="nofollow" to the corresponding links and maybe even a meta tag "noindex, nofollow" to the generated page.
There are in fact quite a lot of threads that explain this in detail, please review the following threads.
- Adding meta tag for robots in secondary pages (http://forum.coppermine-gallery.net/index.php/topic,32810.0.html)
- Where can I add the no index code so Search Engines don't list my site? (http://forum.coppermine-gallery.net/index.php/topic,52769.0.html)
- No robots (http://forum.coppermine-gallery.net/index.php/topic,40721.0.html)
- Doppelindizierung durch Suchmaschinen vermeiden (http://forum.coppermine-gallery.net/index.php/topic,43177.0.html) (German)
Thanks for the quick reply. I didn't know that, but now you mention it that makes sense......don't laugh, but that would explain the 67,000 entries in my sitemap.xml ;)
I'll take a closer look at the topics you suggest. Sorry I should have looked harder in the forum before posting this.