Not sure if my robots.txt is okay Not sure if my robots.txt is okay
 

News:

CPG Release 1.6.26
Correct PHP8.2 issues with user and language managers.
Additional fixes for PHP 8.2
Correct PHP8 error with SMF 2.0 bridge.
Correct IPTC supplimental category parsing.
Download and info HERE

Main Menu

Not sure if my robots.txt is okay

Started by macsimoin, July 02, 2006, 04:46:50 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

macsimoin

Since I've registered a new domain and relocated my gallery into the root of the new domain, I added a robots.txt file there as well. Can someone have a look at this and let me know if it looks okay or if it's in the right place?

#
# robots.txt for http://www.kiltphotos.com/ and friends
#

User-agent: Zealbot
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: linko
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Microsoft.URL.Control
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libwww
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: Download Ninja
Disallow: /

User-agent: wget
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: k2spider
Disallow: /

User-agent: NPBot
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: Konqueror
Disallow: /

User-agent: msnbot
Disallow:

User-agent: psbot
Disallow:

User-agent: Googlebot
Disallow:

User-agent: Googlebot-Image
Disallow:

User-agent: *
Disallow: /albums/
Disallow: /bridge/
Disallow: /docs/
Disallow: /images/
Disallow: /include/
Disallow: /lang/
Disallow: /logs/
Disallow: /plugins/
Disallow: /README_FILES/
Disallow: /sql/
Disallow: /themes/


Thanks. :)