User-Agent: * Disallow: /js/ Disallow: /templates/ Disallow: /archi/ Disallow: /libraries/ Disallow: /cache/ Disallow: /cgi-bin/ Disallow: /DWtabs/ Disallow: /libraries/ Disallow: /includes/ Disallow: /include/ Disallow: /admin/ Disallow: /lib/ Disallow: /simplehtmldom/ # # Google did not liked my site structure # User-agent: Googlebot Disallow: /cheap/100485423/Ring/jewellery.html Disallow: /cheapest/ Disallow: /cheap/ Allow: / User-agent: Googlebot-Image Disallow: /cheapest/ Disallow: /cheap/ Allow: / # # My own list # User-agent: spbot Disallow: / User-agent: twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler Disallow: / User-agent: Exabot Disallow: / # # Crawlers that are kind enough to obey, but which we'd rather not have. # User-Agent: YahooSeeker Disallow: / User-Agent: BecomeBot Disallow: / # # A growing list of pest bots # User-Agent: BaiDuSpider Disallow: / User-Agent: CherryPicker Disallow: / User-Agent: Crescent Internet ToolPak Disallow: / User-Agent: DISCo Pump Disallow: / User-Agent: EmailSiphon Disallow: / User-Agent: htdig Disallow: / User-Agent: HTTrack Disallow: / User-Agent: LNSpiderGuy Disallow: / User-Agent: Netprospector Disallow: / User-Agent: PerMan Surfer Disallow: / User-Agent: psbot Disallow: / User-Agent: szukacz Disallow: / User-Agent: RPT-HTTPClient Disallow: / User-Agent: Teleport Disallow: / User-Agent: Telesoft Disallow: / User-Agent: Ultraseek Disallow: / User-Agent: vspider Disallow: / User-Agent: WebCopier Disallow: / User-Agent: WebEMailExtractor Disallow: / User-Agent: WebReaper Disallow: / User-Agent: Webster Pro Disallow: / User-Agent: WebStripper Disallow: / User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: InternetSeer.com Disallow: / # Some bots are known to be trouble, particularly those designed to copy # entire sites. Please obey robots.txt. User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / # # Sorry, wget in its recursive mode is a frequent problem. # Please refer to the manual and use it properly; there is a # wait = x number of seconds option you can use to set the delay between hits, # for instance. # User-agent: wget Disallow: / # # The 'grub' distributed client has been *very* poorly behaved. # User-agent: grub-client Disallow: / User-Agent: grub Disallow: / # # Doesn't follow robots.txt # User-agent: k2spider Disallow: / # # Hits many times per second, not acceptable # http://www.nameprotect.com/botinfo.html User-agent: NPBot Disallow: / # A capture bot, downloads gazillions of pages. # http://www.webreaper.net/ User-agent: WebReaper Disallow: / Sitemap: http://www.xyshop.co.uk/sitemap.xml Sitemap: http://www.xyshop.co.uk/sitemap_products2.xml Sitemap: http://www.xyshop.co.uk/sitemap_products3.xml Sitemap: http://www.xyshop.co.uk/sitemap_products4.xml Sitemap: http://www.xyshop.co.uk/sitemap_products5.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_6.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_7.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_8.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_9.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_10.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_11.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_12.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_13.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_14.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_15.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_16.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_17.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_18.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_19.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_20.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_21.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_22.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_23.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_24.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_25.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_26.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_27.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_28.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_29.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_30.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_31.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_32.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_33.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_34.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_35.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_36.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_37.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_38.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_39.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_40.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_41.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_42.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_43.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_44.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_45.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_46.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_47.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_48.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_49.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_50.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_51.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_52.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_53.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_54.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_55.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_56.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_57.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_58.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_59.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_60.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_61.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_62.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_63.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_64.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_65.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_66.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_67.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_68.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_69.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_70.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_71.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_72.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_73.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_74.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_75.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_76.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_77.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_78.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_79.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_80.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_81.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_82.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_83.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_84.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_85.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_86.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_87.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_88.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_89.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_90.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_91.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_92.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_93.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_94.xml Sitemap: http://www.xyshop.co.uk/sitemap_products_95.xml