1. 定义搜索引擎:User-agent。
  2. Disallow 禁止爬取。
  3. Allow 允许。默认情况下,都是允许的。例如:禁止admin文件夹下的所有文件,除了.html的网页。如果用Disallow一个一个的禁止,太浪费时间了。此时用Allow就解决这个问题
  4. $ 结束符。例:允许所有的以.html结尾的文件。不管前面有多长的URL,只要以.html结尾都允许
  5. * 通配符符号0或多个任意字符。例:屏蔽所有的动态URL
  6. Sitemap 声明网站地图。
  7. 根据UA分组










  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
  1. #7
    You ought to take part in a contest for one of the greatest websites on the net. I am going to highly recommend this site!
    site2019-11-28 23:40:12回复
  2. #6
    I enjoy what you guys tend to be up too. Such clever work and reporting! Keep up the amazing works guys I've incorporated you guys to blogroll.
    free sex chat2019-11-30 21:26:33回复
  3. #5
    I believe what you published made a ton of sense. However, what about this? what if you were to create a awesome post title? I ain't suggesting your content isn't good., however what if you added a title to possibly get a person's attention? I mean SEO之robots.txt爬虫协议_版主资源网 is kinda plain. You ought to glance at Yahoo's home page and watch how they create post headlines to grab viewers to open the links. You might add a video or a picture or two to get readers interested about what you've written. In my opinion, it might make your website a little livelier.
    xvideos2019-12-08 21:52:58回复
  4. #4
    Hi it's me, I am also visiting this site daily, this web site is genuinely fastidious and the people are in fact sharing good thoughts.
    tube2019-12-21 17:31:45回复
  5. #3
    Thanks for sharing your info. I really appreciate your efforts and I am waiting for your next post thank you once again.
    xxx2019-12-23 10:54:51回复
  6. #2
    No matter if some one searches for his essential thing, thus he/she wants to be available that in detail, thus that thing is maintained over here.
    fuck2019-12-27 6:49:31回复
  7. #1
    Fantastic blog! Do you have any hints for aspiring writers? I'm hoping to start my own blog soon but I'm a little lost on everything. Would you propose starting with a free platform like Wordpress or go for a paid option? There are so many options out there that I'm completely overwhelmed .. Any ideas? Many thanks!
    this site2019-12-27 11:04:11回复