nginx禁止垃圾蜘蛛/爬虫,禁止Scrapy等工具的抓取以及禁止非GET|HEAD|POST方式的抓取

作者:zbygseo外链优化      来源:原创      时间:2020-09-10 01:00:33      点击:

#禁止Scrapy等工具的抓取

if ($http_user_agent ~* "Scrapy|HttpClient") {

     return 403;

}

#禁止垃圾蜘蛛

if ($http_user_agent ~* "Yellowbrandprotectionbot|Nimbostratus-Bot|SurdotlyBot|Seekport Crawler|Pinterestbot|MegaIndex|Linespider|GrapeshotCrawler|coccocbot|Cincraw|MSIECrawler|Barkrowler|002pctoolspider|AspiegelBot|BLEXBot|MJ12bot|AhrefsBot|Yahoo! Slurp|Yahoo! Slurp China|YoudaoBot|MSNBot|ia_archiver|Tomato Bot|SemrushBot|DotBot"){

return 403;

}


#禁止非GET|HEAD|POST方式的抓取


if ($request_method !~ ^(GET|HEAD|POST)$) {

    return 403;

}



上一篇:python-又拍云-将指定目录下的文 | 下一篇:

随机推荐
  • 更多展示机会