nginx block常用爬虫

代码如下
agent匹配不区分大小写

if ($http_user_agent ~* "qihoobot|Googlebot-Mobile|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo! Slurp China|YoudaoBot|Sosospider|Sogou spider|Sogou web spider|MSNBot|ia_archiver|Tomato Bot|bingbot|ChinasoSpider|Sogou inst spider") {
    return 403; 
}

上面的没有 baidu 百度的为

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)