定期更新ip的脚本:
#!/bin/sh
#定时更新公司IP, 以用来过滤
#author: Felix Zhang
#date: 2012-12-29
filedir=/opt/logdata/companyip
adate=$(date -d "today" +"%Y%m%d")
filename="${filedir}/ip.${adate}"
ip=`/usr/bin/host yourcompany.3322.org|awk '{print $4}'`
if [ '' != "`grep $ip ${filename}`" ]; then
exit 0
fi
echo "$ip" >> ${filename}
#Set how long you want to save
save_days=30
#delete 30 days ago nginx log files
find ${filedir} -mtime +${save_days} -exec rm -rf {} \;
分析日志的脚本:
#!/bin/sh
ipdir=/opt/logdata/companyip
adate=$(date -d "today" +"%Y%m%d")
ipfile="${ipdir}/ip.${adate}"
ipreg="127.0.0.1"
if [ -e ${ipfile} ]; then
ipreg=`cat ${ipfile} |sed ':a N;s/\n/|/;ta'`
echo "1"
fi
if [ "${ipreg}" = "" ]; then
ipreg="127.0.0.1"
echo "2"
fi
echo ${ipreg}
#cat ip.test |grep -E -v '127.0.0.1|126.23.23.44'
fileName=$1;
echo '分析文件'$fileName
cat $fileName | egrep -v ${ipreg} |awk '{print $7}'
这样分析日志时就可以过滤掉自己公司的IP了.
当然根据机器人的特征也可以过滤机器人, 此处不在多说了, 此处仅给出几个机器人
cat ${logfile} |grep -E -v ${ipreg} |grep -E -v "DNSPod-monitor|bot.htm|spider.htm|webmasters.htm" >${cleanlogfile}