我做的应用是以Spring为系统的基础框架,mysql为后台数据库.在tomcat上发布后,总是不能进行热部署(reload),多次reload后,就会出OutOfMemory PermGen,
为此烦恼了很久,总于下定决心找找根源.
经过3天的不懈努力,小有成果,记录下来
实际上下面的分析都已经没什么用了,如果你使用tomcat6.0.26及以后的版本,我所说的这些情况都已经被处理了,并且比我处理的还要多很多.可以下载tomcat6.0.26的源代码
看看WebappClassLoader类的处理就成了.
通过分析工具的分析(用了YourKit,以及JDK1.6/bin下的jps/jmap/jhat),发现有下面几个方面会造成memory leak.
1.SystemClassLoader与WebappClassLoader加载的类相互引用,tomcat reload只是卸载WebappClassloader中的class,SystemClassLoader是不会卸载的(否则其他应用也停止了).但是WebappClassloader加载的类被SystemClassLoader引用的化,WebappClassloader中的相关类就不会被JVM进行垃圾收集
目前发现2种容易产生这种leak的现象.
a.在使用java.lang.ThreadLocal的时候很容易产生这种情况
b.使用jdbc驱动,而且不是在tomcat中配置的公共连接池.则java.sql.DriverManager一定会产生这种现象
ThreadLocal.set(Object),如果这个Object是WebappsClassLoader加载的,使用之后没有做ThreadLocal.set(null)或者ThreadLocal.remove(),就会产生memory leak.
由于ThreadLocal实际上操作的是java.lang.Thread类中的ThreadLocalMap,Thread类是由SystemClassLoder加载的.而这个线程实例(main thread)在tomcat reload的时候不会销毁重建,必然就产生了SystemClassLoder中的类引用WebappsClassLoader的类.
DriverManager也是由SystemClassLoder载入的,当初始化某个JDBC驱动的时候,会向DriverManager中注册该驱动,通常是***.driver,例如com.mysql.jdbc.Driver
这个Driver是通过class.forName()加载的,通常也是加载到WebappClassLoader.这就出现了两个classLoader中的类的交叉引用.导致memory leak.
解决办法:
写一个ServletContextListener,在contextDestroyed方法中统一删除当前Thread的ThreadLocalMap中的内容.
public class ApplicationCleanListener implements ServletContextListener {
public void contextInitialized(ServletContextEvent event) {
}
public void contextDestroyed(ServletContextEvent event) {
//处理ThreadLocal
ThreadLocalCleanUtil.clearThreadLocals();
/*
* 如果数据故驱动是通过应用服务器(tomcat etc...)中配置的<公用>连接池,这里不需要 否则必须卸载Driver
*
* 原因: DriverManager是System classloader加载的, Driver是webappclassloader加载的,
* driver保存在DriverManager中,在reload过程中,由于system
* classloader不会销毁,driverManager就一直保持着对driver的引用,
* driver无法卸载,与driver关联的其他类
* ,例如DataSource,jdbcTemplate,dao,service....都无法卸载
*/
try {
System.out.println("clean jdbc Driver......");
for (Enumeration e = DriverManager.getDrivers(); e
.hasMoreElements();) {
Driver driver = (Driver) e.nextElement();
if (driver.getClass().getClassLoader() == getClass()
.getClassLoader()) {
DriverManager.deregisterDriver(driver);
}
}
} catch (Exception e) {
System.out
.println("Exception cleaning up java.sql.DriverManager's driver: "
+ e.getMessage());
}
}
}
/**
* 这个类根据
*/
public class ThreadLocalCleanUtil {
/**
* 得到当前线程组中的所有线程 description:
*
* @return
*/
private static Thread[] getThreads() {
ThreadGroup tg = Thread.currentThread().getThreadGroup();
while (tg.getParent() != null) {
tg = tg.getParent();
}
int threadCountGuess = tg.activeCount() + 50;
Thread[] threads = new Thread[threadCountGuess];
int threadCountActual = tg.enumerate(threads);
while (threadCountActual == threadCountGuess) {
threadCountGuess *= 2;
threads = new Thread[threadCountGuess];
threadCountActual = tg.enumerate(threads);
}
return threads;
}
public static void clearThreadLocals() {
ClassLoader classloader = Thread
.currentThread()
.getContextClassLoader();
Thread[] threads = getThreads();
try {
Field threadLocalsField = Thread.class
.getDeclaredField("threadLocals");
threadLocalsField.setAccessible(true);
Field inheritableThreadLocalsField = Thread.class
.getDeclaredField("inheritableThreadLocals");
inheritableThreadLocalsField.setAccessible(true);
Class tlmClass = Class
.forName("java.lang.ThreadLocal$ThreadLocalMap");
Field tableField = tlmClass.getDeclaredField("table");
tableField.setAccessible(true);
for (int i = 0; i < threads.length; ++i) {
if (threads[i] == null)
continue;
Object threadLocalMap = threadLocalsField.get(threads[i]);
clearThreadLocalMap(threadLocalMap, tableField, classloader);
threadLocalMap = inheritableThreadLocalsField.get(threads[i]);
clearThreadLocalMap(threadLocalMap, tableField, classloader);
}
} catch (Exception e) {
e.printStackTrace();
}
}
private static void clearThreadLocalMap(Object map,
Field internalTableField, ClassLoader classloader)
throws NoSuchMethodException, IllegalAccessException,
NoSuchFieldException, InvocationTargetException {
if (map != null) {
Method mapRemove = map.getClass().getDeclaredMethod("remove",
new Class[] { ThreadLocal.class });
mapRemove.setAccessible(true);
Object[] table = (Object[]) internalTableField.get(map);
int staleEntriesCount = 0;
if (table != null) {
for (int j = 0; j < table.length; ++j) {
if (table[j] != null) {
boolean remove = false;
Object key = ((Reference) table[j]).get();
if ((key != null)
&& (key.getClass().getClassLoader() == classloader)) {
remove = true;
System.out.println("clean threadLocal key,class="
+ key.getClass().getCanonicalName()
+ ",value=" + key.toString());
}
Field valueField = table[j]
.getClass()
.getDeclaredField("value");
valueField.setAccessible(true);
Object value = valueField.get(table[j]);
if ((value != null)
&& (value.getClass().getClassLoader() == classloader)) {
remove = true;
System.out.println("clean threadLocal value,class="
+ value.getClass().getCanonicalName()
+ ",value=" + value.toString());
}
if (remove) {
if (key == null)
++staleEntriesCount;
else {
mapRemove.invoke(map, new Object[] { key });
}
}
}
}
}
if (staleEntriesCount > 0) {
Method mapRemoveStale = map
.getClass()
.getDeclaredMethod("expungeStaleEntries", new Class[0]);
mapRemoveStale.setAccessible(true);
mapRemoveStale.invoke(map, new Object[0]);
}
}
}
}
2.对于使用mysql JDBC驱动的:mysql JDBC驱动会启动一个Timer Thread,这个线程在reload的时候也是无法自动销毁.
因此,需要强制结束这个timer
可以在 上面的ApplicationCleanListener中加入如下代码:
try {
Class ConnectionImplClass = Thread
.currentThread()
.getContextClassLoader()
.loadClass("com.mysql.jdbc.ConnectionImpl");
if (ConnectionImplClass != null
&& ConnectionImplClass.getClassLoader() == getClass()
.getClassLoader()) {
System.out.println("clean mysql timer......");
Field f = ConnectionImplClass.getDeclaredField("cancelTimer");
f.setAccessible(true);
Timer timer = (Timer) f.get(null);
timer.cancel();
}
} catch (java.lang.ClassNotFoundException e1) {
// do nothing
} catch (Exception e) {
System.out
.println("Exception cleaning up MySQL cancellation timer: "
+ e.getMessage());
}
3.common-logging+log4j似乎也会导致leak,看网上有人说在ApplicationCleanListene6中加入这行代码就可以:
LogFactory.release(Thread.currentThread().getContextClassLoader());
我没试成功,懒得再找原因,直接换成了slf4j+logback,没有问题.据说slf4j+logback的性能还要更好.
后记:
tomcat-6.0.26之前的版本(我用的是tomcat-6.0.18),加入上述ApplicationCleanListener后,多次reload,不会出现outOfMemory.
但要注意,第一次启动后,reload一次,内存会增加,也就是看着还是由memory Leak,但是重复reload,内存始终保持在第一次reload的大小.似乎tomcat始终保留了双WebappClassLoader.因此,配置内存要小心些,至少要保证能够load两倍的你的所有jar包的大小(当然,是指Perm的内存大小).
测试过程中最好加上 JVM参数 -verbosegc,这样,在做GC的时候可以直观的看到class被卸载.